Skip to contentSkip to conversation

Error mounting FC8 NFS server with 2.6.31-rc3 NFSv4 client.

DISCUSSION Ben Greear 4 participants 25 messages Jul 20, 2009 – Jul 22, 2009

Subsystems (2)
lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I tried mounting an NFS server running FC8 (2.6.26.8-57.fc8 kernel) using a Fedora 11 system running an un-patched 2.6.31-rc3 64-bit kernel.

I am not sure at all that the FC8 system is set up to handle NFSv4 properly, but I was expecting some sort of useful error if that was the case.

Instead, I get this continually spewing to /var/log/messages:

Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
...

On the file-server, I see this:

Jul 19 04:47:59 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out
Jul 19 04:49:29 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out
Jul 19 04:49:29 fs2 ntpd[2585]: kernel time sync status change 0001
Jul 19 04:50:59 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out

I added some debug patches (on top of my other patches, including those to nfs) and got some debug info:

It seems that nfs4_init_clientid is returning -2, and establish_clid is returning -2. This causes reclaim lease logic to fail, and that causes state manager to print out the error repeatedly. -2 means ENOENT.

As far as I can tell, the mount never completes, staying in D state and filling up logs (I deleted a 16GB /var/log/messages file a few minutes ago!)

The mount command I'm trying is:
mount -t nfs4 192.168.100.6:/export/tmp /mnt/lf/nfs4-0

My kernel config is found here:
http://www.candelatech.com/oss/i7_config.txt

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw
Quoted 32 lines · Ben Greear

I tried mounting an NFS server running FC8 (2.6.26.8-57.fc8 kernel) using a Fedora 11 system running an un-patched 2.6.31-rc3 64-bit kernel.

I am not sure at all that the FC8 system is set up to handle NFSv4 properly, but I was expecting some sort of useful error if that was the case.

Instead, I get this continually spewing to /var/log/messages:

Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
Error: state manager failed on NFSv4 server 192.168.100.6 with error 2
...

On the file-server, I see this:

Jul 19 04:47:59 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out
Jul 19 04:49:29 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out
Jul 19 04:49:29 fs2 ntpd[2585]: kernel time sync status change 0001
Jul 19 04:50:59 fs2 kernel: nfs4_cb: server 192.168.100.196 not responding, timed out

I added some debug patches (on top of my other patches, including those to nfs) and got some debug info:

It seems that nfs4_init_clientid is returning -2, and establish_clid is returning -2. This causes reclaim lease logic to fail, and that causes state manager to print out the error repeatedly. -2 means ENOENT.

As far as I can tell, the mount never completes, staying in D state and filling up logs (I deleted a 16GB /var/log/messages file a few minutes ago!)

Quoted 7 lines

The mount command I'm trying is:
mount -t nfs4 192.168.100.6:/export/tmp /mnt/lf/nfs4-0

My kernel config is found here:
http://www.candelatech.com/oss/i7_config.txt

What does /var/lib/nfs/v4recovery look like on the server?

Trond

author
#2 lore raw
Quoted 2 lines · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

I had my /etc/exports looking like this on the file-server:

/export/tmp 192.168.100.0/24(rw)

When I changed it to be:
/export/tmp 192.168.100.0/24(rw,fsid=0)

And mounted 192.168.100.6:/ instead of 192.168.100.6:/export/tmp then it mounts properly and appears to work just fine.

If I leave file-server configured properly with fsid=0, but try to mount 192.168.100.6:/export/tmp then I get an error about no such file or directory, which also appears to be correct behaviour.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#3 lore raw
Quoted 9 lines · Ben Greear
Quoted 2 lines · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Now, we _should_ fix the wretched NFS server so that it doesn't do NFSv4 mounts when there is no configured root partition. We _should_ also fix the damned thing so that it doesn't return illegal values.

Trond

author
#4 lore raw
Quoted 4 lines · Trond Myklebust
Quoted 3 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

Quoted 8 lines
Quoted 4 lines

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

Quoted 3 lines

Now, we _should_ fix the wretched NFS server so that it doesn't do NFSv4 mounts when there is no configured root partition. We _should_ also fix the damned thing so that it doesn't return illegal values.

Sounds fine to me.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#5 lore raw
Quoted 10 lines · Ben Greear
Quoted 9 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 6 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Trond

author
#6 lore raw
Quoted 10 lines · Trond Myklebust
Quoted 9 lines · Ben Greear
Quoted 8 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Thanks,
Ben

Quoted 2 lines

Trond

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#7 lore raw
Quoted 17 lines · Ben Greear
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Ben Greear
Quoted 11 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

Quoted 8 lines
Quoted 1 line

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Trond

author
#8 lore raw
Quoted 22 lines · Trond Myklebust
Quoted 21 lines · Ben Greear
Quoted 10 lines · Trond Myklebust
Quoted 9 lines · Ben Greear
Quoted 8 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Quoted 5 lines
Quoted 2 lines

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Yes, it did.

Thanks,
Ben

Quoted 2 lines

Trond

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#9 lore raw
Quoted 23 lines · Ben Greear
Quoted 22 lines · Trond Myklebust
Quoted 21 lines · Ben Greear
Quoted 10 lines · Trond Myklebust
Quoted 9 lines · Ben Greear
Quoted 8 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Yes, it did.

OK. Then the NFSv4.1 merge has screwed up the state management. We do NOT want to hold client id state on the server when there is no file state...

Trond

#10 lore raw
Quoted 3 lines · Trond Myklebust

OK. Then the NFSv4.1 merge has screwed up the state management. We do NOT want to hold client id state on the server when there is no file state...

Hmm. I doubt that as I can reproduce the same problem (at least the mount hanging, not necessarily the errors) with .30, which I presume is before the merge.

I have seen the errors too, but that may have been with 2.6.31 (though definitely without NFS_V4_1). I seem unable to reproduce those with .30.

On the server (.30) in /etc/exportfs:
/srv/exports 10.19.66.0/24(rw,subtree_check,sync,mp,nohide)
/srv/exports/data 10.19.66.0/24(rw,subtree_check,sync,mp,nohide)

# exportfs
/srv/exports         10.19.66.0/24
/srv/exports/data    10.19.66.0/24

On the client (.30.2):

# mount thorin:/data /mnt -t nfs4
<hangs>

Cheers,
FJP

#11 lore raw
In reply to Ben Greear
Quoted 23 lines · Ben Greear
Quoted 22 lines · Trond Myklebust
Quoted 21 lines · Ben Greear
Quoted 10 lines · Trond Myklebust
Quoted 9 lines · Ben Greear
Quoted 8 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Yes, it did.

OK. The following patch should fix that particular regression.

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised. I'm going to leave that for the moment, though, since that bug wasn't introduced by my patch, and it doesn't affect NFSv4.0.

Cheers,
Trond
-----------------------------------------------------------------
From: Trond Myklebust <[email protected]>
NFSv4: Fix an NFSv4 mount regression

Commit 008f55d0e019943323c20a03493a2ba5672a4cc8 (nfs41: recover lease in _nfs4_lookup_root) forces the state manager to always run on mount. This is a bug in the case of NFSv4.0, which doesn't require us to send a setclientid until we want to grab file state.

In any case, this is completely the wrong place to be doing state management. Moving that code into nfs4_init_session...

---

 fs/nfs/client.c   |   18 +++---------------
 fs/nfs/nfs4_fs.h  |    6 ++++++
 fs/nfs/nfs4proc.c |   24 +++++++++++++++++-------
 3 files changed, 26 insertions(+), 22 deletions(-)
fs/nfs/client.c +3 -15
index c2d0616..8d25ccb 100644
@@ -1242,20 +1242,6 @@ error:
 	return error;
 }
 
-/*
- * Initialize a session.
- * Note: save the mount rsize and wsize for create_server negotiation.
- */
-static void nfs4_init_session(struct nfs_client *clp,
-			      unsigned int wsize, unsigned int rsize)
-{
-#if defined(CONFIG_NFS_V4_1)
-	if (nfs4_has_session(clp)) {
-		clp->cl_session->fc_attrs.max_rqst_sz = wsize;
-		clp->cl_session->fc_attrs.max_resp_sz = rsize;
-	}
-#endif /* CONFIG_NFS_V4_1 */
-}
 
 /*
  * Session has been established, and the client marked ready.
@@ -1350,7 +1336,9 @@ struct nfs_server *nfs4_create_server(const struct nfs_parsed_mount_data *data,
 	BUG_ON(!server->nfs_client->rpc_ops);
 	BUG_ON(!server->nfs_client->rpc_ops->file_inode_ops);
 
-	nfs4_init_session(server->nfs_client, server->wsize, server->rsize);
+	error = nfs4_init_session(server);
+	if (error < 0)
+		goto error;
 
 	/* Probe the root fh to retrieve its FSID */
 	error = nfs4_path_walk(server, mntfh, data->nfs_server.export_path);
fs/nfs/nfs4_fs.h +6 -0
index 61bc3a3..6ea07a3 100644
@@ -220,6 +220,7 @@ extern void nfs4_destroy_session(struct nfs4_session *session);
 extern struct nfs4_session *nfs4_alloc_session(struct nfs_client *clp);
 extern int nfs4_proc_create_session(struct nfs_client *, int reset);
 extern int nfs4_proc_destroy_session(struct nfs4_session *);
+extern int nfs4_init_session(struct nfs_server *server);
 #else /* CONFIG_NFS_v4_1 */
 static inline int nfs4_setup_sequence(struct nfs_client *clp,
 		struct nfs4_sequence_args *args, struct nfs4_sequence_res *res,
@@ -227,6 +228,11 @@ static inline int nfs4_setup_sequence(struct nfs_client *clp,
 {
 	return 0;
 }
+
+static inline int nfs4_init_session(struct nfs_server *server)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 
 extern struct nfs4_state_maintenance_ops *nfs4_state_renewal_ops[];
fs/nfs/nfs4proc.c +17 -7
index ff0c080..df24f67 100644
@@ -2040,15 +2040,9 @@ static int _nfs4_lookup_root(struct nfs_server *server, struct nfs_fh *fhandle,
 		.rpc_argp = &args,
 		.rpc_resp = &res,
 	};
-	int status;
 
 	nfs_fattr_init(info->fattr);
-	status = nfs4_recover_expired_lease(server);
-	if (!status)
-		status = nfs4_check_client_ready(server->nfs_client);
-	if (!status)
-		status = nfs4_call_sync(server, &msg, &args, &res, 0);
-	return status;
+	return nfs4_call_sync(server, &msg, &args, &res, 0);
 }
 
 static int nfs4_lookup_root(struct nfs_server *server, struct nfs_fh *fhandle,
@@ -4793,6 +4787,22 @@ int nfs4_proc_destroy_session(struct nfs4_session *session)
 	return status;
 }
 
+int nfs4_init_session(struct nfs_server *server)
+{
+	struct nfs_client *clp = server->nfs_client;
+	int ret;
+
+	if (!nfs4_has_session(clp))
+		return 0;
+
+	clp->cl_session->fc_attrs.max_rqst_sz = server->wsize;
+	clp->cl_session->fc_attrs.max_resp_sz = server->rsize;
+	ret = nfs4_recover_expired_lease(server);
+	if (!ret)
+		ret = nfs4_check_client_ready(clp);
+	return ret;
+}
+
 /*
  * Renew the cl_session lease.
  */
#12 lore raw
Quoted 32 lines · Trond Myklebust
Quoted 23 lines · Ben Greear
Quoted 22 lines · Trond Myklebust
Quoted 21 lines · Ben Greear
Quoted 10 lines · Trond Myklebust
Quoted 9 lines · Ben Greear
Quoted 8 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Yes, it did.

Quoted 7 lines

OK. The following patch should fix that particular regression.

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised. I'm going to leave that for the moment, though, since that bug wasn't introduced by my patch, and it doesn't affect NFSv4.0.

With that patch applied, I now get the expected behaviour when mounting a server with no NFSv4 root from a 2.6.31-rc3 based client:

root@heimdal:~# mount -t nfs4 munkholm.local:/ /mnt
mount.nfs4: mounting munkholm.local:/ failed, reason given by server:
  No such file or directory

Trond

author
#13 lore raw
Quoted 6 lines · Trond Myklebust
Quoted 5 lines

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised. I'm going to leave that for the moment, though, since that bug wasn't introduced by my patch, and it doesn't affect NFSv4.0.

Quoted 8 lines

With that patch applied, I now get the expected behaviour when mounting a server with no NFSv4 root from a 2.6.31-rc3 based client:

root@heimdal:~# mount -t nfs4 munkholm.local:/ /mnt
mount.nfs4: mounting munkholm.local:/ failed, reason given by server:
   No such file or directory

Trond

I just successfully tested it on my systems as well, so I think this does indeed fix the problem.

Thanks!
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#14 lore raw
In reply to Trond Myklebust
Quoted 30 lines · Trond Myklebust
Quoted 29 lines · Ben Greear
Quoted 27 lines · Trond Myklebust
Quoted 16 lines · Ben Greear
Quoted 15 lines · Trond Myklebust
Quoted 14 lines · Ben Greear
Quoted 11 lines · Trond Myklebust
Quoted 7 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

So how should the client behave, when a screwed up server allows it to mount but starts returning illegal values for setclientid? The only thing I can see we could do is to tell the user EINSANESERVER...

Well, it could just fail the mount and give up and not overly spam /var/log/messages in a tight loop perhaps?

Quoted 7 lines
Quoted 1 line

This doesn't happen at mount time. It happens when you open a file.

Not for me, and evidently not for the other person that reported similar results. All I had to do was attempt the mount (which never completed).

Thanks,
Ben

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Quoted 7 lines
Quoted 1 line

Yes, it did.

OK. The following patch should fix that particular regression.

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised.

The rsize/wsize is copied into the session prior to the create_session call (triggered by the state management code you moved), and is used for session negotiation. At this point the nfs_client cl_cons_state is set to NFS_CS_SESSION_INITING (see nfs4_alloc_session), so the nfs_client is not initialized. The cl_cons_state is set to NFS_CS_READY after a successful create_session call.

-->Andy

Quoted 143 lines

I'm going to leave that for the moment, though,
since that bug wasn't introduced by my patch, and it doesn't affect NFSv4.0.

Cheers,
Trond
-----------------------------------------------------------------
From: Trond Myklebust <[email protected]>
NFSv4: Fix an NFSv4 mount regression

Commit 008f55d0e019943323c20a03493a2ba5672a4cc8 (nfs41: recover lease in _nfs4_lookup_root) forces the state manager to always run on mount. This is a bug in the case of NFSv4.0, which doesn't require us to send a setclientid until we want to grab file state.

In any case, this is completely the wrong place to be doing state management. Moving that code into nfs4_init_session...

---

fs/nfs/client.c   |   18 +++---------------
fs/nfs/nfs4_fs.h  |    6 ++++++
fs/nfs/nfs4proc.c |   24 +++++++++++++++++-------
3 files changed, 26 insertions(+), 22 deletions(-)
fs/nfs/client.c +0 -3
index c2d0616..8d25ccb 100644
@@ -1242,20 +1242,6 @@ error:
 	return error;
 }
 
-/*
- * Initialize a session.
- * Note: save the mount rsize and wsize for create_server  

negotiation.
- */

fs/nfs/client.c +0 -10
-static void nfs4_init_session(struct nfs_client *clp,
-			      unsigned int wsize, unsigned int rsize)
-{
-#if defined(CONFIG_NFS_V4_1)
-	if (nfs4_has_session(clp)) {
-		clp->cl_session->fc_attrs.max_rqst_sz = wsize;
-		clp->cl_session->fc_attrs.max_resp_sz = rsize;
-	}
-#endif /* CONFIG_NFS_V4_1 */
-}

/*
* Session has been established, and the client marked ready.

fs/nfs/client.c +3 -1
@@ -1350,7 +1336,9 @@ struct nfs_server *nfs4_create_server(const  
 struct nfs_parsed_mount_data *data,
 	BUG_ON(!server->nfs_client->rpc_ops);
 	BUG_ON(!server->nfs_client->rpc_ops->file_inode_ops);
 
-	nfs4_init_session(server->nfs_client, server->wsize, server->rsize);
+	error = nfs4_init_session(server);
+	if (error < 0)
+		goto error;
 
 	/* Probe the root fh to retrieve its FSID */
	error = nfs4_path_walk(server, mntfh, data->nfs_server.export_path);
fs/nfs/nfs4_fs.h +1 -0
index 61bc3a3..6ea07a3 100644
@@ -220,6 +220,7 @@ extern void nfs4_destroy_session(struct  
 nfs4_session *session);
 extern struct nfs4_session *nfs4_alloc_session(struct nfs_client *clp);
 extern int nfs4_proc_create_session(struct nfs_client *, int reset);
 extern int nfs4_proc_destroy_session(struct nfs4_session *);
+extern int nfs4_init_session(struct nfs_server *server);
 #else /* CONFIG_NFS_v4_1 */
 static inline int nfs4_setup_sequence(struct nfs_client *clp,

struct nfs4_sequence_args *args, struct nfs4_sequence_res *res,

fs/nfs/nfs4_fs.h +5 -0
@@ -227,6 +228,11 @@ static inline int nfs4_setup_sequence(struct  
 nfs_client *clp,
 {
 	return 0;
 }
+
+static inline int nfs4_init_session(struct nfs_server *server)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 

extern struct nfs4_state_maintenance_ops *nfs4_state_renewal_ops[];

fs/nfs/nfs4proc.c +1 -7
index ff0c080..df24f67 100644
@@ -2040,15 +2040,9 @@ static int _nfs4_lookup_root(struct  
 nfs_server *server, struct nfs_fh *fhandle,
 		.rpc_argp = &args,
 		.rpc_resp = &res,
 	};
-	int status;
 
 	nfs_fattr_init(info->fattr);
-	status = nfs4_recover_expired_lease(server);
-	if (!status)
-		status = nfs4_check_client_ready(server->nfs_client);
-	if (!status)
-		status = nfs4_call_sync(server, &msg, &args, &res, 0);
-	return status;
+	return nfs4_call_sync(server, &msg, &args, &res, 0);
 }
 

static int nfs4_lookup_root(struct nfs_server *server, struct nfs_fh *fhandle,

fs/nfs/nfs4proc.c +16 -0
@@ -4793,6 +4787,22 @@ int nfs4_proc_destroy_session(struct  
 nfs4_session *session)
 	return status;
 }
 
+int nfs4_init_session(struct nfs_server *server)
+{
+	struct nfs_client *clp = server->nfs_client;
+	int ret;
+
+	if (!nfs4_has_session(clp))
+		return 0;
+
+	clp->cl_session->fc_attrs.max_rqst_sz = server->wsize;
+	clp->cl_session->fc_attrs.max_resp_sz = server->rsize;
+	ret = nfs4_recover_expired_lease(server);
+	if (!ret)
+		ret = nfs4_check_client_ready(clp);
+	return ret;
+}
+
 /*
 * Renew the cl_session lease.

*/

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#15 lore raw
Quoted 11 lines · Andy Adamson
Quoted 3 lines · Trond Myklebust

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised.

The rsize/wsize is copied into the session prior to the create_session call (triggered by the state management code you moved), and is used for session negotiation. At this point the nfs_client cl_cons_state is set to NFS_CS_SESSION_INITING (see nfs4_alloc_session), so the nfs_client is not initialized. The cl_cons_state is set to NFS_CS_READY after a successful create_session call.

The call to nfs4_init_session() is in nfs4_create_server(). It can be called several times _after_ the nfs_client has been initialised when you mount more than one partition from the same NFS server.

If that is the case, and if you use different rsize/wsize values on those different mounts, then you will end up clobbering the values of fc_attrs.max_rqst_sz, and fc_attrs.max_resp_sz, having set them to the wsize/rsize that was set by the very last mount call.

AFAICS, what you _should_ be doing in nfs4_init_session, is something like

	if (clp->cl_session->fc_attrs.max_rqst_sz < server->wsize)
		clp->cl_session->fc_attrs.max_rqst_sz = server->wsize;

Trond

#16 lore raw
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Andy Adamson
Quoted 3 lines · Trond Myklebust

Note that there is a bug remaining inside nfs4_init_session(): we shouldn't be copying the rsize/wsize into the nfs_client if the latter was already initialised.

The rsize/wsize is copied into the session prior to the create_session call (triggered by the state management code you moved), and is used for session negotiation. At this point the nfs_client cl_cons_state is set to NFS_CS_SESSION_INITING (see nfs4_alloc_session), so the nfs_client is not initialized. The cl_cons_state is set to NFS_CS_READY after a successful create_session call.

Quoted 8 lines

The call to nfs4_init_session() is in nfs4_create_server(). It can be called several times _after_ the nfs_client has been initialised when you mount more than one partition from the same NFS server.

If that is the case, and if you use different rsize/wsize values on those different mounts, then you will end up clobbering the values of fc_attrs.max_rqst_sz, and fc_attrs.max_resp_sz, having set them to the wsize/rsize that was set by the very last mount call.

You are right, this is a bug.

Quoted 7 lines

AFAICS, what you _should_ be doing in nfs4_init_session, is something like

	if (clp->cl_session->fc_attrs.max_rqst_sz < server->wsize)
		clp->cl_session->fc_attrs.max_rqst_sz = server->wsize;

Currently, we only support one session per nfs_client - one session for all partition mounts to the same NFS server.
For NFSv4.1 the per partition struct nfs_server rsize/wsize is used to construct rpc requests that are sent over the session so the rsize/wsize must not be larger than the session max_resp_sz/max_rqst_sz.

nfs4_init_session should simply return if the nfs_client cl_cons_state is not NFS_CS_SESSION_INITING.
I shouldn't be trying to set the session max_resp_sz/max_rqst_sz to the rsize/wsize, but rather to the maximum rsize/wsize supported by the client.
If the server accepts or increases the max_resp_sz/max_rqst_sz then all is well.
If the server reduces the max_resp_sz/max_rqst_sz, the maximum rsize/wsize available for NFSv4.1 partition mounts to the server needs to be reduced accordingly. So the nfs_server rsize/wsize needs to be bound by the session max_resp_sz/max_rqst_sz as well as by the maximum supported size.

If you think this is correct, I'll send a patch.

-->Andy

Quoted 4 lines

Trond

#17 lore raw
Quoted 4 lines · Andy Adamson

nfs4_init_session should simply return if the nfs_client cl_cons_state is not NFS_CS_SESSION_INITING.
I shouldn't be trying to set the session max_resp_sz/max_rqst_sz to the rsize/wsize, but rather to the maximum rsize/wsize supported by

Quoted 8 lines

the client.
If the server accepts or increases the max_resp_sz/max_rqst_sz then all is well.
If the server reduces the max_resp_sz/max_rqst_sz, the maximum rsize/ wsize available for NFSv4.1 partition mounts to the server needs to be reduced accordingly. So the nfs_server rsize/wsize needs to be bound by the session max_resp_sz/max_rqst_sz as well as by the maximum supported size.

Well.... The rsize/wsize is one thing, but how about acls? We simply don't know how big they may become.

An alternative would be to let the server choose the max_rqst_sz/max_resp_sz by always requesting the maximum allowed value.

Trond

#18 lore raw
Quoted 14 lines · Trond Myklebust
Quoted 4 lines · Andy Adamson

nfs4_init_session should simply return if the nfs_client cl_cons_state is not NFS_CS_SESSION_INITING.
I shouldn't be trying to set the session max_resp_sz/max_rqst_sz to the rsize/wsize, but rather to the maximum rsize/wsize supported by

Quoted 8 lines

the client.
If the server accepts or increases the max_resp_sz/max_rqst_sz then all is well.
If the server reduces the max_resp_sz/max_rqst_sz, the maximum rsize/ wsize available for NFSv4.1 partition mounts to the server needs to be reduced accordingly. So the nfs_server rsize/wsize needs to be bound by the session max_resp_sz/max_rqst_sz as well as by the maximum supported size.

Quoted 2 lines

Well.... The rsize/wsize is one thing, but how about acls? We simply don't know how big they may become.

Yeah - has anything been submitted for 4.2 to fix this? (a readdir cookie - like thingie)

Quoted 4 lines

An alternative would be to let the server choose the max_rqst_sz/max_resp_sz by always requesting the maximum allowed value.

OK - maximum allowed value instead of maximum client supported value...

Quoted 3 lines

Trond

#19 lore raw
Quoted 6 lines · Andy Adamson
Quoted 2 lines · Trond Myklebust

Well.... The rsize/wsize is one thing, but how about acls? We simply don't know how big they may become.

Yeah - has anything been submitted for 4.2 to fix this? (a readdir cookie - like thingie)

Not really. You pretty much have to set ACLs atomically, or you may end up with security issues.

The good news is, though, that we should be able to figure out an upper limit by looking at the max_rqst_sz/max_resp_sz that the server wants to set.

#20 lore raw
Quoted 10 lines · Trond Myklebust
Quoted 6 lines · Andy Adamson
Quoted 2 lines · Trond Myklebust

Well.... The rsize/wsize is one thing, but how about acls? We simply don't know how big they may become.

Yeah - has anything been submitted for 4.2 to fix this? (a readdir cookie - like thingie)

Not really. You pretty much have to set ACLs atomically, or you may end up with security issues.

Sure, but we could help get ACL.

Quoted 5 lines

The good news is, though, that we should be able to figure out an upper limit by looking at the max_rqst_sz/max_resp_sz that the server wants to set.

Yep.

-->Andy

Quoted 3 lines
#21 lore raw
In reply to Trond Myklebust
Quoted 2 lines · Trond Myklebust

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Eh, no. In my case both client and server are 2.6.30 which did not yet have NFS_V4_1 (and in all my .31 configs I did disable it).

#22 lore raw
Quoted 6 lines · Frans Pop
Quoted 2 lines · Trond Myklebust

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Eh, no. In my case both client and server are 2.6.30 which did not yet have NFS_V4_1 (and in all my .31 configs I did disable it).

...and you are seeing the state manager running when you try to mount an NFSv4 partition using the 2.6.30 client?

Trond

#23 lore raw
Quoted 10 lines · Trond Myklebust
Quoted 6 lines · Frans Pop
Quoted 2 lines · Trond Myklebust

Ah... You have NFS_V4_1 enabled despite the Kconfig warning... Does the bug occur when you turn this off too?

Eh, no. In my case both client and server are 2.6.30 which did not yet have NFS_V4_1 (and in all my .31 configs I did disable it).

...and you are seeing the state manager running when you try to mount an NFSv4 partition using the 2.6.30 client?

Per my previous message, I think there may be two related issues:
- mount hanging when no NFS4 root is defined, reproducible on .30
- state errors occurring in that situation, which may have been introduced with .31

I've just checked and when I saw the state errors I was running .31-rc3 on the client, and partially also on the server. I saw error 2, 5 and 10022. See log below.

I first switched back to .30 on the server to see if that fixed the mount problem (it didn't) and later also switched back to .30 on the client.

(Sorry for the kernel version confusion, but I've been working on different issues requiring frequent reboots; but I hope the info helps.)

Cheers,
FJP

Log from the client:

Jul 19 20:10:16 aragorn kernel: Linux version 2.6.31-rc3
[...]
Jul 20 00:44:45 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 2
Jul 20 00:49:27 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 00:52:30 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 00:55:30 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 00:59:39 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:01:15 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
[... server rebooted from 2.6.31-rc3 to 2.6.30 ...]
Jul 20 01:06:37 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:09:37 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:12:37 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:15:37 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:18:37 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 10022
Jul 20 01:23:52 aragorn kernel: Error: state manager failed on NFSv4 server thorin with error 5

Unrelated, but I just noticed the following on the server (with 2.6.30); not sure what exactly triggered that, maybe a stop/start of the NFS services:

Jul 20 01:23:22 thorin kernel: RPC: Unregistered udp transport module.
Jul 20 01:23:22 thorin kernel: RPC: Unregistered tcp transport module.
Jul 20 01:23:22 thorin kernel: nfsd: failed to unregister auth.unix.gid cache
Jul 20 01:23:22 thorin kernel: ------------[ cut here ]------------
Jul 20 01:23:22 thorin kernel: WARNING: at /home/fjp/projects/kernel/linux-2.6/fs/proc/generic.c:847 
remove_proc_entry+0x164/0x1b0()
Jul 20 01:23:22 thorin kernel: remove_proc_entry: removing non-empty directory 'net/rpc', leaking at least 'auth.unix.gid'
Jul 20 01:23:22 thorin kernel: Modules linked in: ipv6 exportfs sunrpc(-) dm_mod loop evdev ehci_hcd usbcore mv643xx_eth libphy 
gpio_keys [last unloaded: auth_rpcgss]
Jul 20 01:23:22 thorin kernel: [<c002ab6c>] (unwind_backtrace+0x0/0xd4) from [<c003fd78>] (warn_slowpath_common+0x4c/0x68)
Jul 20 01:23:22 thorin kernel: [<c003fd78>] (warn_slowpath_common+0x4c/0x68) from [<c003fdd0>] (warn_slowpath_fmt+0x28/0x38)
Jul 20 01:23:22 thorin kernel: [<c003fdd0>] (warn_slowpath_fmt+0x28/0x38) from [<c00e43c4>] (remove_proc_entry+0x164/0x1b0)
Jul 20 01:23:22 thorin kernel: [<c00e43c4>] (remove_proc_entry+0x164/0x1b0) from [<c00628b8>] (sys_delete_module+0x1d0/0x230)
Jul 20 01:23:22 thorin kernel: [<c00628b8>] (sys_delete_module+0x1d0/0x230) from [<c0024ce0>] (ret_fast_syscall+0x0/0x2c)
Jul 20 01:23:22 thorin kernel: ---[ end trace b5320efdfa0e8be3 ]---
#24 lore raw
In reply to Ben Greear
Quoted 15 lines · Ben Greear
Quoted 1 line · Trond Myklebust

What does /var/lib/nfs/v4recovery look like on the server?

The server was misconfigured, but I still think the client should behave better in this case. If you cannot reproduce it, let me know and I can try to be more specific. If you still want the v4recovery information, let me know and I'll send it.

I had my /etc/exports looking like this on the file-server:

/export/tmp 192.168.100.0/24(rw)

When I changed it to be:
/export/tmp 192.168.100.0/24(rw,fsid=0)

Quoted 7 lines

And mounted 192.168.100.6:/ instead of 192.168.100.6:/export/tmp then it mounts properly and appears to work just fine.

If I leave file-server configured properly with fsid=0, but try to mount 192.168.100.6:/export/tmp then I get an error about no such file or directory, which also appears to be correct behaviour.

Duh! I had exactly the same problem this week when I "quickly" set up an NFS server on one of my boxes: forgot to add fsid=0 for the NFS4 root. Result was the mount hanging on the client and the same errors as Ben reported.

It would be great if this configuration problem was detected better.

P.S. /var/lib/nfs/v4recovery on the server was empty in my case.