NFSv4 uninitialized mtime

cat badmtime.c

vi badmtime.c
gcc badmtime.c -o badmtime
./badmtime
ls -l badmtime.out

stat badmtime.out

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Quoted 9 lines · Stuart Anderson

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

Quoted 5 lines

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Doesn't Jeff's patch fix it?

Trond

From: Jeff Layton <[email protected]>
Date: Tue, 5 Jun 2007 14:49:03 -0400
NFS4: on a O_EXCL OPEN make sure SETATTR sets the fields holding the
verifier
Subject: No Subject

The Linux NFS4 client simply skips over the bitmask in an O_EXCL open call and so it doesn't bother to reset any fields that may be holding the verifier. This patch has us save the first two words of the bitmask (which is all the current client has #defines for). The client then later checks this bitmask and turns on the appropriate flags in the sattr->ia_verify field for the following SETATTR call.

This patch only currently checks to see if the server used the atime and mtime slots for the verifier (which is what the Linux server uses for this). I'm not sure of what other fields the server could reasonably use, but adding checks for others should be trivial.

Signed-off-by Jeff Layton <[email protected]>

Signed-off-by Trond Myklebust <[email protected]>

---

 fs/nfs/nfs4proc.c       |   20 ++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |    9 +++++++--
 include/linux/nfs4.h    |    1 +
 include/linux/nfs_xdr.h |    1 +
 4 files changed, 29 insertions(+), 2 deletions(-)

fs/nfs/nfs4proc.c +20 -0

index 3cc7544..fee2d14 100644
@@ -943,6 +943,22 @@ static struct nfs4_state *nfs4_open_delegated(struct inode *inode, int flags, st
 }
 
 /*
+ * on an EXCLUSIVE create, the server should send back a bitmask with FATTR4-*
+ * fields corresponding to attributes that were used to store the verifier.
+ * Make sure we clobber those fields in the later setattr call
+ */
+static inline void nfs4_exclusive_attrset(struct nfs4_opendata *opendata, struct iattr *sattr)
+{
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_ACCESS) &&
+	    !(sattr->ia_valid & ATTR_ATIME_SET))
+		sattr->ia_valid |= ATTR_ATIME;
+
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_MODIFY) &&
+	    !(sattr->ia_valid & ATTR_MTIME_SET))
+		sattr->ia_valid |= ATTR_MTIME;
+}
+
+/*
  * Returns a referenced nfs4_state
  */
 static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct iattr *sattr, struct rpc_cred *cred, struct nfs4_state **res)

@@ -973,6 +989,9 @@ static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct
 	if (status != 0)
 		goto err_opendata_free;
 
+	if (opendata->o_arg.open_flags & O_EXCL)
+		nfs4_exclusive_attrset(opendata, sattr);
+
 	status = -ENOMEM;
 	state = nfs4_opendata_to_nfs4_state(opendata);
 	if (state == NULL)

@@ -1784,6 +1803,7 @@ nfs4_proc_create(struct inode *dir, struct dentry *dentry, struct iattr *sattr,
 		status = nfs4_do_setattr(state->inode, &fattr, sattr, state);
 		if (status == 0)
 			nfs_setattr_update_inode(state->inode, sattr);
+		nfs_post_op_update_inode(state->inode, &fattr);
 	}
 	if (status == 0 && (nd->flags & LOOKUP_OPEN) != 0)
 		status = nfs4_intent_set_file(nd, &path, state);

fs/nfs/nfs4xdr.c +7 -2

index 8003c91..5efd314 100644
@@ -3269,7 +3269,7 @@ static int decode_delegation(struct xdr_stream *xdr, struct nfs_openres *res)
 static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
 {
         __be32 *p;
-        uint32_t bmlen;
+        uint32_t savewords, bmlen, i;
         int status;
 
         status = decode_op_hdr(xdr, OP_OPEN);

@@ -3287,7 +3287,12 @@ static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
                 goto xdr_error;
 
         READ_BUF(bmlen << 2);
-        p += bmlen;
+	savewords = min_t(uint32_t, bmlen, NFS4_BITMAP_SIZE);
+	for (i = 0; i < savewords; ++i)
+		READ32(res->attrset[i]);
+
+	p += (bmlen - savewords);
+
 	return decode_delegation(xdr, res);
 xdr_error:
 	dprintk("%s: Bitmap too large! Length = %u\n", __FUNCTION__, bmlen);

include/linux/nfs4.h +1 -0

index 7e7f33a..8726491 100644
@@ -15,6 +15,7 @@
 
 #include <linux/types.h>
 
+#define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
 #define NFS4_STATEID_SIZE	16
 #define NFS4_FHSIZE		128

include/linux/nfs_xdr.h +1 -0

index 10c26ed..f7100df 100644
@@ -144,6 +144,7 @@ struct nfs_openres {
 	nfs4_stateid		delegation;
 	__u32			do_recall;
 	__u64			maxsize;
+	__u32			attrset[NFS4_BITMAP_SIZE];
 };
 
 /*

Unfortunately this patch did not have any observable affects.

Quoted 146 lines · Trond Myklebust

Quoted 9 lines · Stuart Anderson

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

Quoted 5 lines

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Doesn't Jeff's patch fix it?

Trond

From: Jeff Layton <[email protected]>
Date: Tue, 5 Jun 2007 14:49:03 -0400
NFS4: on a O_EXCL OPEN make sure SETATTR sets the fields holding the
verifier
Subject: No Subject

The Linux NFS4 client simply skips over the bitmask in an O_EXCL open call and so it doesn't bother to reset any fields that may be holding the verifier. This patch has us save the first two words of the bitmask (which is all the current client has #defines for). The client then later checks this bitmask and turns on the appropriate flags in the sattr->ia_verify field for the following SETATTR call.

This patch only currently checks to see if the server used the atime and mtime slots for the verifier (which is what the Linux server uses for this). I'm not sure of what other fields the server could reasonably use, but adding checks for others should be trivial.

Signed-off-by Jeff Layton <[email protected]>

Signed-off-by Trond Myklebust <[email protected]>

---

 fs/nfs/nfs4proc.c       |   20 ++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |    9 +++++++--
 include/linux/nfs4.h    |    1 +
 include/linux/nfs_xdr.h |    1 +
 4 files changed, 29 insertions(+), 2 deletions(-)

fs/nfs/nfs4proc.c +20 -0

index 3cc7544..fee2d14 100644
@@ -943,6 +943,22 @@ static struct nfs4_state *nfs4_open_delegated(struct inode *inode, int flags, st
 }
 
 /*
+ * on an EXCLUSIVE create, the server should send back a bitmask with FATTR4-*
+ * fields corresponding to attributes that were used to store the verifier.
+ * Make sure we clobber those fields in the later setattr call
+ */
+static inline void nfs4_exclusive_attrset(struct nfs4_opendata *opendata, struct iattr *sattr)
+{
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_ACCESS) &&
+	    !(sattr->ia_valid & ATTR_ATIME_SET))
+		sattr->ia_valid |= ATTR_ATIME;
+
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_MODIFY) &&
+	    !(sattr->ia_valid & ATTR_MTIME_SET))
+		sattr->ia_valid |= ATTR_MTIME;
+}
+
+/*
  * Returns a referenced nfs4_state
  */
 static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct iattr *sattr, struct rpc_cred *cred, struct nfs4_state **res)

@@ -973,6 +989,9 @@ static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct
 	if (status != 0)
 		goto err_opendata_free;
 
+	if (opendata->o_arg.open_flags & O_EXCL)
+		nfs4_exclusive_attrset(opendata, sattr);
+
 	status = -ENOMEM;
 	state = nfs4_opendata_to_nfs4_state(opendata);
 	if (state == NULL)

@@ -1784,6 +1803,7 @@ nfs4_proc_create(struct inode *dir, struct dentry *dentry, struct iattr *sattr,
 		status = nfs4_do_setattr(state->inode, &fattr, sattr, state);
 		if (status == 0)
 			nfs_setattr_update_inode(state->inode, sattr);
+		nfs_post_op_update_inode(state->inode, &fattr);
 	}
 	if (status == 0 && (nd->flags & LOOKUP_OPEN) != 0)
 		status = nfs4_intent_set_file(nd, &path, state);

fs/nfs/nfs4xdr.c +7 -2

index 8003c91..5efd314 100644
@@ -3269,7 +3269,7 @@ static int decode_delegation(struct xdr_stream *xdr, struct nfs_openres *res)
 static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
 {
         __be32 *p;
-        uint32_t bmlen;
+        uint32_t savewords, bmlen, i;
         int status;
 
         status = decode_op_hdr(xdr, OP_OPEN);

@@ -3287,7 +3287,12 @@ static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
                 goto xdr_error;
 
         READ_BUF(bmlen << 2);
-        p += bmlen;
+	savewords = min_t(uint32_t, bmlen, NFS4_BITMAP_SIZE);
+	for (i = 0; i < savewords; ++i)
+		READ32(res->attrset[i]);
+
+	p += (bmlen - savewords);
+
 	return decode_delegation(xdr, res);
 xdr_error:
 	dprintk("%s: Bitmap too large! Length = %u\n", __FUNCTION__, bmlen);

include/linux/nfs4.h +1 -0

index 7e7f33a..8726491 100644
@@ -15,6 +15,7 @@
 
 #include <linux/types.h>
 
+#define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
 #define NFS4_STATEID_SIZE	16
 #define NFS4_FHSIZE		128

include/linux/nfs_xdr.h +1 -0

index 10c26ed..f7100df 100644
@@ -144,6 +144,7 @@ struct nfs_openres {
 	nfs4_stateid		delegation;
 	__u32			do_recall;
 	__u64			maxsize;
+	__u32			attrset[NFS4_BITMAP_SIZE];
 };
 
 /*

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

More precisely, applying this patch to the 2.6.20.14 kernel plus the revalidate-the-fsid patch did not result in any changes. Does the O_EXCL patch require some other supporting patches relative to 2.6.20.14?

or perhaps it is necessary to rebase to a newer kernel before applying it? even tough it applies, builds and runs with 2.6.20.14?

Thanks.

Quoted 154 lines

Unfortunately this patch did not have any observable affects.

Quoted 146 lines · Trond Myklebust

Quoted 9 lines · Stuart Anderson

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

Quoted 5 lines

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Doesn't Jeff's patch fix it?

Trond

From: Jeff Layton <[email protected]>
Date: Tue, 5 Jun 2007 14:49:03 -0400
NFS4: on a O_EXCL OPEN make sure SETATTR sets the fields holding the
verifier
Subject: No Subject

The Linux NFS4 client simply skips over the bitmask in an O_EXCL open call and so it doesn't bother to reset any fields that may be holding the verifier. This patch has us save the first two words of the bitmask (which is all the current client has #defines for). The client then later checks this bitmask and turns on the appropriate flags in the sattr->ia_verify field for the following SETATTR call.

This patch only currently checks to see if the server used the atime and mtime slots for the verifier (which is what the Linux server uses for this). I'm not sure of what other fields the server could reasonably use, but adding checks for others should be trivial.

Signed-off-by Jeff Layton <[email protected]>

Signed-off-by Trond Myklebust <[email protected]>

---

 fs/nfs/nfs4proc.c       |   20 ++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |    9 +++++++--
 include/linux/nfs4.h    |    1 +
 include/linux/nfs_xdr.h |    1 +
 4 files changed, 29 insertions(+), 2 deletions(-)

fs/nfs/nfs4proc.c +20 -0

index 3cc7544..fee2d14 100644
@@ -943,6 +943,22 @@ static struct nfs4_state *nfs4_open_delegated(struct inode *inode, int flags, st
 }
 
 /*
+ * on an EXCLUSIVE create, the server should send back a bitmask with FATTR4-*
+ * fields corresponding to attributes that were used to store the verifier.
+ * Make sure we clobber those fields in the later setattr call
+ */
+static inline void nfs4_exclusive_attrset(struct nfs4_opendata *opendata, struct iattr *sattr)
+{
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_ACCESS) &&
+	    !(sattr->ia_valid & ATTR_ATIME_SET))
+		sattr->ia_valid |= ATTR_ATIME;
+
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_MODIFY) &&
+	    !(sattr->ia_valid & ATTR_MTIME_SET))
+		sattr->ia_valid |= ATTR_MTIME;
+}
+
+/*
  * Returns a referenced nfs4_state
  */
 static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct iattr *sattr, struct rpc_cred *cred, struct nfs4_state **res)

@@ -973,6 +989,9 @@ static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct
 	if (status != 0)
 		goto err_opendata_free;
 
+	if (opendata->o_arg.open_flags & O_EXCL)
+		nfs4_exclusive_attrset(opendata, sattr);
+
 	status = -ENOMEM;
 	state = nfs4_opendata_to_nfs4_state(opendata);
 	if (state == NULL)

@@ -1784,6 +1803,7 @@ nfs4_proc_create(struct inode *dir, struct dentry *dentry, struct iattr *sattr,
 		status = nfs4_do_setattr(state->inode, &fattr, sattr, state);
 		if (status == 0)
 			nfs_setattr_update_inode(state->inode, sattr);
+		nfs_post_op_update_inode(state->inode, &fattr);
 	}
 	if (status == 0 && (nd->flags & LOOKUP_OPEN) != 0)
 		status = nfs4_intent_set_file(nd, &path, state);

fs/nfs/nfs4xdr.c +7 -2

index 8003c91..5efd314 100644
@@ -3269,7 +3269,7 @@ static int decode_delegation(struct xdr_stream *xdr, struct nfs_openres *res)
 static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
 {
         __be32 *p;
-        uint32_t bmlen;
+        uint32_t savewords, bmlen, i;
         int status;
 
         status = decode_op_hdr(xdr, OP_OPEN);

@@ -3287,7 +3287,12 @@ static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
                 goto xdr_error;
 
         READ_BUF(bmlen << 2);
-        p += bmlen;
+	savewords = min_t(uint32_t, bmlen, NFS4_BITMAP_SIZE);
+	for (i = 0; i < savewords; ++i)
+		READ32(res->attrset[i]);
+
+	p += (bmlen - savewords);
+
 	return decode_delegation(xdr, res);
 xdr_error:
 	dprintk("%s: Bitmap too large! Length = %u\n", __FUNCTION__, bmlen);

include/linux/nfs4.h +1 -0

index 7e7f33a..8726491 100644
@@ -15,6 +15,7 @@
 
 #include <linux/types.h>
 
+#define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
 #define NFS4_STATEID_SIZE	16
 #define NFS4_FHSIZE		128

include/linux/nfs_xdr.h +1 -0

index 10c26ed..f7100df 100644
@@ -144,6 +144,7 @@ struct nfs_openres {
 	nfs4_stateid		delegation;
 	__u32			do_recall;
 	__u64			maxsize;
+	__u32			attrset[NFS4_BITMAP_SIZE];
 };
 
 /*

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Quoted 7 lines · On Wed, 27 Jun 2007 17:59:57 -0700 Stuart Anderson <[email protected]>…

More precisely, applying this patch to the 2.6.20.14 kernel plus the revalidate-the-fsid patch did not result in any changes. Does the O_EXCL patch require some other supporting patches relative to 2.6.20.14?

or perhaps it is necessary to rebase to a newer kernel before applying it? even tough it applies, builds and runs with 2.6.20.14?

Quoted 2 lines

Thanks.

With Linux, both client and server side were broken in this respect. The server didn't set the bitmask in the reply and the client didn't look at it anyway. It's possible that Solaris server is broken in this regard as well. It would be interesting to see a capture here, particularly one containing the reply from the CREATE.

Exactly what version of Solaris is the server running?

Quoted 161 lines · Stuart Anderson

Quoted 154 lines

Unfortunately this patch did not have any observable affects.

Quoted 146 lines · Trond Myklebust

Quoted 9 lines · Stuart Anderson

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

Quoted 5 lines

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Doesn't Jeff's patch fix it?

Trond

From: Jeff Layton <[email protected]>
Date: Tue, 5 Jun 2007 14:49:03 -0400
NFS4: on a O_EXCL OPEN make sure SETATTR sets the fields holding the
verifier
Subject: No Subject

The Linux NFS4 client simply skips over the bitmask in an O_EXCL open call and so it doesn't bother to reset any fields that may be holding the verifier. This patch has us save the first two words of the bitmask (which is all the current client has #defines for). The client then later checks this bitmask and turns on the appropriate flags in the sattr->ia_verify field for the following SETATTR call.

This patch only currently checks to see if the server used the atime and mtime slots for the verifier (which is what the Linux server uses for this). I'm not sure of what other fields the server could reasonably use, but adding checks for others should be trivial.

Signed-off-by Jeff Layton <[email protected]>

Signed-off-by Trond Myklebust <[email protected]>

---

 fs/nfs/nfs4proc.c       |   20 ++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |    9 +++++++--
 include/linux/nfs4.h    |    1 +
 include/linux/nfs_xdr.h |    1 +
 4 files changed, 29 insertions(+), 2 deletions(-)

fs/nfs/nfs4proc.c +20 -0

index 3cc7544..fee2d14 100644
@@ -943,6 +943,22 @@ static struct nfs4_state *nfs4_open_delegated(struct inode *inode, int flags, st
 }
 
 /*
+ * on an EXCLUSIVE create, the server should send back a bitmask with FATTR4-*
+ * fields corresponding to attributes that were used to store the verifier.
+ * Make sure we clobber those fields in the later setattr call
+ */
+static inline void nfs4_exclusive_attrset(struct nfs4_opendata *opendata, struct iattr *sattr)
+{
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_ACCESS) &&
+	    !(sattr->ia_valid & ATTR_ATIME_SET))
+		sattr->ia_valid |= ATTR_ATIME;
+
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_MODIFY) &&
+	    !(sattr->ia_valid & ATTR_MTIME_SET))
+		sattr->ia_valid |= ATTR_MTIME;
+}
+
+/*
  * Returns a referenced nfs4_state
  */
 static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct iattr *sattr, struct rpc_cred *cred, struct nfs4_state **res)

@@ -973,6 +989,9 @@ static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct
 	if (status != 0)
 		goto err_opendata_free;
 
+	if (opendata->o_arg.open_flags & O_EXCL)
+		nfs4_exclusive_attrset(opendata, sattr);
+
 	status = -ENOMEM;
 	state = nfs4_opendata_to_nfs4_state(opendata);
 	if (state == NULL)

@@ -1784,6 +1803,7 @@ nfs4_proc_create(struct inode *dir, struct dentry *dentry, struct iattr *sattr,
 		status = nfs4_do_setattr(state->inode, &fattr, sattr, state);
 		if (status == 0)
 			nfs_setattr_update_inode(state->inode, sattr);
+		nfs_post_op_update_inode(state->inode, &fattr);
 	}
 	if (status == 0 && (nd->flags & LOOKUP_OPEN) != 0)
 		status = nfs4_intent_set_file(nd, &path, state);

fs/nfs/nfs4xdr.c +7 -2

index 8003c91..5efd314 100644
@@ -3269,7 +3269,7 @@ static int decode_delegation(struct xdr_stream *xdr, struct nfs_openres *res)
 static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
 {
         __be32 *p;
-        uint32_t bmlen;
+        uint32_t savewords, bmlen, i;
         int status;
 
         status = decode_op_hdr(xdr, OP_OPEN);

@@ -3287,7 +3287,12 @@ static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
                 goto xdr_error;
 
         READ_BUF(bmlen << 2);
-        p += bmlen;
+	savewords = min_t(uint32_t, bmlen, NFS4_BITMAP_SIZE);
+	for (i = 0; i < savewords; ++i)
+		READ32(res->attrset[i]);
+
+	p += (bmlen - savewords);
+
 	return decode_delegation(xdr, res);
 xdr_error:
 	dprintk("%s: Bitmap too large! Length = %u\n", __FUNCTION__, bmlen);

include/linux/nfs4.h +1 -0

index 7e7f33a..8726491 100644
@@ -15,6 +15,7 @@
 
 #include <linux/types.h>
 
+#define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
 #define NFS4_STATEID_SIZE	16
 #define NFS4_FHSIZE		128

include/linux/nfs_xdr.h +1 -0

index 10c26ed..f7100df 100644
@@ -144,6 +144,7 @@ struct nfs_openres {
 	nfs4_stateid		delegation;
 	__u32			do_recall;
 	__u64			maxsize;
+	__u32			attrset[NFS4_BITMAP_SIZE];
 };
 
 /*

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Signature · 1 line

Jeff Layton <[email protected]>

Quoted 15 lines · Jeff Layton

Quoted 11 lines · On Wed, 27 Jun 2007 17:59:57 -0700 Stuart Anderson <[email protected]>…

More precisely, applying this patch to the 2.6.20.14 kernel plus the revalidate-the-fsid patch did not result in any changes. Does the O_EXCL patch require some other supporting patches relative to 2.6.20.14?

or perhaps it is necessary to rebase to a newer kernel before applying it?
even tough it applies, builds and runs with 2.6.20.14?

Thanks.

Quoted 6 lines

With Linux, both client and server side were broken in this respect. The server didn't set the bitmask in the reply and the client didn't look at it anyway. It's possible that Solaris server is broken in this
regard as well. It would be interesting to see a capture here, particularly one containing the reply from the CREATE.

Possible, but when we switch the client from Linux to Solaris and share
the same filesytem from the same server there are no O_EXCL problems.

I will see if we can capture the bits. Any recomendations on which tool
with what options would be most useful to capture the useful traffic?

Quoted 2 lines

Exactly what version of Solaris is the server running?

# cat /etc/release
                        Solaris 10 11/06 s10x_u3wos_10 X86
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006
# uname -v
Generic_118855-36

Somewhat related is the reason that we are tring NFSV4 on Linux clients is
that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem
on the Solaris server is ZFS, e.g., cp -p will generate interesting error
messages. While I would rather stay with NFSV4 if possible, has anyone gotten this configuration to work with V3?

Quoted 8 lines

Somewhat related is the reason that we are tring NFSV4 on Linux clients is
that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem
on the Solaris server is ZFS, e.g., cp -p will generate interesting error
messages. While I would rather stay with NFSV4 if possible, has anyone gotten this configuration to work with V3?

It will not. ZFS does not support the "old" ACL definition; only NFSv4 support.

Quoted 10 lines · Spencer Shepler

Quoted 9 lines

Somewhat related is the reason that we are tring NFSV4 on Linux clients is
that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem
on the Solaris server is ZFS, e.g., cp -p will generate interesting error
messages. While I would rather stay with NFSV4 if possible, has anyone
gotten this configuration to work with V3?

Quoted 3 lines

It will not. ZFS does not support the "old" ACL definition; only NFSv4 support.

Understood, so this is probably not a bug and just a feature, but why does a Linux client try and then generate application level errors? A Solaris client is willing to NFS mount vers=3 a ZFS filesystem and then
run GNU cp -rp, for example, without complaint. Perhaps there is a simple
method determine if ACL's are available/supported on a particular NFS mount?

Quoted 15 lines · Stuart Anderson

Quoted 10 lines · Spencer Shepler

Quoted 9 lines

Somewhat related is the reason that we are tring NFSV4 on Linux clients is
that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem
on the Solaris server is ZFS, e.g., cp -p will generate interesting error
messages. While I would rather stay with NFSV4 if possible, has anyone
gotten this configuration to work with V3?

Quoted 3 lines

It will not. ZFS does not support the "old" ACL definition; only NFSv4 support.

Quoted 8 lines

Understood, so this is probably not a bug and just a feature, but why does a Linux client try and then generate application level errors? A Solaris client is willing to NFS mount vers=3 a ZFS filesystem and then
run GNU cp -rp, for example, without complaint. Perhaps there is a simple
method determine if ACL's are available/supported on a particular NFS mount?

Does it fail on the "get" or "set" of the ACL?

Quoted 30 lines · Spencer Shepler

Quoted 16 lines · Stuart Anderson

Quoted 11 lines · Spencer Shepler

Quoted 10 lines

Somewhat related is the reason that we are tring NFSV4 on Linux clients is
that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem
on the Solaris server is ZFS, e.g., cp -p will generate interesting
error
messages. While I would rather stay with NFSV4 if possible, has anyone
gotten this configuration to work with V3?

Quoted 3 lines

It will not. ZFS does not support the "old" ACL definition; only NFSv4 support.

Quoted 10 lines

Understood, so this is probably not a bug and just a feature, but why
does a Linux client try and then generate application level errors? A Solaris client is willing to NFS mount vers=3 a ZFS filesystem and then
run GNU cp -rp, for example, without complaint. Perhaps there is a simple
method determine if ACL's are available/supported on a particular NFS
mount?

Quoted 1 line

Does it fail on the "get" or "set" of the ACL?

set

Here is an strace snippet from running cp -rp on a Linux NFSV3 client of a Solaris 10 ZFS server:

getxattr("test1/oink", "system.posix_acl_access", "\x02\x00\x00\x00 \x01\x00\x06\x00\xff\xff\xff\xff\x04\x00\x04\x00\xff\xff\xff\xff\x10 \x00\x07\x00\xff\xff\xff\xff \x00\x04\x00\xff\xff\xff\xff", 132) = 36 setxattr("test2/oink", "system.posix_acl_access", "\x02\x00\x00\x00 \x01\x00\x06\x00\xff\xff\xff\xff\x04\x00\x04\x00\xff\xff\xff\xff\x10 \x00\x07\x00\xff\xff\xff\xff \x00\x04\x00\xff\xff\xff\xff", 36, 0) = -1 EINVAL (Invalid argument)

We used to have this same problem with Linux NFSV3 mounts from Solaris QFS servers
as well, but that was fixed in a recent update to QFS, so I am not really sure
who is "responsible" for bridging ACL's between the old/new schemes.

The Solaris NFSv3 server will, upon "get acl", map the underlying ZFS ACL format to the NFSv3/ACL format. However, on setting of an acl, an error is returned (NOSUP). The decision was made that because of the imperfect mapping on acl set in this path that an error would be returned.

I believe the Solaris client would map the ENOTSUP back to the application as is and that may help the application adapt its behavior (avoid erroring out if that isn't desired).

Quoted 13 lines · Jeff Layton

Quoted 9 lines · On Wed, 27 Jun 2007 17:59:57 -0700 Stuart Anderson <[email protected]>…

More precisely, applying this patch to the 2.6.20.14 kernel plus the revalidate-the-fsid patch did not result in any changes. Does the O_EXCL patch require some other supporting patches relative to 2.6.20.14?

or perhaps it is necessary to rebase to a newer kernel before applying it? even tough it applies, builds and runs with 2.6.20.14?

Thanks.

Quoted 5 lines

With Linux, both client and server side were broken in this respect. The server didn't set the bitmask in the reply and the client didn't look at it anyway. It's possible that Solaris server is broken in this regard as well. It would be interesting to see a capture here, particularly one containing the reply from the CREATE.

Possible, but when we switch the client from Linux to Solaris and share the same filesytem from the same server there are no O_EXCL problems.

I will see if we can capture the bits. Any recomendations on which tool with what options would be most useful to capture the useful traffic?

Quoted 2 lines

Exactly what version of Solaris is the server running?

# cat /etc/release 
                        Solaris 10 11/06 s10x_u3wos_10 X86
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006
# uname -v
Generic_118855-36

Somewhat related is the reason that we are tring NFSV4 on Linux clients is that with NFSV3 ACL's do not work between Linux and Solaris if the filesystem on the Solaris server is ZFS, e.g., cp -p will generate interesting error messages. While I would rather stay with NFSV4 if possible, has anyone gotten this configuration to work with V3?

Quoted 165 lines

Quoted 161 lines · Stuart Anderson

Quoted 154 lines

Unfortunately this patch did not have any observable affects.

Quoted 146 lines · Trond Myklebust

Quoted 9 lines · Stuart Anderson

The following simple program creates files with un-initialized mtime values on a Linux NFSv4 client mounting from a Solaris NFSv4 server--at least the mtimes are wildly different each time the program is run. The problem is reproducible but does not happen under any of the following circumstances:

1) Drop O_EXCL from open() call.
2) NFS mount using v3.
3) Switch NFS v4 client from Linux to Solaris.

Quoted 5 lines

This is on an FC4 machine with the 2.6.20.14 kernel plus Trond's recent revalidate-the-fsid-on-the-current-dir-not-the-root-dir patch and nfs-utils 1.0.9-16 backported from CentOS 5.

Thanks.

Doesn't Jeff's patch fix it?

Trond

From: Jeff Layton <[email protected]>
Date: Tue, 5 Jun 2007 14:49:03 -0400
NFS4: on a O_EXCL OPEN make sure SETATTR sets the fields holding the
verifier
Subject: No Subject

The Linux NFS4 client simply skips over the bitmask in an O_EXCL open call and so it doesn't bother to reset any fields that may be holding the verifier. This patch has us save the first two words of the bitmask (which is all the current client has #defines for). The client then later checks this bitmask and turns on the appropriate flags in the sattr->ia_verify field for the following SETATTR call.

This patch only currently checks to see if the server used the atime and mtime slots for the verifier (which is what the Linux server uses for this). I'm not sure of what other fields the server could reasonably use, but adding checks for others should be trivial.

Signed-off-by Jeff Layton <[email protected]>

Signed-off-by Trond Myklebust <[email protected]>

---

 fs/nfs/nfs4proc.c       |   20 ++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |    9 +++++++--
 include/linux/nfs4.h    |    1 +
 include/linux/nfs_xdr.h |    1 +
 4 files changed, 29 insertions(+), 2 deletions(-)

fs/nfs/nfs4proc.c +20 -0

index 3cc7544..fee2d14 100644
@@ -943,6 +943,22 @@ static struct nfs4_state *nfs4_open_delegated(struct inode *inode, int flags, st
 }
 
 /*
+ * on an EXCLUSIVE create, the server should send back a bitmask with FATTR4-*
+ * fields corresponding to attributes that were used to store the verifier.
+ * Make sure we clobber those fields in the later setattr call
+ */
+static inline void nfs4_exclusive_attrset(struct nfs4_opendata *opendata, struct iattr *sattr)
+{
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_ACCESS) &&
+	    !(sattr->ia_valid & ATTR_ATIME_SET))
+		sattr->ia_valid |= ATTR_ATIME;
+
+	if ((opendata->o_res.attrset[1] & FATTR4_WORD1_TIME_MODIFY) &&
+	    !(sattr->ia_valid & ATTR_MTIME_SET))
+		sattr->ia_valid |= ATTR_MTIME;
+}
+
+/*
  * Returns a referenced nfs4_state
  */
 static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct iattr *sattr, struct rpc_cred *cred, struct nfs4_state **res)

@@ -973,6 +989,9 @@ static int _nfs4_do_open(struct inode *dir, struct path *path, int flags, struct
 	if (status != 0)
 		goto err_opendata_free;
 
+	if (opendata->o_arg.open_flags & O_EXCL)
+		nfs4_exclusive_attrset(opendata, sattr);
+
 	status = -ENOMEM;
 	state = nfs4_opendata_to_nfs4_state(opendata);
 	if (state == NULL)

@@ -1784,6 +1803,7 @@ nfs4_proc_create(struct inode *dir, struct dentry *dentry, struct iattr *sattr,
 		status = nfs4_do_setattr(state->inode, &fattr, sattr, state);
 		if (status == 0)
 			nfs_setattr_update_inode(state->inode, sattr);
+		nfs_post_op_update_inode(state->inode, &fattr);
 	}
 	if (status == 0 && (nd->flags & LOOKUP_OPEN) != 0)
 		status = nfs4_intent_set_file(nd, &path, state);

fs/nfs/nfs4xdr.c +7 -2

index 8003c91..5efd314 100644
@@ -3269,7 +3269,7 @@ static int decode_delegation(struct xdr_stream *xdr, struct nfs_openres *res)
 static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
 {
         __be32 *p;
-        uint32_t bmlen;
+        uint32_t savewords, bmlen, i;
         int status;
 
         status = decode_op_hdr(xdr, OP_OPEN);

@@ -3287,7 +3287,12 @@ static int decode_open(struct xdr_stream *xdr, struct nfs_openres *res)
                 goto xdr_error;
 
         READ_BUF(bmlen << 2);
-        p += bmlen;
+	savewords = min_t(uint32_t, bmlen, NFS4_BITMAP_SIZE);
+	for (i = 0; i < savewords; ++i)
+		READ32(res->attrset[i]);
+
+	p += (bmlen - savewords);
+
 	return decode_delegation(xdr, res);
 xdr_error:
 	dprintk("%s: Bitmap too large! Length = %u\n", __FUNCTION__, bmlen);

include/linux/nfs4.h +1 -0

index 7e7f33a..8726491 100644
@@ -15,6 +15,7 @@
 
 #include <linux/types.h>
 
+#define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
 #define NFS4_STATEID_SIZE	16
 #define NFS4_FHSIZE		128

include/linux/nfs_xdr.h +1 -0

index 10c26ed..f7100df 100644
@@ -144,6 +144,7 @@ struct nfs_openres {
 	nfs4_stateid		delegation;
 	__u32			do_recall;
 	__u64			maxsize;
+	__u32			attrset[NFS4_BITMAP_SIZE];
 };
 
 /*

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Signature · 1 line

Jeff Layton <[email protected]>

Signature · 2 lines

Stuart Anderson  [email protected]
http://www.ligo.caltech.edu/~anderson

Quoted 20 lines · On Wed, 27 Jun 2007 19:53:27 -0700 Stuart Anderson <[email protected]>…

Quoted 18 lines · Jeff Layton

Quoted 9 lines · On Wed, 27 Jun 2007 17:59:57 -0700 Stuart Anderson <[email protected]>…

More precisely, applying this patch to the 2.6.20.14 kernel plus the revalidate-the-fsid patch did not result in any changes. Does the O_EXCL patch require some other supporting patches relative to 2.6.20.14?

or perhaps it is necessary to rebase to a newer kernel before applying it? even tough it applies, builds and runs with 2.6.20.14?

Thanks.

With Linux, both client and server side were broken in this respect. The server didn't set the bitmask in the reply and the client didn't look at it anyway. It's possible that Solaris server is broken in this regard as well. It would be interesting to see a capture here, particularly one containing the reply from the CREATE.

Quoted 3 lines

Possible, but when we switch the client from Linux to Solaris and share the same filesytem from the same server there are no O_EXCL problems.

NFSv3 didn't use this bitmask, so clients tended to just clobber the mtime and atime on the subsequent setattr. It's possible that Solaris is using the old semantics here with NFSv4. Of course, all of this is speculation :-), a capture should give us a better clue.

Yes, it looks like Solaris is sending back a zeroed out attrmask on exclusive create. I'm testing against:

Solaris Nevada snv_54 X86
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 04 December 2006

SunOS solaris 5.11 snv_54 i86pc i386 i86pc

My guess is that they're using NFSv3 semantics in their NFSv4 client to set the mtime and atime, and that's why it "works" there. This patchrev is pretty old by now, so they may have already patched this issue. I'll see if I can patch it and test again.

All this, of course, is contingent upon my having interpreted the spec correctly. I think I have, but wouldn't mind if someone sanity checked me on it.

Quoted 10 lines · Jeff Layton

Yes, it looks like Solaris is sending back a zeroed out attrmask on exclusive create. I'm testing against:

Solaris Nevada snv_54 X86
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 04 December 2006

SunOS solaris 5.11 snv_54 i86pc i386 i86pc

Quoted 8 lines

My guess is that they're using NFSv3 semantics in their NFSv4 client to set the mtime and atime, and that's why it "works" there. This patchrev is pretty old by now, so they may have already patched this issue. I'll see if I can patch it and test again.

All this, of course, is contingent upon my having interpreted the spec correctly. I think I have, but wouldn't mind if someone sanity checked me on it.

The only nit I can see is that the server is in theory allowed to store the cookie in some writable attribute other than atime/mtime, but I can't think of that many that would make sense. Perhaps time_backup and time_create for those servers that support them? However since we don't care about their values and since they don't support a SET_TO_SERVER_TIME4 option, I'm not sure we want to bother...

Trond