Skip to contentSkip to conversation

RFC: pnfs: LAYOUTGET/DEVINFO submission, v2

PATCH In review Fred Isaman 3 participants 31 messages Sep 18, 2010 – Sep 20, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover RFC: pnfs: LAYOUTGET/DEVINFO submission, v2

This fixes various issue brought up in comments to the previous version.

Among the more substantial changes:

find_pnfs_driver now grabs module reference
driver init/uninit_mountpoint argument changed to struct nfs_server io_ops moved into pnfs_layoutdriver_type
handling of pnfs_layout_from_open_stateid changed
file driver no longer uses vmalloc
deviceid moved from general to driver layer CONFIG_PNFS_FILE_LAYOUT removed, just uses CONFIG_NFS_V4_1 error handling of LAYOUTGET and GETDEVICEINFO rpc calls changed

For those interested, the patches can also be found at
git://linux-nfs.org/~iisaman/linux-pnfs.git,
under the branch devinfo-submit-rc4-trond

A rebase of Benny's pnfs-submit branch onto these patches is under the branch bennys-pnfs-submit-on-devinfo-rc4

Fred

This is the start of code implementing pnfs, based on RFC 5661. Since sending the whole thing at once would be overwhelming, we are trying to break it into bite sized chunks. This chunk implements the mount/umount infrastructure, as well as sending the LAYOUTGET and GETDEVTICEINFO calls on io (but not actually using the information for io). Note that two major simplifications to the protocol will be made throughout the initial submission process: only the file layout driver is considered, and only whole file layouts are requested.

These patches apply against Trond's nfs-for-2.6.37 branch.

patches 01-08 implement the mount/umount hooks patches 09-12 implement LAYOUTGET and GETDEVICEINFO

#1 lore raw

PATCH 1/12 NFSD: remove duplicate NFS4_STATEID_SIZE

From: Andy Adamson <[email protected]>

Already accepted by Bruce

---
 fs/nfsd/nfs4callback.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
fs/nfsd/nfs4callback.c +0 -1
index 988cbb3..014482c 100644
@@ -41,7 +41,6 @@
 
 #define NFSPROC4_CB_NULL 0
 #define NFSPROC4_CB_COMPOUND 1
-#define NFS4_STATEID_SIZE 16
 
 /* Index of predefined Linux callback client operations */
 
Signature · 1 line
1.7.2.1
#2 lore raw

PATCH 2/12 SUNRPC: define xdr_decode_opaque_fixed

In reply to Fred Isaman

From: Benny Halevy <[email protected]>

A helper for decoding a fixed length opaque value. Returns a pointer to the next item in the xdr stream.

---
 include/linux/sunrpc/xdr.h |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
include/linux/sunrpc/xdr.h +7 -0
index 35cf2e8..23dc117 100644
@@ -131,6 +131,13 @@ xdr_decode_hyper(__be32 *p, __u64 *valp)
 	return p + 2;
 }
 
+static inline __be32 *
+xdr_decode_opaque_fixed(__be32 *p, void *ptr, unsigned int len)
+{
+	memcpy(ptr, p, len);
+	return p + XDR_QUADLEN(len);
+}
+
 /*
  * Adjust kvec to reflect end of xdr'ed data (RPC client XDR)
  */
Signature · 1 line
1.7.2.1
#3 lore raw

PATCH 3/12 RFC: pnfsd, pnfs: protocol level pnfs constants

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

Use only layoutreturn constant for both returns and recalls. (return_* works better for recall_type rather the other way around)

Signed-off-by TBD - melding/reorganization of several patches
---
 include/linux/nfs4.h |   39 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 39 insertions(+), 0 deletions(-)
include/linux/nfs4.h +39 -0
index 07e40c6..6798fc3 100644
@@ -471,6 +471,8 @@ enum lock_type4 {
 #define FATTR4_WORD1_TIME_MODIFY        (1UL << 21)
 #define FATTR4_WORD1_TIME_MODIFY_SET    (1UL << 22)
 #define FATTR4_WORD1_MOUNTED_ON_FILEID  (1UL << 23)
+#define FATTR4_WORD1_FS_LAYOUT_TYPES    (1UL << 30)
+#define FATTR4_WORD2_LAYOUT_BLKSIZE     (1UL << 1)
 
 #define NFSPROC4_NULL 0
 #define NFSPROC4_COMPOUND 1
@@ -550,6 +552,43 @@ enum state_protect_how4 {
 	SP4_SSV		= 2
 };
 
+enum pnfs_layouttype {
+	LAYOUT_NFSV4_1_FILES  = 1,
+	LAYOUT_OSD2_OBJECTS = 2,
+	LAYOUT_BLOCK_VOLUME = 3,
+};
+
+/* used for both layout return and recall */
+enum pnfs_layoutreturn_type {
+	RETURN_FILE = 1,
+	RETURN_FSID = 2,
+	RETURN_ALL  = 3
+};
+
+enum pnfs_iomode {
+	IOMODE_READ = 1,
+	IOMODE_RW = 2,
+	IOMODE_ANY = 3,
+};
+
+enum pnfs_notify_deviceid_type4 {
+	NOTIFY_DEVICEID4_CHANGE = 1 << 1,
+	NOTIFY_DEVICEID4_DELETE = 1 << 2,
+};
+
+#define NFL4_UFLG_MASK			0x0000003F
+#define NFL4_UFLG_DENSE			0x00000001
+#define NFL4_UFLG_COMMIT_THRU_MDS	0x00000002
+#define NFL4_UFLG_STRIPE_UNIT_SIZE_MASK	0xFFFFFFC0
+
+/* Encoded in the loh_body field of type layouthint4 */
+enum filelayout_hint_care4 {
+	NFLH4_CARE_DENSE		= NFL4_UFLG_DENSE,
+	NFLH4_CARE_COMMIT_THRU_MDS	= NFL4_UFLG_COMMIT_THRU_MDS,
+	NFLH4_CARE_STRIPE_UNIT_SIZE	= 0x00000040,
+	NFLH4_CARE_STRIPE_COUNT		= 0x00000080
+};
+
 #endif
 #endif
 
Signature · 1 line
1.7.2.1
#4 lore raw

PATCH 4/12 RFC: nfs: change stateid to be a union

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

In NFSv4.1 the stateid consists of the other and seqid fields. For layout processing we need to numerically compare the seqid value of layout stateids. To do so, introduce a union to nfs4_stateid to swtich between opaque(16 bytes) and opaque(12 bytes) / __be32

---
 fs/nfs/callback_proc.c |    8 ++++----
 include/linux/nfs4.h   |   15 +++++++++++++--
 2 files changed, 17 insertions(+), 6 deletions(-)
fs/nfs/callback_proc.c +4 -4
index 930d10f..2950fca 100644
@@ -118,11 +118,11 @@ int nfs41_validate_delegation_stateid(struct nfs_delegation *delegation, const n
 	if (delegation == NULL)
 		return 0;
 
-	/* seqid is 4-bytes long */
-	if (((u32 *) &stateid->data)[0] != 0)
+	if (stateid->stateid.seqid != 0)
 		return 0;
-	if (memcmp(&delegation->stateid.data[4], &stateid->data[4],
-		   sizeof(stateid->data)-4))
+	if (memcmp(&delegation->stateid.stateid.other,
+		   &stateid->stateid.other,
+		   NFS4_STATEID_OTHER_SIZE))
 		return 0;
 
 	return 1;
include/linux/nfs4.h +13 -2
index 6798fc3..2dde7c8 100644
@@ -17,7 +17,9 @@
 
 #define NFS4_BITMAP_SIZE	2
 #define NFS4_VERIFIER_SIZE	8
-#define NFS4_STATEID_SIZE	16
+#define NFS4_STATEID_SEQID_SIZE 4
+#define NFS4_STATEID_OTHER_SIZE 12
+#define NFS4_STATEID_SIZE	(NFS4_STATEID_SEQID_SIZE + NFS4_STATEID_OTHER_SIZE)
 #define NFS4_FHSIZE		128
 #define NFS4_MAXPATHLEN		PATH_MAX
 #define NFS4_MAXNAMLEN		NAME_MAX
@@ -167,7 +169,16 @@ struct nfs4_acl {
 };
 
 typedef struct { char data[NFS4_VERIFIER_SIZE]; } nfs4_verifier;
-typedef struct { char data[NFS4_STATEID_SIZE]; } nfs4_stateid;
+
+struct nfs41_stateid {
+	__be32 seqid;
+	char other[NFS4_STATEID_OTHER_SIZE];
+} __attribute__ ((packed));
+
+typedef union {
+	char data[NFS4_STATEID_SIZE];
+	struct nfs41_stateid stateid;
+} nfs4_stateid;
 
 enum nfs_opnum4 {
 	OP_ACCESS = 3,
Signature · 1 line
1.7.2.1
#5 lore raw

PATCH 5/12 RFC: nfs: ask for layouttypes during fsinfo call

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

This information will be used to determine which layout driver, if any, to use for subsequent IO on this filesystem. Each driver is assigned an integer id, with 0 reserved to indicate no driver.

The server can in theory return multiple ids. However, our current client implementation only notes the first entry and ignores the rest.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/nfs4proc.c       |    2 +-
 fs/nfs/nfs4xdr.c        |   57 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfs_xdr.h |    1 +
 3 files changed, 59 insertions(+), 1 deletions(-)
fs/nfs/nfs4proc.c +1 -1
index c46e45e..c0d9ff4 100644
@@ -129,7 +129,7 @@ const u32 nfs4_fsinfo_bitmap[2] = { FATTR4_WORD0_MAXFILESIZE
 			| FATTR4_WORD0_MAXREAD
 			| FATTR4_WORD0_MAXWRITE
 			| FATTR4_WORD0_LEASE_TIME,
-			0
+			FATTR4_WORD1_FS_LAYOUT_TYPES
 };
 
 const u32 nfs4_fs_locations_bitmap[2] = {
fs/nfs/nfs4xdr.c +57 -0

69 lines not on this page · raw message

include/linux/nfs_xdr.h +1 -0
index 172df83..7d68a5c 100644
@@ -113,6 +113,7 @@ struct nfs_fsinfo {
 	__u32			dtpref;	/* pref. readdir transfer size */
 	__u64			maxfilesize;
 	__u32			lease_time; /* in seconds */
+	__u32			layouttype; /* supported pnfs layout driver */
 };
 
 struct nfs_fsstat {
Signature · 1 line
1.7.2.1
#6 lore raw

Re: [PATCH 05/12] RFC: nfs: ask for layouttypes during fsinfo call

Quoted 22 lines · Fred Isaman

From: The pNFS Team <[email protected]>

This information will be used to determine which layout driver, if any, to use for subsequent IO on this filesystem. Each driver is assigned an integer id, with 0 reserved to indicate no driver.

The server can in theory return multiple ids. However, our current client implementation only notes the first entry and ignores the rest.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/nfs4proc.c       |    2 +-
 fs/nfs/nfs4xdr.c        |   57 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfs_xdr.h |    1 +
 3 files changed, 59 insertions(+), 1 deletions(-)
fs/nfs/nfs4proc.c +0 -0
index c46e45e..c0d9ff4 100644

fs/nfs/nfs4proc.c const u32 nfs4_fsinfo_bitmap[2] = { FATTR4_WORD0_MAXFILESIZE · line 132

Quoted 5 lines
fs/nfs/nfs4proc.c +1 -1
@@ -129,7 +129,7 @@ const u32 nfs4_fsinfo_bitmap[2] = { FATTR4_WORD0_MAXFILESIZE
 			| FATTR4_WORD0_MAXREAD
 			| FATTR4_WORD0_MAXWRITE
 			| FATTR4_WORD0_LEASE_TIME,
-			0
+			FATTR4_WORD1_FS_LAYOUT_TYPES

Note that originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), we set this only under CONFIG_NFS_V4_1, but it should be ok to ask for it anyhow.

Benny

Quoted 90 lines
fs/nfs/nfs4proc.c +0 -0
 };
 
 const u32 nfs4_fs_locations_bitmap[2] = {
fs/nfs/nfs4xdr.c +57 -0
index b0bd7ef..e138237 100644
@@ -3868,6 +3868,60 @@ xdr_error:
 	return status;
 }
 
+/*
+ * Decode potentially multiple layout types. Currently we only support
+ * one layout driver per file system.
+ */
+static int decode_first_pnfs_layout_type(struct xdr_stream *xdr,
+					 uint32_t *layouttype)
+{
+	uint32_t *p;
+	int num;
+
+	p = xdr_inline_decode(xdr, 4);
+	if (unlikely(!p))
+		goto out_overflow;
+	num = be32_to_cpup(p);
+
+	/* pNFS is not supported by the underlying file system */
+	if (num == 0) {
+		*layouttype = 0;
+		return 0;
+	}
+	if (num > 1)
+		printk(KERN_INFO "%s: Warning: Multiple pNFS layout drivers "
+			"per filesystem not supported\n", __func__);
+
+	/* Decode and set first layout type, move xdr->p past unused types */
+	p = xdr_inline_decode(xdr, num * 4);
+	if (unlikely(!p))
+		goto out_overflow;
+	*layouttype = be32_to_cpup(p);
+	return 0;
+out_overflow:
+	print_overflow_msg(__func__, xdr);
+	return -EIO;
+}
+
+/*
+ * The type of file system exported.
+ * Note we must ensure that layouttype is set in any non-error case.
+ */
+static int decode_attr_pnfstype(struct xdr_stream *xdr, uint32_t *bitmap,
+				uint32_t *layouttype)
+{
+	int status = 0;
+
+	dprintk("%s: bitmap is %x\n", __func__, bitmap[1]);
+	if (unlikely(bitmap[1] & (FATTR4_WORD1_FS_LAYOUT_TYPES - 1U)))
+		return -EIO;
+	if (bitmap[1] & FATTR4_WORD1_FS_LAYOUT_TYPES) {
+		status = decode_first_pnfs_layout_type(xdr, layouttype);
+		bitmap[1] &= ~FATTR4_WORD1_FS_LAYOUT_TYPES;
+	} else
+		*layouttype = 0;
+	return status;
+}
 
 static int decode_fsinfo(struct xdr_stream *xdr, struct nfs_fsinfo *fsinfo)
 {
@@ -3894,6 +3948,9 @@ static int decode_fsinfo(struct xdr_stream *xdr, struct nfs_fsinfo *fsinfo)
 	if ((status = decode_attr_maxwrite(xdr, bitmap, &fsinfo->wtmax)) != 0)
 		goto xdr_error;
 	fsinfo->wtpref = fsinfo->wtmax;
+	status = decode_attr_pnfstype(xdr, bitmap, &fsinfo->layouttype);
+	if (status)
+		goto xdr_error;
 
 	status = verify_attr_len(xdr, savep, attrlen);
 xdr_error:
include/linux/nfs_xdr.h +1 -0
index 172df83..7d68a5c 100644
@@ -113,6 +113,7 @@ struct nfs_fsinfo {
 	__u32			dtpref;	/* pref. readdir transfer size */
 	__u64			maxfilesize;
 	__u32			lease_time; /* in seconds */
+	__u32			layouttype; /* supported pnfs layout driver */
 };
 
 struct nfs_fsstat {
#7 lore raw

Re: [PATCH 05/12] RFC: nfs: ask for layouttypes during fsinfo call

Quoted 33 lines · Benny Halevy
Quoted 25 lines · Fred Isaman

From: The pNFS Team <[email protected]>

This information will be used to determine which layout driver, if any, to use for subsequent IO on this filesystem.  Each driver is assigned an integer id, with 0 reserved to indicate no driver.

The server can in theory return multiple ids.  However, our current client implementation only notes the first entry and ignores the rest.

Signed-off-by TBD - melding/reorganization of several patches

---
 fs/nfs/nfs4proc.c       |    2 +-
 fs/nfs/nfs4xdr.c        |   57 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/nfs_xdr.h |    1 +

 3 files changed, 59 insertions(+), 1 deletions(-)
fs/nfs/nfs4proc.c +0 -0
index c46e45e..c0d9ff4 100644
@@ -129,7 +129,7 @@ const u32 nfs4_fsinfo_bitmap[2] = { FATTR4_WORD0_MAXFILESIZE
                      | FATTR4_WORD0_MAXREAD
                      | FATTR4_WORD0_MAXWRITE
                      | FATTR4_WORD0_LEASE_TIME,
Quoted 2 lines

-                     0
+                     FATTR4_WORD1_FS_LAYOUT_TYPES

Note that originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), we set this only under CONFIG_NFS_V4_1, but it should be ok to ask for it anyhow.

Quoted 2 lines

Benny

Yes.

Fred

Quoted 95 lines
Quoted 89 lines

 };

 const u32 nfs4_fs_locations_bitmap[2] = {

fs/nfs/nfs4xdr.c +57 -0
index b0bd7ef..e138237 100644
@@ -3868,6 +3868,60 @@ xdr_error:
       return status;
  }
 
+/*
+ * Decode potentially multiple layout types. Currently we only support
+ * one layout driver per file system.
+ */
+static int decode_first_pnfs_layout_type(struct xdr_stream *xdr,
+                                      uint32_t *layouttype)
+{
+     uint32_t *p;
+     int num;
+
+     p = xdr_inline_decode(xdr, 4);
+     if (unlikely(!p))
+             goto out_overflow;
+     num = be32_to_cpup(p);
+
+     /* pNFS is not supported by the underlying file system */
+     if (num == 0) {
+             *layouttype = 0;
+             return 0;
+     }
+     if (num > 1)
+             printk(KERN_INFO "%s: Warning: Multiple pNFS layout drivers "
+                     "per filesystem not supported\n", __func__);
+
+     /* Decode and set first layout type, move xdr->p past unused types */
+     p = xdr_inline_decode(xdr, num * 4);
+     if (unlikely(!p))
+             goto out_overflow;
+     *layouttype = be32_to_cpup(p);
+     return 0;
+out_overflow:
+     print_overflow_msg(__func__, xdr);
+     return -EIO;
+}
+
+/*
+ * The type of file system exported.
+ * Note we must ensure that layouttype is set in any non-error case.
+ */
+static int decode_attr_pnfstype(struct xdr_stream *xdr, uint32_t *bitmap,
+                             uint32_t *layouttype)
+{
+     int status = 0;
+
+     dprintk("%s: bitmap is %x\n", __func__, bitmap[1]);
+     if (unlikely(bitmap[1] & (FATTR4_WORD1_FS_LAYOUT_TYPES - 1U)))
+             return -EIO;
+     if (bitmap[1] & FATTR4_WORD1_FS_LAYOUT_TYPES) {
+             status = decode_first_pnfs_layout_type(xdr, layouttype);
+             bitmap[1] &= ~FATTR4_WORD1_FS_LAYOUT_TYPES;
+     } else
+             *layouttype = 0;
+     return status;
+}
 
  static int decode_fsinfo(struct xdr_stream *xdr, struct nfs_fsinfo *fsinfo)
  {
@@ -3894,6 +3948,9 @@ static int decode_fsinfo(struct xdr_stream *xdr, struct nfs_fsinfo *fsinfo)
       if ((status = decode_attr_maxwrite(xdr, bitmap, &fsinfo->wtmax)) != 0)
               goto xdr_error;
       fsinfo->wtpref = fsinfo->wtmax;
+     status = decode_attr_pnfstype(xdr, bitmap, &fsinfo->layouttype);
+     if (status)
+             goto xdr_error;
 
       status = verify_attr_len(xdr, savep, attrlen);

 xdr_error:

include/linux/nfs_xdr.h +0 -0
index 172df83..7d68a5c 100644
@@ -113,6 +113,7 @@ struct nfs_fsinfo {

      __u32                   dtpref; /* pref. readdir transfer size */ __u64                   maxfilesize;
      __u32                   lease_time; /* in seconds */
+     __u32                   layouttype; /* supported pnfs layout driver */ };

Quoted 1 line

 struct nfs_fsstat {

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#8 lore raw

PATCH 6/12 RFC: nfs: set layout driver

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 6 files changed, 141 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h
fs/nfs/Makefile +1 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)	+= nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
 			   delegation.o idmap.o \
 			   callback.o callback_xdr.o callback_proc.o \
 			   nfs4namespace.o
+nfs-$(CONFIG_NFS_V4_1)	+= pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
fs/nfs/client.c +4 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
 #include "iostat.h"
 #include "internal.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_CLIENT
 
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
 	if (server->wsize > NFS_MAX_FILE_IO_SIZE)
 		server->wsize = NFS_MAX_FILE_IO_SIZE;
 	server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+	set_pnfs_layoutdriver(server, fsinfo->layouttype);
+
 	server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
 	server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
 {
 	dprintk("--> nfs_free_server()\n");
 
+	unset_pnfs_layoutdriver(server);
 	spin_lock(&nfs_client_lock);
 	list_del(&server->client_link);
 	list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0

78 lines not on this page · raw message

fs/nfs/pnfs.h new file +56 -0

56 lines not on this page · raw message

include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_CLIENT		0x0200
 #define NFSDBG_MOUNT		0x0400
 #define NFSDBG_FSCACHE		0x0800
+#define NFSDBG_PNFS		0x1000
 #define NFSDBG_ALL		0xFFFF
 
 #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +1 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {
 	u32			acl_bitmask;	/* V4 bitmask representing the ACEs
 						   that are supported on this
 						   filesystem */
+	struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */
 #endif
 	void (*destroy)(struct nfs_server *);
 
Signature · 1 line
1.7.2.1
#9 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 6 files changed, 141 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h
fs/nfs/Makefile +1 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)	+= nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
 			   delegation.o idmap.o \
 			   callback.o callback_xdr.o callback_proc.o \
 			   nfs4namespace.o
+nfs-$(CONFIG_NFS_V4_1)	+= pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
 #include "iostat.h"
 #include "internal.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
 	if (server->wsize > NFS_MAX_FILE_IO_SIZE)
 		server->wsize = NFS_MAX_FILE_IO_SIZE;
 	server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+	set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

	if (nfs4_has_session(clp) &&
	    (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS))
		set_pnfs_layoutdriver(server, fsinfo->layouttype);

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

Quoted 174 lines
fs/nfs/client.c +1 -0
 	server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
 	server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
 {
 	dprintk("--> nfs_free_server()\n");
 
+	unset_pnfs_layoutdriver(server);
 	spin_lock(&nfs_client_lock);
 	list_del(&server->client_link);
 	list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY		NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {
+	return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+	nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+	struct pnfs_layoutdriver_type *ld_type = NULL;
+
+	if (id == 0)
+		goto out_no_driver;
+	ld_type = find_pnfs_driver(id);
+	if (!ld_type) {
+		request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+		ld_type = find_pnfs_driver(id);
+		if (!ld_type) {
+			dprintk("%s: No pNFS module found for %u.\n",
+				__func__, id);
+			goto out_no_driver;
+		}
+	}
+	server->pnfs_curr_ld = ld_type;
+	dprintk("%s: pNFS module for %u set\n", __func__, id);
+	return;
+
+out_no_driver:
+	dprintk("%s: Using NFSv4 I/O\n", __func__);
+	server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_CLIENT		0x0200
 #define NFSDBG_MOUNT		0x0400
 #define NFSDBG_FSCACHE		0x0800
+#define NFSDBG_PNFS		0x1000
 #define NFSDBG_ALL		0xFFFF
 
 #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +0 -0
index c82ee7c..29a821d 100644

include/linux/nfs_fs_sb.h struct nfs_server { · line 147

Quoted 4 lines
include/linux/nfs_fs_sb.h +1 -0
@@ -144,6 +144,7 @@ struct nfs_server {
 	u32			acl_bitmask;	/* V4 bitmask representing the ACEs
 						   that are supported on this
 						   filesystem */
+	struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */

This is under #ifdef CONFIG_NFS_V4
Why not only under CONFIG_NFS_V4_1?
(see 9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules)

Benny

Quoted 3 lines
include/linux/nfs_fs_sb.h +0 -0
 #endif
 	void (*destroy)(struct nfs_server *);
 
#10 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 53 lines · Benny Halevy
Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches

---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++ include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +

 6 files changed, 141 insertions(+), 0 deletions(-)

 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h

fs/nfs/Makefile +0 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                          delegation.o idmap.o \
                          callback.o callback_xdr.o callback_proc.o \

                         nfs4namespace.o +nfs-$(CONFIG_NFS_V4_1)       += pnfs.o nfs-$(CONFIG_SYSCTL) += sysctl.o nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o

fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
  #include "iostat.h"
  #include "internal.h"
  #include "fscache.h"
+#include "pnfs.h"
 
  #define NFSDBG_FACILITY              NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
       if (server->wsize > NFS_MAX_FILE_IO_SIZE)
               server->wsize = NFS_MAX_FILE_IO_SIZE;
       server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

Quoted 8 lines

       if (nfs4_has_session(clp) && (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS)) set_pnfs_layoutdriver(server, fsinfo->layouttype);

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

This is called from the mount code. It is never called for a DS (unless the DS is also a MDS, in which case it will be called in the MDS role).

Quoted 168 lines
Quoted 168 lines
fs/nfs/client.c +1 -0
       server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
       server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
  {
       dprintk("--> nfs_free_server()\n");
 
+     unset_pnfs_layoutdriver(server);
       spin_lock(&nfs_client_lock);
       list_del(&server->client_link);
       list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY              NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {
+     return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+     nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+     struct pnfs_layoutdriver_type *ld_type = NULL;
+
+     if (id == 0)
+             goto out_no_driver;
+     ld_type = find_pnfs_driver(id);
+     if (!ld_type) {
+             request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+             ld_type = find_pnfs_driver(id);
+             if (!ld_type) {
+                     dprintk("%s: No pNFS module found for %u.\n",
+                             __func__, id);
+                     goto out_no_driver;
+             }
+     }
+     server->pnfs_curr_ld = ld_type;
+     dprintk("%s: pNFS module for %u set\n", __func__, id);
+     return;
+
+out_no_driver:
+     dprintk("%s: Using NFSv4 I/O\n", __func__);
+     server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
  #define NFSDBG_CLIENT                0x0200
  #define NFSDBG_MOUNT         0x0400
  #define NFSDBG_FSCACHE               0x0800
+#define NFSDBG_PNFS          0x1000
  #define NFSDBG_ALL           0xFFFF
 
Quoted 14 lines
Quoted 10 lines
include/linux/nfs_fs.h +0 -0
  #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +0 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {

      u32                     acl_bitmask;    /* V4 bitmask representing the ACEs that are supported on this filesystem */
+     struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */

This is under #ifdef CONFIG_NFS_V4
Why not only under CONFIG_NFS_V4_1?
(see 9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules)

That was part of the campaign to remove as many ifdef statements as possible. I have no preference one way or the other.

Fred

Quoted 11 lines

Benny

Quoted 3 lines

 #endif
      void (*destroy)(struct nfs_server *);

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#11 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 251 lines · Fred Isaman
Quoted 53 lines · Benny Halevy
Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 6 files changed, 141 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h
fs/nfs/Makefile +1 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                         delegation.o idmap.o \
                         callback.o callback_xdr.o callback_proc.o \
                         nfs4namespace.o
+nfs-$(CONFIG_NFS_V4_1)       += pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
 #include "iostat.h"
 #include "internal.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY              NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
      if (server->wsize > NFS_MAX_FILE_IO_SIZE)
              server->wsize = NFS_MAX_FILE_IO_SIZE;
      server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

Quoted 8 lines
       if (nfs4_has_session(clp) &&
           (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS))
               set_pnfs_layoutdriver(server, fsinfo->layouttype);

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

This is called from the mount code. It is never called for a DS (unless the DS is also a MDS, in which case it will be called in the MDS role).

Quoted 168 lines
Quoted 168 lines
fs/nfs/client.c +1 -0
      server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
      server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
 {
      dprintk("--> nfs_free_server()\n");
 
+     unset_pnfs_layoutdriver(server);
      spin_lock(&nfs_client_lock);
      list_del(&server->client_link);
      list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY              NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {
+     return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+     nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+     struct pnfs_layoutdriver_type *ld_type = NULL;
+
+     if (id == 0)
+             goto out_no_driver;
+     ld_type = find_pnfs_driver(id);
+     if (!ld_type) {
+             request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+             ld_type = find_pnfs_driver(id);
+             if (!ld_type) {
+                     dprintk("%s: No pNFS module found for %u.\n",
+                             __func__, id);
+                     goto out_no_driver;
+             }
+     }
+     server->pnfs_curr_ld = ld_type;
+     dprintk("%s: pNFS module for %u set\n", __func__, id);
+     return;
+
+out_no_driver:
+     dprintk("%s: Using NFSv4 I/O\n", __func__);
+     server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_CLIENT                0x0200
 #define NFSDBG_MOUNT         0x0400
 #define NFSDBG_FSCACHE               0x0800
+#define NFSDBG_PNFS          0x1000
 #define NFSDBG_ALL           0xFFFF
 
Quoted 14 lines

include/linux/nfs_fs_sb.h struct nfs_server { · line 147

Quoted 10 lines
include/linux/nfs_fs.h +0 -0
 #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +1 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {
      u32                     acl_bitmask;    /* V4 bitmask representing the ACEs
                                                 that are supported on this
                                                 filesystem */
+     struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */

This is under #ifdef CONFIG_NFS_V4
Why not only under CONFIG_NFS_V4_1?
(see 9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules)

Quoted 2 lines

That was part of the campaign to remove as many ifdef statements as possible. I have no preference one way or the other.

I'll defer this to Trond then.

In any case, I think that this patch can be merged with the following one - "full mount/umount infrastructure" for completeness.

Benny

Quoted 14 lines

Fred

Quoted 11 lines

Benny

include/linux/nfs_fs_sb.h struct nfs_server { · line 150

Quoted 3 lines
include/linux/nfs_fs_sb.h +0 -0
 #endif
      void (*destroy)(struct nfs_server *);
 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 255 lines · Benny Halevy
Quoted 251 lines · Fred Isaman
Quoted 53 lines · Benny Halevy
Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches

---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++ include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +

 6 files changed, 141 insertions(+), 0 deletions(-)

 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h

fs/nfs/Makefile +0 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                          delegation.o idmap.o \
                          callback.o callback_xdr.o callback_proc.o \

                         nfs4namespace.o +nfs-$(CONFIG_NFS_V4_1)       += pnfs.o nfs-$(CONFIG_SYSCTL) += sysctl.o nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o

fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
  #include "iostat.h"
  #include "internal.h"
  #include "fscache.h"
+#include "pnfs.h"
 
  #define NFSDBG_FACILITY              NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
       if (server->wsize > NFS_MAX_FILE_IO_SIZE)
               server->wsize = NFS_MAX_FILE_IO_SIZE;
       server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

Quoted 8 lines

       if (nfs4_has_session(clp) && (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS)) set_pnfs_layoutdriver(server, fsinfo->layouttype);

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

This is called from the mount code.  It is never called for a DS (unless the DS is also a MDS, in which case it will be called in the MDS role).

Quoted 168 lines
Quoted 168 lines
fs/nfs/client.c +1 -0
       server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
       server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
  {
       dprintk("--> nfs_free_server()\n");
 
+     unset_pnfs_layoutdriver(server);
       spin_lock(&nfs_client_lock);
       list_del(&server->client_link);
       list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY              NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {
+     return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+     nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+     struct pnfs_layoutdriver_type *ld_type = NULL;
+
+     if (id == 0)
+             goto out_no_driver;
+     ld_type = find_pnfs_driver(id);
+     if (!ld_type) {
+             request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+             ld_type = find_pnfs_driver(id);
+             if (!ld_type) {
+                     dprintk("%s: No pNFS module found for %u.\n",
+                             __func__, id);
+                     goto out_no_driver;
+             }
+     }
+     server->pnfs_curr_ld = ld_type;
+     dprintk("%s: pNFS module for %u set\n", __func__, id);
+     return;
+
+out_no_driver:
+     dprintk("%s: Using NFSv4 I/O\n", __func__);
+     server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
  #define NFSDBG_CLIENT                0x0200
  #define NFSDBG_MOUNT         0x0400
  #define NFSDBG_FSCACHE               0x0800
+#define NFSDBG_PNFS          0x1000
  #define NFSDBG_ALL           0xFFFF
 
Quoted 14 lines
Quoted 10 lines
include/linux/nfs_fs.h +0 -0
  #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +0 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {

      u32                     acl_bitmask;    /* V4 bitmask representing the ACEs that are supported on this filesystem */
+     struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */

This is under #ifdef CONFIG_NFS_V4
Why not only under CONFIG_NFS_V4_1?
(see 9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules)

Quoted 2 lines

That was part of the campaign to remove as many ifdef statements as possible.  I have no preference one way or the other.

Quoted 7 lines

I'll defer this to Trond then.

In any case, I think that this patch can be merged with the following one - "full mount/umount infrastructure" for completeness.

Benny

This was an attempt to minimize the size of the patch touching the current NFS code patch.

Fred

Quoted 19 lines
Quoted 14 lines

Fred

Quoted 11 lines

Benny

Quoted 3 lines

 #endif
      void (*destroy)(struct nfs_server *);

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#13 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

In reply to Fred Isaman
Quoted 58 lines · Fred Isaman
Quoted 57 lines · Benny Halevy
Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 6 files changed, 141 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h
fs/nfs/Makefile +1 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                         delegation.o idmap.o \
                         callback.o callback_xdr.o callback_proc.o \
                         nfs4namespace.o
+nfs-$(CONFIG_NFS_V4_1)       += pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
 #include "iostat.h"
 #include "internal.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY              NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
      if (server->wsize > NFS_MAX_FILE_IO_SIZE)
              server->wsize = NFS_MAX_FILE_IO_SIZE;
      server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

       if (nfs4_has_session(clp) &&
           (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS))
               set_pnfs_layoutdriver(server, fsinfo->layouttype);
Quoted 8 lines
Quoted 4 lines

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

This is called from the mount code. It is never called for a DS (unless the DS is also a MDS, in which case it will be called in the MDS role).

So at least let's add a BUG_ON and a comment since we're relying on side effects of code implemented elsewhere.

Benny

P.S. I wonder what happens if you try to explicitly mount a DS (which is not an MDS).

Quoted 201 lines
Quoted 170 lines
Quoted 170 lines
fs/nfs/client.c +1 -0
      server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
      server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
 {
      dprintk("--> nfs_free_server()\n");
 
+     unset_pnfs_layoutdriver(server);
      spin_lock(&nfs_client_lock);
      list_del(&server->client_link);
      list_del(&server->master_link);
fs/nfs/pnfs.c new file +78 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY              NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {
+     return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+     nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+     struct pnfs_layoutdriver_type *ld_type = NULL;
+
+     if (id == 0)
+             goto out_no_driver;
+     ld_type = find_pnfs_driver(id);
+     if (!ld_type) {
+             request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+             ld_type = find_pnfs_driver(id);
+             if (!ld_type) {
+                     dprintk("%s: No pNFS module found for %u.\n",
+                             __func__, id);
+                     goto out_no_driver;
+             }
+     }
+     server->pnfs_curr_ld = ld_type;
+     dprintk("%s: pNFS module for %u set\n", __func__, id);
+     return;
+
+out_no_driver:
+     dprintk("%s: Using NFSv4 I/O\n", __func__);
+     server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_CLIENT                0x0200
 #define NFSDBG_MOUNT         0x0400
 #define NFSDBG_FSCACHE               0x0800
+#define NFSDBG_PNFS          0x1000
 #define NFSDBG_ALL           0xFFFF
 
Quoted 14 lines

include/linux/nfs_fs_sb.h struct nfs_server { · line 147

Quoted 10 lines
include/linux/nfs_fs.h +0 -0
 #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +1 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {
      u32                     acl_bitmask;    /* V4 bitmask representing the ACEs
                                                 that are supported on this
                                                 filesystem */
+     struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */

This is under #ifdef CONFIG_NFS_V4
Why not only under CONFIG_NFS_V4_1?
(see 9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules)

That was part of the campaign to remove as many ifdef statements as possible. I have no preference one way or the other.

Fred

Quoted 11 lines

Benny

include/linux/nfs_fs_sb.h struct nfs_server { · line 150

Quoted 3 lines
include/linux/nfs_fs_sb.h +0 -0
 #endif
      void (*destroy)(struct nfs_server *);
 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 64 lines · Benny Halevy
Quoted 63 lines · Fred Isaman
Quoted 61 lines · Benny Halevy
Quoted 43 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches

---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++ include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +

 6 files changed, 141 insertions(+), 0 deletions(-)

 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h

fs/nfs/Makefile +0 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                          delegation.o idmap.o \
                          callback.o callback_xdr.o callback_proc.o \

                         nfs4namespace.o +nfs-$(CONFIG_NFS_V4_1)       += pnfs.o nfs-$(CONFIG_SYSCTL) += sysctl.o nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o

fs/nfs/client.c +1 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
  #include "iostat.h"
  #include "internal.h"
  #include "fscache.h"
+#include "pnfs.h"
 
  #define NFSDBG_FACILITY              NFSDBG_CLIENT
 

fs/nfs/client.c static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo * · line 903

Quoted 5 lines
fs/nfs/client.c +2 -0
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
       if (server->wsize > NFS_MAX_FILE_IO_SIZE)
               server->wsize = NFS_MAX_FILE_IO_SIZE;
       server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+

Originally (9170cf5 pnfs_submit: set and unset pnfs layoutdriver modules), nfs4_init_pnfs called set_pnfs_layoutdriver under the following conditions:

       if (nfs4_has_session(clp) && (clp->cl_exchange_flags & EXCHGID4_FLAG_USE_PNFS_MDS)) set_pnfs_layoutdriver(server, fsinfo->layouttype);

So in the DS only case we don't want to set server->pnfs_curr_ld. did you test your code with a standalone DS to make there are no ill side effects in this case?

Quoted 7 lines
Quoted 3 lines

This is called from the mount code.  It is never called for a DS (unless the DS is also a MDS, in which case it will be called in the MDS role).

So at least let's add a BUG_ON and a comment since we're relying on side effects of code implemented elsewhere.

OK.

Quoted 6 lines

Benny

P.S. I wonder what happens if you try to explicitly mount a DS (which is not an MDS).

Note there are two possibilities here, DS only, or DS and normal NFS server (but not a MDS).

In the second case, the mount should work, and it does. In the first, the client should abort when it sees the EXCHID flags (which it doesn't...this needs to be fixed) or when the server sends NFS4ERR_NOTSUPP to the PUTROOTFH, which it MUST do.

Fred

#15 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

In reply to Fred Isaman
Quoted 92 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 6 files changed, 141 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h
fs/nfs/Makefile +1 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)	+= nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
 			   delegation.o idmap.o \
 			   callback.o callback_xdr.o callback_proc.o \
 			   nfs4namespace.o
+nfs-$(CONFIG_NFS_V4_1)	+= pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
fs/nfs/client.c +4 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
 #include "iostat.h"
 #include "internal.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_CLIENT
 
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
 	if (server->wsize > NFS_MAX_FILE_IO_SIZE)
 		server->wsize = NFS_MAX_FILE_IO_SIZE;
 	server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+	set_pnfs_layoutdriver(server, fsinfo->layouttype);
+
 	server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
 	server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
 {
 	dprintk("--> nfs_free_server()\n");
 
+	unset_pnfs_layoutdriver(server);
 	spin_lock(&nfs_client_lock);
 	list_del(&server->client_link);
 	list_del(&server->master_link);
fs/nfs/pnfs.c new file +27 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.

fs/nfs/pnfs.c line 37

Quoted 10 lines
fs/nfs/pnfs.c new file +10 -0
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY		NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {

nit: the curly brace should be moved down a line.

Benny

Quoted 127 lines
fs/nfs/pnfs.c new file +41 -0
+	return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+	nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+	struct pnfs_layoutdriver_type *ld_type = NULL;
+
+	if (id == 0)
+		goto out_no_driver;
+	ld_type = find_pnfs_driver(id);
+	if (!ld_type) {
+		request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+		ld_type = find_pnfs_driver(id);
+		if (!ld_type) {
+			dprintk("%s: No pNFS module found for %u.\n",
+				__func__, id);
+			goto out_no_driver;
+		}
+	}
+	server->pnfs_curr_ld = ld_type;
+	dprintk("%s: pNFS module for %u set\n", __func__, id);
+	return;
+
+out_no_driver:
+	dprintk("%s: Using NFSv4 I/O\n", __func__);
+	server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_CLIENT		0x0200
 #define NFSDBG_MOUNT		0x0400
 #define NFSDBG_FSCACHE		0x0800
+#define NFSDBG_PNFS		0x1000
 #define NFSDBG_ALL		0xFFFF
 
 #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +1 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {
 	u32			acl_bitmask;	/* V4 bitmask representing the ACEs
 						   that are supported on this
 						   filesystem */
+	struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */
 #endif
 	void (*destroy)(struct nfs_server *);
 
#16 lore raw

Re: [PATCH 06/12] RFC: nfs: set layout driver

Quoted 106 lines · Benny Halevy
Quoted 92 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Put in the infrastructure that uses information returned from the server at mount to select a layout driver module.

In this patch, a stub is used that always returns "no driver found".

Signed-off-by TBD - melding/reorganization of several patches

---
 fs/nfs/Makefile           |    1 +
 fs/nfs/client.c           |    4 ++
 fs/nfs/pnfs.c             |   78 +++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/pnfs.h             |   56 ++++++++++++++++++++++++++++++++ include/linux/nfs_fs.h    |    1 +
 include/linux/nfs_fs_sb.h |    1 +

 6 files changed, 141 insertions(+), 0 deletions(-)

 create mode 100644 fs/nfs/pnfs.c
 create mode 100644 fs/nfs/pnfs.h

fs/nfs/Makefile +0 -0
index da7fda6..bb9e773 100644
@@ -15,5 +15,6 @@ nfs-$(CONFIG_NFS_V4)        += nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
                          delegation.o idmap.o \
                          callback.o callback_xdr.o callback_proc.o \

                         nfs4namespace.o +nfs-$(CONFIG_NFS_V4_1)       += pnfs.o nfs-$(CONFIG_SYSCTL) += sysctl.o nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o

fs/nfs/client.c +4 -0
index 4e7df2a..eed1212 100644
@@ -48,6 +48,7 @@
  #include "iostat.h"
  #include "internal.h"
  #include "fscache.h"
+#include "pnfs.h"
 
  #define NFSDBG_FACILITY              NFSDBG_CLIENT
 
@@ -898,6 +899,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
       if (server->wsize > NFS_MAX_FILE_IO_SIZE)
               server->wsize = NFS_MAX_FILE_IO_SIZE;
       server->wpages = (server->wsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+     set_pnfs_layoutdriver(server, fsinfo->layouttype);
+
       server->wtmult = nfs_block_bits(fsinfo->wtmult, NULL);
 
       server->dtsize = nfs_block_size(fsinfo->dtpref, NULL);
@@ -1017,6 +1020,7 @@ void nfs_free_server(struct nfs_server *server)
  {
       dprintk("--> nfs_free_server()\n");
 
+     unset_pnfs_layoutdriver(server);
       spin_lock(&nfs_client_lock);
       list_del(&server->client_link);
       list_del(&server->master_link);
fs/nfs/pnfs.c new file +27 -0
new file mode 100644
index 0000000..2e5dba1
@@ -0,0 +1,78 @@
+/*
+ *  pNFS functions to call and manage layout drivers.
+ *
+ *  Copyright (c) 2002 [year of first publication]
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.

fs/nfs/pnfs.c line 37

Quoted 10 lines
fs/nfs/pnfs.c new file +10 -0
+ */
+
+#include <linux/nfs_fs.h>
+#include "pnfs.h"
+
+#define NFSDBG_FACILITY              NFSDBG_PNFS
+
+/* STUB that returns the equivalent of "no module found" */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver(u32 id) {

nit: the curly brace should be moved down a line.

Quoted 2 lines

Benny

OK.

Fred

Quoted 132 lines
Quoted 120 lines
fs/nfs/pnfs.c new file +41 -0
+     return NULL;
+}
+
+/* Unitialize a mountpoint in a layout driver */
+void
+unset_pnfs_layoutdriver(struct nfs_server *nfss)
+{
+     nfss->pnfs_curr_ld = NULL;
+}
+
+/*
+ * Try to set the server's pnfs module to the pnfs layout type specified by id.
+ * Currently only one pNFS layout driver per filesystem is supported.
+ *
+ * @id layout type. Zero (illegal layout type) indicates pNFS not in use.
+ */
+void
+set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
+{
+     struct pnfs_layoutdriver_type *ld_type = NULL;
+
+     if (id == 0)
+             goto out_no_driver;
+     ld_type = find_pnfs_driver(id);
+     if (!ld_type) {
+             request_module("%s-%u", LAYOUT_NFSV4_1_MODULE_PREFIX, id);
+             ld_type = find_pnfs_driver(id);
+             if (!ld_type) {
+                     dprintk("%s: No pNFS module found for %u.\n",
+                             __func__, id);
+                     goto out_no_driver;
+             }
+     }
+     server->pnfs_curr_ld = ld_type;
+     dprintk("%s: pNFS module for %u set\n", __func__, id);
+     return;
+
+out_no_driver:
+     dprintk("%s: Using NFSv4 I/O\n", __func__);
+     server->pnfs_curr_ld = NULL;
+}
fs/nfs/pnfs.h new file +56 -0
new file mode 100644
index 0000000..c628ef1
@@ -0,0 +1,56 @@
+/*
+ *  pNFS client data structures.
+ *
+ *  Copyright (c) 2002
+ *  The Regents of the University of Michigan
+ *  All Rights Reserved
+ *
+ *  Dean Hildebrand <[email protected]>
+ *
+ *  Permission is granted to use, copy, create derivative works, and
+ *  redistribute this software and such derivative works for any purpose,
+ *  so long as the name of the University of Michigan is not used in
+ *  any advertising or publicity pertaining to the use or distribution
+ *  of this software without specific, written prior authorization. If
+ *  the above copyright notice or any other identification of the
+ *  University of Michigan is included in any copy of any portion of
+ *  this software, then the disclaimer below must also be included.
+ *
+ *  This software is provided as is, without representation or warranty
+ *  of any kind either express or implied, including without limitation
+ *  the implied warranties of merchantability, fitness for a particular
+ *  purpose, or noninfringement.  The Regents of the University of
+ *  Michigan shall not be liable for any damages, including special,
+ *  indirect, incidental, or consequential damages, with respect to any
+ *  claim arising out of or in connection with the use of the software,
+ *  even if it has been or is hereafter advised of the possibility of
+ *  such damages.
+ */
+
+#ifndef FS_NFS_PNFS_H
+#define FS_NFS_PNFS_H
+
+#ifdef CONFIG_NFS_V4_1
+
+#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
+
+/* Per-layout driver specific registration structure */
+struct pnfs_layoutdriver_type {
+};
+
+void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
+void unset_pnfs_layoutdriver(struct nfs_server *);
+
+#else  /* CONFIG_NFS_V4_1 */
+
+static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
+{
+}
+
+static inline void unset_pnfs_layoutdriver(struct nfs_server *s)
+{
+}
+
+#endif /* CONFIG_NFS_V4_1 */
+
+#endif /* FS_NFS_PNFS_H */
include/linux/nfs_fs.h +1 -0
index d929b18..aba3da2 100644
@@ -615,6 +615,7 @@ nfs_fileid_to_ino_t(u64 fileid)
  #define NFSDBG_CLIENT                0x0200
  #define NFSDBG_MOUNT         0x0400
  #define NFSDBG_FSCACHE               0x0800
+#define NFSDBG_PNFS          0x1000
  #define NFSDBG_ALL           0xFFFF
 
  #ifdef __KERNEL__
include/linux/nfs_fs_sb.h +0 -0
index c82ee7c..29a821d 100644
@@ -144,6 +144,7 @@ struct nfs_server {
Quoted 7 lines

      u32                     acl_bitmask;    /* V4 bitmask representing the ACEs that are supported on this filesystem */
+     struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */ #endif
      void (*destroy)(struct nfs_server *);

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#17 lore raw

PATCH 7/12 RFC: pnfs: full mount/umount infrastructure

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

Allow a module implementing a layout type to register, and have its mount/umount routines called for filesystems that the server declares support it.

Signed-off-by TBD - melding/reorganization of several patches
---
 Documentation/filesystems/nfs/00-INDEX |    2 +
 Documentation/filesystems/nfs/pnfs.txt |   48 ++++++++++++++++++
 fs/nfs/Kconfig                         |    2 +-
 fs/nfs/pnfs.c                          |   82 +++++++++++++++++++++++++++++++-
 fs/nfs/pnfs.h                          |    9 ++++
 5 files changed, 140 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/filesystems/nfs/pnfs.txt
Documentation/filesystems/nfs/00-INDEX +2 -0
index 2f68cd6..8d930b9 100644
@@ -12,5 +12,7 @@ nfs-rdma.txt
 	- how to install and setup the Linux NFS/RDMA client and server software
 nfsroot.txt
 	- short guide on setting up a diskless box with NFS root filesystem.
+pnfs.txt
+	- short explanation of some of the internals of the pnfs code
 rpc-cache.txt
 	- introduction to the caching mechanisms in the sunrpc layer.
Documentation/filesystems/nfs/pnfs.txt new file +48 -0

48 lines not on this page · raw message

fs/nfs/Kconfig +1 -1
index 6c2aad4..5f1b936 100644
@@ -78,7 +78,7 @@ config NFS_V4_1
 	depends on NFS_V4 && EXPERIMENTAL
 	help
 	  This option enables support for minor version 1 of the NFSv4 protocol
-	  (draft-ietf-nfsv4-minorversion1) in the kernel's NFS client.
+	  (RFC 5661) in the kernel's NFS client.
 
 	  If unsure, say N.
 
fs/nfs/pnfs.c +80 -2

105 lines not on this page · raw message

fs/nfs/pnfs.h +9 -0
index c628ef1..61531f3 100644
@@ -36,8 +36,17 @@
 
 /* Per-layout driver specific registration structure */
 struct pnfs_layoutdriver_type {
+	struct list_head pnfs_tblid;
+	const u32 id;
+	const char *name;
+	struct module *owner;
+	int (*initialize_mountpoint) (struct nfs_server *);
+	int (*uninitialize_mountpoint) (struct nfs_server *);
 };
 
+extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
+extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
+
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
 
Signature · 1 line
1.7.2.1
#18 lore raw

Re: [PATCH 07/12] RFC: pnfs: full mount/umount infrastructure

Quoted 21 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Allow a module implementing a layout type to register, and have its mount/umount routines called for filesystems that the server declares support it.

Signed-off-by TBD - melding/reorganization of several patches
---
 Documentation/filesystems/nfs/00-INDEX |    2 +
 Documentation/filesystems/nfs/pnfs.txt |   48 ++++++++++++++++++
 fs/nfs/Kconfig                         |    2 +-
 fs/nfs/pnfs.c                          |   82 +++++++++++++++++++++++++++++++-
 fs/nfs/pnfs.h                          |    9 ++++
 5 files changed, 140 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/filesystems/nfs/pnfs.txt
Documentation/filesystems/nfs/00-INDEX +0 -0
index 2f68cd6..8d930b9 100644

Documentation/filesystems/nfs/00-INDEX nfs-rdma.txt · line 16

Quoted 5 lines
Documentation/filesystems/nfs/00-INDEX +2 -0
@@ -12,5 +12,7 @@ nfs-rdma.txt
 	- how to install and setup the Linux NFS/RDMA client and server software
 nfsroot.txt
 	- short guide on setting up a diskless box with NFS root filesystem.
+pnfs.txt
+	- short explanation of some of the internals of the pnfs code

that is, pnfs _client_ code...

Quoted 83 lines
Documentation/filesystems/nfs/00-INDEX +0 -0
 rpc-cache.txt
 	- introduction to the caching mechanisms in the sunrpc layer.
Documentation/filesystems/nfs/pnfs.txt new file +48 -0
new file mode 100644
index 0000000..bc0b9cf
@@ -0,0 +1,48 @@
+Reference counting in pnfs:
+==========================
+
+The are several inter-related caches.  We have layouts which can
+reference multiple devices, each of which can reference multiple data servers.
+Each data server can be referenced by multiple devices.  Each device
+can be referenced by multiple layouts.  To keep all of this straight,
+we need to reference count.
+
+
+struct pnfs_layout_hdr
+----------------------
+The on-the-wire command LAYOUTGET corresponds to struct
+pnfs_layout_segment, usually referred to by the variable name lseg.
+Each nfs_inode may hold a pointer to a cache of of these layout
+segments in nfsi->layout, of type struct pnfs_layout_hdr.
+
+We reference the header for the inode pointing to it, across each
+outstanding RPC call that references it (LAYOUTGET, fs/nfs/LAYOUTRETURN,
+LAYOUTCOMMIT), and for each lseg held within.
+
+Each header is also (when non-empty) put on a list associated with
+struct nfs_client (cl_layouts).  Being put on this list does not bump
+the reference count, as the layout is kept around by the lseg that
+keeps it in the list.
+
+deviceid_cache
+--------------
+lsegs reference device ids, which are resolved per nfs_client and
+layout driver type.  The device ids are held in a RCU cache (struct
+nfs4_deviceid_cache).  The cache itself is referenced across each
+mount.  The entries (struct nfs4_deviceid) themselves are held across
+the lifetime of each lseg referencing them.
+
+RCU is used because the deviceid is basically a write once, read many
+data structure.  The hlist size of 32 buckets needs better
+justification, but seems reasonable given that we can have multiple
+deviceid's per filesystem, and multiple filesystems per nfs_client.
+
+The hash code is copied from the nfsd code base.  A discussion of
+hashing and variations of this algorithm can be found at:
+http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809
+
+data server cache
+-----------------
+file driver devices refer to data servers, which are kept in a module
+level cache.  Its reference is held over the lifetime of the deviceid
+pointing to it.
fs/nfs/Kconfig +1 -1
index 6c2aad4..5f1b936 100644
@@ -78,7 +78,7 @@ config NFS_V4_1
 	depends on NFS_V4 && EXPERIMENTAL
 	help
 	  This option enables support for minor version 1 of the NFSv4 protocol
-	  (draft-ietf-nfsv4-minorversion1) in the kernel's NFS client.
+	  (RFC 5661) in the kernel's NFS client.
 
 	  If unsure, say N.
 
fs/nfs/pnfs.c +5 -1
index 2e5dba1..5a8a676 100644
@@ -32,16 +32,53 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PNFS
 
-/* STUB that returns the equivalent of "no module found" */
+/* Locking:
+ *
+ * pnfs_spinlock:
+ *      protects pnfs_modules_tbl.
+ */

fs/nfs/pnfs.c line 49

Quoted 10 lines
fs/nfs/pnfs.c +10 -0
+static DEFINE_SPINLOCK(pnfs_spinlock);
+
+/*
+ * pnfs_modules_tbl holds all pnfs modules
+ */
+static LIST_HEAD(pnfs_modules_tbl);
+
+/* Return the registered pnfs layout driver module matching given id */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver_locked(u32 id) {

nit: the curly brace should be moved down a line

fs/nfs/pnfs.c line 52

Quoted 3 lines
fs/nfs/pnfs.c +3 -0
+	struct  pnfs_layoutdriver_type *local;
+
+	dprintk("PNFS: %s: Searching for %u\n", __func__, id);

I'd move this printk down, before returning and print the result as well.

fs/nfs/pnfs.c line 56

Quoted 4 lines
fs/nfs/pnfs.c +4 -0
+	list_for_each_entry(local, &pnfs_modules_tbl, pnfs_tblid)
+		if (local->id == id) {
+			if (!try_module_get(local->owner))
+				local = NULL;

If this happens (e.g. in the case the module exited without calling pnfs_unregister_layoutdriver) another (or a different instance) layout driver might have registered on the same id so we need to keep looking.

fs/nfs/pnfs.c line 62

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+			goto out;
+		}
+	local = NULL;
+out:
+	return local;
+}

how about the following?

static struct pnfs_layoutdriver_type *
find_pnfs_driver_locked(u32 id)

{
	struct pnfs_layoutdriver_type *local, *found = NULL;

	list_for_each_entry (local, &pnfs_modules_tbl, pnfs_tblid)
		if (local->id == id) {
			if (!try_module_get(local->owner))
				continue;
			found = local;
			break;
		}
out:
	dprintk("%s: layout_type %u found %p\n", __func__, id, found);
	return found;
}
Quoted 38 lines
fs/nfs/pnfs.c +18 -1
+
 static struct pnfs_layoutdriver_type *
 find_pnfs_driver(u32 id) {
-	return NULL;
+	struct  pnfs_layoutdriver_type *local;
+
+	spin_lock(&pnfs_spinlock);
+	local = find_pnfs_driver_locked(id);
+	spin_unlock(&pnfs_spinlock);
+	return local;
 }
 
 /* Unitialize a mountpoint in a layout driver */
 void
 unset_pnfs_layoutdriver(struct nfs_server *nfss)
 {
+	if (nfss->pnfs_curr_ld) {
+		nfss->pnfs_curr_ld->uninitialize_mountpoint(nfss);
+		module_put(nfss->pnfs_curr_ld->owner);
+	}
 	nfss->pnfs_curr_ld = NULL;
 }
 
@@ -68,6 +105,13 @@ set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
 			goto out_no_driver;
 		}
 	}
+	if (ld_type->initialize_mountpoint(server)) {
+		printk(KERN_ERR
+		       "%s: Error initializing mount point for layout driver %u.\n",
+		       __func__, id);
+		module_put(ld_type->owner);
+		goto out_no_driver;
+	}
 	server->pnfs_curr_ld = ld_type;
 	dprintk("%s: pNFS module for %u set\n", __func__, id);
 	return;

fs/nfs/pnfs.c out_no_driver: · line 129

Quoted 10 lines
fs/nfs/pnfs.c +7 -0
@@ -76,3 +120,37 @@ out_no_driver:
 	dprintk("%s: Using NFSv4 I/O\n", __func__);
 	server->pnfs_curr_ld = NULL;
 }
+
+int
+pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+	int status = -EINVAL;
+	struct pnfs_layoutdriver_type *tmp;
+

Since we're relying on the fact the ld_type->id != 0 let's add

	BUG_ON(ld_type->id == 0);

Benny

Quoted 49 lines
fs/nfs/pnfs.c +27 -0
+	spin_lock(&pnfs_spinlock);
+	tmp = find_pnfs_driver_locked(ld_type->id);
+	if (!tmp) {
+		list_add(&ld_type->pnfs_tblid, &pnfs_modules_tbl);
+		status = 0;
+		dprintk("%s Registering id:%u name:%s\n", __func__, ld_type->id,
+			ld_type->name);
+	} else {
+		module_put(tmp->owner);
+		printk(KERN_ERR "%s Module with id %d already loaded!\n",
+			__func__, ld_type->id);
+	}
+	spin_unlock(&pnfs_spinlock);
+
+	return status;
+}
+EXPORT_SYMBOL_GPL(pnfs_register_layoutdriver);
+
+void
+pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+	dprintk("%s Deregistering id:%u\n", __func__, ld_type->id);
+	spin_lock(&pnfs_spinlock);
+	list_del(&ld_type->pnfs_tblid);
+	spin_unlock(&pnfs_spinlock);
+}
+EXPORT_SYMBOL_GPL(pnfs_unregister_layoutdriver);
fs/nfs/pnfs.h +9 -0
index c628ef1..61531f3 100644
@@ -36,8 +36,17 @@
 
 /* Per-layout driver specific registration structure */
 struct pnfs_layoutdriver_type {
+	struct list_head pnfs_tblid;
+	const u32 id;
+	const char *name;
+	struct module *owner;
+	int (*initialize_mountpoint) (struct nfs_server *);
+	int (*uninitialize_mountpoint) (struct nfs_server *);
 };
 
+extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
+extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
+
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
 
#19 lore raw

Re: [PATCH 07/12] RFC: pnfs: full mount/umount infrastructure

Quoted 17 lines · Benny Halevy
Quoted 16 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Allow a module implementing a layout type to register, and have its mount/umount routines called for filesystems that the server declares support it.

Signed-off-by TBD - melding/reorganization of several patches

---
 Documentation/filesystems/nfs/00-INDEX |    2 + Documentation/filesystems/nfs/pnfs.txt |   48 ++++++++++++++++++ fs/nfs/Kconfig                         |    2 +-
 fs/nfs/pnfs.c                          |   82 +++++++++++++++++++++++++++++++- fs/nfs/pnfs.h                          |    9 ++++

 5 files changed, 140 insertions(+), 3 deletions(-)

 create mode 100644 Documentation/filesystems/nfs/pnfs.txt

Quoted 13 lines
Quoted 10 lines
Documentation/filesystems/nfs/00-INDEX +0 -0
index 2f68cd6..8d930b9 100644
@@ -12,5 +12,7 @@ nfs-rdma.txt

      - how to install and setup the Linux NFS/RDMA client and server software nfsroot.txt
      - short guide on setting up a diskless box with NFS root filesystem. +pnfs.txt
+     - short explanation of some of the internals of the pnfs code

that is, pnfs _client_ code...

OK

Quoted 87 lines
Quoted 87 lines

 rpc-cache.txt
      - introduction to the caching mechanisms in the sunrpc layer.

Documentation/filesystems/nfs/pnfs.txt new file +48 -0
new file mode 100644
index 0000000..bc0b9cf
@@ -0,0 +1,48 @@
+Reference counting in pnfs:
+==========================
+
+The are several inter-related caches.  We have layouts which can
+reference multiple devices, each of which can reference multiple data servers.
+Each data server can be referenced by multiple devices.  Each device
+can be referenced by multiple layouts.  To keep all of this straight,
+we need to reference count.
+
+
+struct pnfs_layout_hdr
+----------------------
+The on-the-wire command LAYOUTGET corresponds to struct
+pnfs_layout_segment, usually referred to by the variable name lseg.
+Each nfs_inode may hold a pointer to a cache of of these layout
+segments in nfsi->layout, of type struct pnfs_layout_hdr.
+
+We reference the header for the inode pointing to it, across each
+outstanding RPC call that references it (LAYOUTGET, fs/nfs/LAYOUTRETURN,
+LAYOUTCOMMIT), and for each lseg held within.
+
+Each header is also (when non-empty) put on a list associated with
+struct nfs_client (cl_layouts).  Being put on this list does not bump
+the reference count, as the layout is kept around by the lseg that
+keeps it in the list.
+
+deviceid_cache
+--------------
+lsegs reference device ids, which are resolved per nfs_client and
+layout driver type.  The device ids are held in a RCU cache (struct
+nfs4_deviceid_cache).  The cache itself is referenced across each
+mount.  The entries (struct nfs4_deviceid) themselves are held across
+the lifetime of each lseg referencing them.
+
+RCU is used because the deviceid is basically a write once, read many
+data structure.  The hlist size of 32 buckets needs better
+justification, but seems reasonable given that we can have multiple
+deviceid's per filesystem, and multiple filesystems per nfs_client.
+
+The hash code is copied from the nfsd code base.  A discussion of
+hashing and variations of this algorithm can be found at:
+http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809
+
+data server cache
+-----------------
+file driver devices refer to data servers, which are kept in a module
+level cache.  Its reference is held over the lifetime of the deviceid
+pointing to it.
fs/nfs/Kconfig +0 -0
index 6c2aad4..5f1b936 100644
@@ -78,7 +78,7 @@ config NFS_V4_1

      depends on NFS_V4 && EXPERIMENTAL
      help
        This option enables support for minor version 1 of the NFSv4 protocol
-       (draft-ietf-nfsv4-minorversion1) in the kernel's NFS client.
+       (RFC 5661) in the kernel's NFS client.

        If unsure, say N.

fs/nfs/pnfs.c +9 -1
index 2e5dba1..5a8a676 100644
@@ -32,16 +32,53 @@
 
  #define NFSDBG_FACILITY              NFSDBG_PNFS
 
-/* STUB that returns the equivalent of "no module found" */
+/* Locking:
+ *
+ * pnfs_spinlock:
+ *      protects pnfs_modules_tbl.
+ */
+static DEFINE_SPINLOCK(pnfs_spinlock);
+
+/*
+ * pnfs_modules_tbl holds all pnfs modules
Quoted 8 lines

fs/nfs/pnfs.c line 49

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+ */
+static LIST_HEAD(pnfs_modules_tbl);
+
+/* Return the registered pnfs layout driver module matching given id */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver_locked(u32 id) {

nit: the curly brace should be moved down a line

OK

Quoted 8 lines

fs/nfs/pnfs.c line 52

Quoted 4 lines
fs/nfs/pnfs.c +3 -0
+     struct  pnfs_layoutdriver_type *local;
+
+     dprintk("PNFS: %s: Searching for %u\n", __func__, id);

I'd move this printk down, before returning and print the result as well.

OK

Quoted 9 lines

fs/nfs/pnfs.c line 56

Quoted 4 lines
fs/nfs/pnfs.c +4 -0
+     list_for_each_entry(local, &pnfs_modules_tbl, pnfs_tblid)
+             if (local->id == id) {
+                     if (!try_module_get(local->owner))
+                             local = NULL;

If this happens (e.g. in the case the module exited without calling pnfs_unregister_layoutdriver) another (or a different instance) layout driver might have registered on the same id so we need to keep looking.

There can only be a single driver registered with a particular id. So continuing the search is pointless.

Quoted 80 lines

fs/nfs/pnfs.c line 62

Quoted 7 lines
fs/nfs/pnfs.c +6 -0
+                     goto out;
+             }
+     local = NULL;
+out:
+     return local;
+}

how about the following?

static struct pnfs_layoutdriver_type * find_pnfs_driver_locked(u32 id)
{
       struct pnfs_layoutdriver_type *local, *found = NULL;

       list_for_each_entry (local, &pnfs_modules_tbl, pnfs_tblid) if (local->id == id) {
                       if (!try_module_get(local->owner)) continue;
                       found = local;
                       break;
               }
out:
       dprintk("%s: layout_type %u found %p\n", __func__, id, found); return found;
}

Quoted 38 lines
fs/nfs/pnfs.c +1 -0
+

 static struct pnfs_layoutdriver_type *
 find_pnfs_driver(u32 id) {
-     return NULL;
+     struct  pnfs_layoutdriver_type *local;
+
+     spin_lock(&pnfs_spinlock);
+     local = find_pnfs_driver_locked(id);
+     spin_unlock(&pnfs_spinlock);
+     return local;
 }

 /* Unitialize a mountpoint in a layout driver */
 void
 unset_pnfs_layoutdriver(struct nfs_server *nfss)
 {
+     if (nfss->pnfs_curr_ld) {
+             nfss->pnfs_curr_ld->uninitialize_mountpoint(nfss);
+             module_put(nfss->pnfs_curr_ld->owner);
+     }
      nfss->pnfs_curr_ld = NULL;
 }

fs/nfs/pnfs.c +7 -0
@@ -68,6 +105,13 @@ set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
                       goto out_no_driver;
               }
       }
+     if (ld_type->initialize_mountpoint(server)) {
+             printk(KERN_ERR
+                    "%s: Error initializing mount point for layout driver %u.\n",
+                    __func__, id);
+             module_put(ld_type->owner);
+             goto out_no_driver;
+     }
       server->pnfs_curr_ld = ld_type;
       dprintk("%s: pNFS module for %u set\n", __func__, id);
       return;

fs/nfs/pnfs.c out_no_driver: · line 129

Quoted 10 lines
fs/nfs/pnfs.c +7 -0
@@ -76,3 +120,37 @@ out_no_driver:
       dprintk("%s: Using NFSv4 I/O\n", __func__);
       server->pnfs_curr_ld = NULL;
  }
+
+int
+pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+     int status = -EINVAL;
+     struct pnfs_layoutdriver_type *tmp;
+

Since we're relying on the fact the ld_type->id != 0 let's add
       BUG_ON(ld_type->id == 0);

Quoted 1 line

Benny

OK.

Fred

Quoted 55 lines
Quoted 40 lines
fs/nfs/pnfs.c +27 -0
+     spin_lock(&pnfs_spinlock);
+     tmp = find_pnfs_driver_locked(ld_type->id);
+     if (!tmp) {
+             list_add(&ld_type->pnfs_tblid, &pnfs_modules_tbl);
+             status = 0;
+             dprintk("%s Registering id:%u name:%s\n", __func__, ld_type->id,
+                     ld_type->name);
+     } else {
+             module_put(tmp->owner);
+             printk(KERN_ERR "%s Module with id %d already loaded!\n",
+                     __func__, ld_type->id);
+     }
+     spin_unlock(&pnfs_spinlock);
+
+     return status;
+}
+EXPORT_SYMBOL_GPL(pnfs_register_layoutdriver);
+
+void
+pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+     dprintk("%s Deregistering id:%u\n", __func__, ld_type->id);
+     spin_lock(&pnfs_spinlock);
+     list_del(&ld_type->pnfs_tblid);
+     spin_unlock(&pnfs_spinlock);
+}
+EXPORT_SYMBOL_GPL(pnfs_unregister_layoutdriver);
fs/nfs/pnfs.h +4 -0
index c628ef1..61531f3 100644
@@ -36,8 +36,17 @@
 
  /* Per-layout driver specific registration structure */
  struct pnfs_layoutdriver_type {
+     struct list_head pnfs_tblid;
+     const u32 id;
+     const char *name;
+     struct module *owner;

fs/nfs/pnfs.h line 52

Quoted 10 lines
fs/nfs/pnfs.h +5 -0
+     int (*initialize_mountpoint) (struct nfs_server *);
+     int (*uninitialize_mountpoint) (struct nfs_server *);
  };
 
+extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
+extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
+
  void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
  void unset_pnfs_layoutdriver(struct nfs_server *);
 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#20 lore raw

Re: [PATCH 07/12] RFC: pnfs: full mount/umount infrastructure

Quoted 151 lines · Fred Isaman
Quoted 17 lines · Benny Halevy
Quoted 16 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Allow a module implementing a layout type to register, and have its mount/umount routines called for filesystems that the server declares support it.

Signed-off-by TBD - melding/reorganization of several patches
---
 Documentation/filesystems/nfs/00-INDEX |    2 +
 Documentation/filesystems/nfs/pnfs.txt |   48 ++++++++++++++++++
 fs/nfs/Kconfig                         |    2 +-
 fs/nfs/pnfs.c                          |   82 +++++++++++++++++++++++++++++++-
 fs/nfs/pnfs.h                          |    9 ++++
 5 files changed, 140 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/filesystems/nfs/pnfs.txt
Quoted 13 lines

Documentation/filesystems/nfs/00-INDEX nfs-rdma.txt · line 16

Quoted 10 lines
Documentation/filesystems/nfs/00-INDEX +2 -0
index 2f68cd6..8d930b9 100644
@@ -12,5 +12,7 @@ nfs-rdma.txt
      - how to install and setup the Linux NFS/RDMA client and server software
 nfsroot.txt
      - short guide on setting up a diskless box with NFS root filesystem.
+pnfs.txt
+     - short explanation of some of the internals of the pnfs code

that is, pnfs _client_ code...

OK

Quoted 87 lines
Quoted 87 lines
Documentation/filesystems/nfs/00-INDEX +0 -0
 rpc-cache.txt
      - introduction to the caching mechanisms in the sunrpc layer.
Documentation/filesystems/nfs/pnfs.txt new file +48 -0
new file mode 100644
index 0000000..bc0b9cf
@@ -0,0 +1,48 @@
+Reference counting in pnfs:
+==========================
+
+The are several inter-related caches.  We have layouts which can
+reference multiple devices, each of which can reference multiple data servers.
+Each data server can be referenced by multiple devices.  Each device
+can be referenced by multiple layouts.  To keep all of this straight,
+we need to reference count.
+
+
+struct pnfs_layout_hdr
+----------------------
+The on-the-wire command LAYOUTGET corresponds to struct
+pnfs_layout_segment, usually referred to by the variable name lseg.
+Each nfs_inode may hold a pointer to a cache of of these layout
+segments in nfsi->layout, of type struct pnfs_layout_hdr.
+
+We reference the header for the inode pointing to it, across each
+outstanding RPC call that references it (LAYOUTGET, fs/nfs/LAYOUTRETURN,
+LAYOUTCOMMIT), and for each lseg held within.
+
+Each header is also (when non-empty) put on a list associated with
+struct nfs_client (cl_layouts).  Being put on this list does not bump
+the reference count, as the layout is kept around by the lseg that
+keeps it in the list.
+
+deviceid_cache
+--------------
+lsegs reference device ids, which are resolved per nfs_client and
+layout driver type.  The device ids are held in a RCU cache (struct
+nfs4_deviceid_cache).  The cache itself is referenced across each
+mount.  The entries (struct nfs4_deviceid) themselves are held across
+the lifetime of each lseg referencing them.
+
+RCU is used because the deviceid is basically a write once, read many
+data structure.  The hlist size of 32 buckets needs better
+justification, but seems reasonable given that we can have multiple
+deviceid's per filesystem, and multiple filesystems per nfs_client.
+
+The hash code is copied from the nfsd code base.  A discussion of
+hashing and variations of this algorithm can be found at:
+http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809
+
+data server cache
+-----------------
+file driver devices refer to data servers, which are kept in a module
+level cache.  Its reference is held over the lifetime of the deviceid
+pointing to it.
fs/nfs/Kconfig +1 -1
index 6c2aad4..5f1b936 100644
@@ -78,7 +78,7 @@ config NFS_V4_1
      depends on NFS_V4 && EXPERIMENTAL
      help
        This option enables support for minor version 1 of the NFSv4 protocol
-       (draft-ietf-nfsv4-minorversion1) in the kernel's NFS client.
+       (RFC 5661) in the kernel's NFS client.
 
        If unsure, say N.
 
fs/nfs/pnfs.c +9 -1
index 2e5dba1..5a8a676 100644
@@ -32,16 +32,53 @@
 
 #define NFSDBG_FACILITY              NFSDBG_PNFS
 
-/* STUB that returns the equivalent of "no module found" */
+/* Locking:
+ *
+ * pnfs_spinlock:
+ *      protects pnfs_modules_tbl.
+ */
+static DEFINE_SPINLOCK(pnfs_spinlock);
+
+/*
+ * pnfs_modules_tbl holds all pnfs modules
Quoted 8 lines

fs/nfs/pnfs.c line 49

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+ */
+static LIST_HEAD(pnfs_modules_tbl);
+
+/* Return the registered pnfs layout driver module matching given id */
+static struct pnfs_layoutdriver_type *
+find_pnfs_driver_locked(u32 id) {

nit: the curly brace should be moved down a line

OK

Quoted 8 lines

fs/nfs/pnfs.c line 52

Quoted 4 lines
fs/nfs/pnfs.c +3 -0
+     struct  pnfs_layoutdriver_type *local;
+
+     dprintk("PNFS: %s: Searching for %u\n", __func__, id);

I'd move this printk down, before returning and print the result as well.

OK

Quoted 5 lines
Quoted 4 lines
fs/nfs/pnfs.c +4 -0
+     list_for_each_entry(local, &pnfs_modules_tbl, pnfs_tblid)
+             if (local->id == id) {
+                     if (!try_module_get(local->owner))
+                             local = NULL;
Quoted 9 lines
Quoted 4 lines

If this happens (e.g. in the case the module exited without calling pnfs_unregister_layoutdriver) another (or a different instance) layout driver might have registered on the same id so we need to keep looking.

There can only be a single driver registered with a particular id. So continuing the search is pointless.

After this conditions happens, find_pnfs_driver_locked will not find the current one so next time pnfs_register_layoutdriver is called, it may register a new layout driver with the same id
and then there will be two entries with the same id. One, for which try_module_get returns zero and another one which you will never find if you don't keep looking for it.

Benny

Quoted 141 lines
Quoted 78 lines

fs/nfs/pnfs.c line 62

Quoted 7 lines
fs/nfs/pnfs.c +6 -0
+                     goto out;
+             }
+     local = NULL;
+out:
+     return local;
+}

how about the following?

static struct pnfs_layoutdriver_type *
find_pnfs_driver_locked(u32 id)

{
       struct pnfs_layoutdriver_type *local, *found = NULL;

       list_for_each_entry (local, &pnfs_modules_tbl, pnfs_tblid)
               if (local->id == id) {
                       if (!try_module_get(local->owner))
                               continue;
                       found = local;
                       break;
               }
out:
       dprintk("%s: layout_type %u found %p\n", __func__, id, found);
       return found;
}
Quoted 38 lines
fs/nfs/pnfs.c +18 -1
+
 static struct pnfs_layoutdriver_type *
 find_pnfs_driver(u32 id) {
-     return NULL;
+     struct  pnfs_layoutdriver_type *local;
+
+     spin_lock(&pnfs_spinlock);
+     local = find_pnfs_driver_locked(id);
+     spin_unlock(&pnfs_spinlock);
+     return local;
 }
 
 /* Unitialize a mountpoint in a layout driver */
 void
 unset_pnfs_layoutdriver(struct nfs_server *nfss)
 {
+     if (nfss->pnfs_curr_ld) {
+             nfss->pnfs_curr_ld->uninitialize_mountpoint(nfss);
+             module_put(nfss->pnfs_curr_ld->owner);
+     }
      nfss->pnfs_curr_ld = NULL;
 }
 
@@ -68,6 +105,13 @@ set_pnfs_layoutdriver(struct nfs_server *server, u32 id)
                      goto out_no_driver;
              }
      }
+     if (ld_type->initialize_mountpoint(server)) {
+             printk(KERN_ERR
+                    "%s: Error initializing mount point for layout driver %u.\n",
+                    __func__, id);
+             module_put(ld_type->owner);
+             goto out_no_driver;
+     }
      server->pnfs_curr_ld = ld_type;
      dprintk("%s: pNFS module for %u set\n", __func__, id);
      return;

fs/nfs/pnfs.c out_no_driver: · line 129

Quoted 10 lines
fs/nfs/pnfs.c +7 -0
@@ -76,3 +120,37 @@ out_no_driver:
      dprintk("%s: Using NFSv4 I/O\n", __func__);
      server->pnfs_curr_ld = NULL;
 }
+
+int
+pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+     int status = -EINVAL;
+     struct pnfs_layoutdriver_type *tmp;
+

Since we're relying on the fact the ld_type->id != 0 let's add

Quoted 3 lines
       BUG_ON(ld_type->id == 0);

Benny

OK.

Fred

Quoted 55 lines
Quoted 40 lines
fs/nfs/pnfs.c +27 -0
+     spin_lock(&pnfs_spinlock);
+     tmp = find_pnfs_driver_locked(ld_type->id);
+     if (!tmp) {
+             list_add(&ld_type->pnfs_tblid, &pnfs_modules_tbl);
+             status = 0;
+             dprintk("%s Registering id:%u name:%s\n", __func__, ld_type->id,
+                     ld_type->name);
+     } else {
+             module_put(tmp->owner);
+             printk(KERN_ERR "%s Module with id %d already loaded!\n",
+                     __func__, ld_type->id);
+     }
+     spin_unlock(&pnfs_spinlock);
+
+     return status;
+}
+EXPORT_SYMBOL_GPL(pnfs_register_layoutdriver);
+
+void
+pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *ld_type)
+{
+     dprintk("%s Deregistering id:%u\n", __func__, ld_type->id);
+     spin_lock(&pnfs_spinlock);
+     list_del(&ld_type->pnfs_tblid);
+     spin_unlock(&pnfs_spinlock);
+}
+EXPORT_SYMBOL_GPL(pnfs_unregister_layoutdriver);
fs/nfs/pnfs.h +4 -0
index c628ef1..61531f3 100644
@@ -36,8 +36,17 @@
 
 /* Per-layout driver specific registration structure */
 struct pnfs_layoutdriver_type {
+     struct list_head pnfs_tblid;
+     const u32 id;
+     const char *name;
+     struct module *owner;

fs/nfs/pnfs.h line 52

Quoted 10 lines
fs/nfs/pnfs.h +5 -0
+     int (*initialize_mountpoint) (struct nfs_server *);
+     int (*uninitialize_mountpoint) (struct nfs_server *);
 };
 
+extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
+extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
+
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

PATCH 8/12 RFC: pnfs: filelayout: introduce minimal file layout driver

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

This driver just registers itself and supplies trivial mount/umount functions.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile         |    3 ++
 fs/nfs/nfs4filelayout.c |   78 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfs_fs.h  |    1 +
 3 files changed, 82 insertions(+), 0 deletions(-)
 create mode 100644 fs/nfs/nfs4filelayout.c
fs/nfs/Makefile +3 -0
index bb9e773..631ffcb 100644
@@ -18,3 +18,6 @@ nfs-$(CONFIG_NFS_V4)	+= nfs4proc.o nfs4xdr.o nfs4state.o nfs4renewd.o \
 nfs-$(CONFIG_NFS_V4_1)	+= pnfs.o
 nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
+
+obj-$(CONFIG_NFS_V4_1) += nfs_layout_nfsv41_files.o
+nfs_layout_nfsv41_files-y := nfs4filelayout.o
fs/nfs/nfs4filelayout.c new file +78 -0

78 lines not on this page · raw message

include/linux/nfs_fs.h +1 -0
index aba3da2..499872f 100644
@@ -616,6 +616,7 @@ nfs_fileid_to_ino_t(u64 fileid)
 #define NFSDBG_MOUNT		0x0400
 #define NFSDBG_FSCACHE		0x0800
 #define NFSDBG_PNFS		0x1000
+#define NFSDBG_PNFS_LD		0x2000
 #define NFSDBG_ALL		0xFFFF
 
 #ifdef __KERNEL__
Signature · 1 line
1.7.2.1
#22 lore raw

PATCH 9/12 RFC: nfs: create and destroy inode's layout cache

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

At the start of the io paths, try to grab the relevant layout information. This will initiate the inode's layout cache, but stubs ensure the cache stays empty.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/file.c          |    5 ++
 fs/nfs/inode.c         |    3 +
 fs/nfs/pnfs.c          |  140 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.h          |   39 +++++++++++++
 fs/nfs/read.c          |    3 +
 include/linux/nfs_fs.h |    3 +
 6 files changed, 193 insertions(+), 0 deletions(-)
fs/nfs/file.c +5 -0
index eb51bd6..10ebdfb 100644
@@ -36,6 +36,7 @@
 #include "internal.h"
 #include "iostat.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_FILE
 
@@ -386,6 +387,10 @@ static int nfs_write_begin(struct file *file, struct address_space *mapping,
 		file->f_path.dentry->d_name.name,
 		mapping->host->i_ino, len, (long long) pos);
 
+	pnfs_update_layout(mapping->host,
+			   nfs_file_open_context(file),
+			   IOMODE_RW);
+
 start:
 	/*
 	 * Prevent starvation issues if someone is doing a consistency
fs/nfs/inode.c +3 -0
index 2f92664..effe985 100644
@@ -48,6 +48,7 @@
 #include "internal.h"
 #include "fscache.h"
 #include "dns_resolve.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_VFS
 
@@ -1409,6 +1410,7 @@ void nfs4_evict_inode(struct inode *inode)
 {
 	truncate_inode_pages(&inode->i_data, 0);
 	end_writeback(inode);
+	pnfs_destroy_layout(NFS_I(inode));
 	/* If we are holding a delegation, return it! */
 	nfs_inode_return_delegation_noreclaim(inode);
 	/* First call standard NFS clear_inode() code */
@@ -1446,6 +1448,7 @@ static inline void nfs4_init_once(struct nfs_inode *nfsi)
 	nfsi->delegation = NULL;
 	nfsi->delegation_state = 0;
 	init_rwsem(&nfsi->rwsem);
+	nfsi->layout = NULL;
 #endif
 }
 
fs/nfs/pnfs.c +140 -0

143 lines not on this page · raw message

fs/nfs/pnfs.h +39 -0
index 61531f3..4ed1b48 100644
@@ -34,6 +34,11 @@
 
 #define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
 
+enum {
+	NFS_LAYOUT_RO_FAILED = 0,	/* get ro layout failed stop trying */
+	NFS_LAYOUT_RW_FAILED,		/* get rw layout failed stop trying */
+};
+
 /* Per-layout driver specific registration structure */
 struct pnfs_layoutdriver_type {
 	struct list_head pnfs_tblid;
@@ -44,14 +49,48 @@ struct pnfs_layoutdriver_type {
 	int (*uninitialize_mountpoint) (struct nfs_server *);
 };
 
+struct pnfs_layout_hdr {
+	unsigned long		refcount;
+	unsigned long		state;
+	struct inode		*inode;
+};
+
 extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
 extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
 
+struct pnfs_layout_segment *
+pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
+		   enum pnfs_iomode access_type);
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
+void pnfs_destroy_layout(struct nfs_inode *);
+
+
+static inline int lo_fail_bit(u32 iomode)
+{
+	return iomode == IOMODE_RW ?
+			 NFS_LAYOUT_RW_FAILED : NFS_LAYOUT_RO_FAILED;
+}
+
+/* Return true if a layout driver is being used for this mountpoint */
+static inline int pnfs_enabled_sb(struct nfs_server *nfss)
+{
+	return nfss->pnfs_curr_ld != NULL;
+}
 
 #else  /* CONFIG_NFS_V4_1 */
 
+static inline void pnfs_destroy_layout(struct nfs_inode *nfsi)
+{
+}
+
+static inline struct pnfs_layout_segment *
+pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
+		   enum pnfs_iomode access_type)
+{
+	return NULL;
+}
+
 static inline void set_pnfs_layoutdriver(struct nfs_server *s, u32 id)
 {
 }
fs/nfs/read.c +3 -0
index 87adc27..f7eb66f 100644
@@ -25,6 +25,7 @@
 #include "internal.h"
 #include "iostat.h"
 #include "fscache.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_PAGECACHE
 
@@ -121,6 +122,7 @@ int nfs_readpage_async(struct nfs_open_context *ctx, struct inode *inode,
 	len = nfs_page_length(page);
 	if (len == 0)
 		return nfs_return_empty_page(page);
+	pnfs_update_layout(inode, ctx, IOMODE_READ);
 	new = nfs_create_request(ctx, inode, page, 0, len);
 	if (IS_ERR(new)) {
 		unlock_page(page);
@@ -625,6 +627,7 @@ int nfs_readpages(struct file *filp, struct address_space *mapping,
 	if (ret == 0)
 		goto read_complete; /* all pages were read */
 
+	pnfs_update_layout(inode, desc.ctx, IOMODE_READ);
 	if (rsize < PAGE_CACHE_SIZE)
 		nfs_pageio_init(&pgio, inode, nfs_pagein_multi, rsize, 0);
 	else
include/linux/nfs_fs.h +3 -0
index 499872f..0833bb6 100644
@@ -188,6 +188,9 @@ struct nfs_inode {
 	struct nfs_delegation	*delegation;
 	fmode_t			 delegation_state;
 	struct rw_semaphore	rwsem;
+
+	/* pNFS layout information */
+	struct pnfs_layout_hdr *layout;
 #endif /* CONFIG_NFS_V4*/
 #ifdef CONFIG_NFS_FSCACHE
 	struct fscache_cookie	*fscache;
Signature · 1 line
1.7.2.1
#23 lore raw

PATCH 10/12 RFC: nfs: client needs to maintain list of inodes with active layouts

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

In particular, server reboot will invalidate all layouts.

Note that in order to have an active layout, we must get a successful response from the server. To avoid adding that machinery, this patch just includes a stub that fakes up a successful return. Since the layout is never referenced for io, this is not a problem.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/client.c           |    4 +-
 fs/nfs/nfs4state.c        |    2 +
 fs/nfs/pnfs.c             |  131 +++++++++++++++++++++++++++++++++++++++++++-
 fs/nfs/pnfs.h             |   14 +++++
 include/linux/nfs_fs_sb.h |    1 +
 5 files changed, 148 insertions(+), 4 deletions(-)
fs/nfs/client.c +3 -1
index eed1212..6fc5c84 100644
@@ -156,7 +156,9 @@ static struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *cl_
 	cred = rpc_lookup_machine_cred();
 	if (!IS_ERR(cred))
 		clp->cl_machine_cred = cred;
-
+#if defined(CONFIG_NFS_V4_1)
+	INIT_LIST_HEAD(&clp->cl_layouts);
+#endif
 	nfs_fscache_get_client_cookie(clp);
 
 	return clp;
fs/nfs/nfs4state.c +2 -0
index 3e2f19b..b53a4ce 100644
@@ -53,6 +53,7 @@
 #include "callback.h"
 #include "delegation.h"
 #include "internal.h"
+#include "pnfs.h"
 
 #define OPENOWNER_POOL_SIZE	8
 
@@ -1447,6 +1448,7 @@ static void nfs4_state_manager(struct nfs_client *clp)
 			}
 			clear_bit(NFS4CLNT_CHECK_LEASE, &clp->cl_state);
 			set_bit(NFS4CLNT_RECLAIM_REBOOT, &clp->cl_state);
+			pnfs_destroy_all_layouts(clp);
 		}
 
 		if (test_and_clear_bit(NFS4CLNT_CHECK_LEASE, &clp->cl_state)) {
fs/nfs/pnfs.c +128 -3

176 lines not on this page · raw message

fs/nfs/pnfs.h +14 -0
index 4ed1b48..1c3eb02 100644
@@ -30,6 +30,13 @@
 #ifndef FS_NFS_PNFS_H
 #define FS_NFS_PNFS_H
 
+struct pnfs_layout_segment {
+	struct list_head fi_list;
+	u32 iomode;
+	struct kref kref;
+	struct pnfs_layout_hdr *layout;
+};
+
 #ifdef CONFIG_NFS_V4_1
 
 #define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
@@ -51,6 +58,8 @@ struct pnfs_layoutdriver_type {
 
 struct pnfs_layout_hdr {
 	unsigned long		refcount;
+	struct list_head	layouts;   /* other client layouts */
+	struct list_head	segs;      /* layout segments list */
 	unsigned long		state;
 	struct inode		*inode;
 };
@@ -64,6 +73,7 @@ pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
 void pnfs_destroy_layout(struct nfs_inode *);
+void pnfs_destroy_all_layouts(struct nfs_client *);
 
 
 static inline int lo_fail_bit(u32 iomode)
@@ -80,6 +90,10 @@ static inline int pnfs_enabled_sb(struct nfs_server *nfss)
 
 #else  /* CONFIG_NFS_V4_1 */
 
+static inline void pnfs_destroy_all_layouts(struct nfs_client *clp)
+{
+}
+
 static inline void pnfs_destroy_layout(struct nfs_inode *nfsi)
 {
 }
include/linux/nfs_fs_sb.h +1 -0
index 29a821d..e670a9c 100644
@@ -82,6 +82,7 @@ struct nfs_client {
 	/* The flags used for obtaining the clientid during EXCHANGE_ID */
 	u32			cl_exchange_flags;
 	struct nfs4_session	*cl_session; 	/* sharred session */
+	struct list_head	cl_layouts;
 #endif /* CONFIG_NFS_V4_1 */
 
 #ifdef CONFIG_NFS_FSCACHE
Signature · 1 line
1.7.2.1
#24 lore raw

PATCH 11/12 RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

Add the ability to actually send LAYOUTGET and GETDEVICEINFO. This also adds in the machinery to handle layout state and the deviceid cache. Note that GETDEVICEINFO is not called directly by the generic layer. Instead it is called by the drivers while parsing the LAYOUTGET opaque data in response to an unknown device id embedded therein. Annoyingly, RFC 5661 only encodes device ids within the driver-specific opaque data.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/nfs4proc.c         |  150 ++++++++++++++++++
 fs/nfs/nfs4xdr.c          |  302 ++++++++++++++++++++++++++++++++++++
 fs/nfs/pnfs.c             |  376 +++++++++++++++++++++++++++++++++++++++++----
 fs/nfs/pnfs.h             |   81 ++++++++++-
 include/linux/nfs4.h      |    2 +
 include/linux/nfs_fs_sb.h |    1 +
 include/linux/nfs_xdr.h   |   49 ++++++
 7 files changed, 933 insertions(+), 28 deletions(-)
fs/nfs/nfs4proc.c +150 -0
index c0d9ff4..83745b1 100644
@@ -55,6 +55,7 @@
 #include "internal.h"
 #include "iostat.h"
 #include "callback.h"
+#include "pnfs.h"
 
 #define NFSDBG_FACILITY		NFSDBG_PROC
 
@@ -5256,6 +5257,155 @@ out:
 	dprintk("<-- %s status=%d\n", __func__, status);
 	return status;
 }
+
+static void
+nfs4_layoutget_prepare(struct rpc_task *task, void *calldata)
+{
+	struct nfs4_layoutget *lgp = calldata;
+	struct inode *ino = lgp->args.inode;
+	struct nfs_server *server = NFS_SERVER(ino);
+
+	dprintk("--> %s\n", __func__);
+	if (nfs4_setup_sequence(server, &lgp->args.seq_args,
+				&lgp->res.seq_res, 0, task))
+		return;
+	rpc_call_start(task);
+}
+
+static void nfs4_layoutget_done(struct rpc_task *task, void *calldata)
+{
+	struct nfs4_layoutget *lgp = calldata;
+
+	dprintk("--> %s\n", __func__);
+
+	if (!nfs4_sequence_done(task, &lgp->res.seq_res))
+		return;
+
+	/* Error handling done later using nfs4_handle_exception to get
+	 * exponential backoff.
+	 */
+	lgp->status = task->tk_status;
+	dprintk("<-- %s\n", __func__);
+}
+
+static void nfs4_layoutget_release(void *calldata)
+{
+	struct nfs4_layoutget *lgp = calldata;
+
+	dprintk("--> %s\n", __func__);
+	put_layout_hdr(lgp->args.inode);
+	if (lgp->res.layout.buf != NULL)
+		free_page((unsigned long) lgp->res.layout.buf);
+	put_nfs_open_context(lgp->args.ctx);
+	kfree(calldata);
+	dprintk("<-- %s\n", __func__);
+}
+
+static const struct rpc_call_ops nfs4_layoutget_call_ops = {
+	.rpc_call_prepare = nfs4_layoutget_prepare,
+	.rpc_call_done = nfs4_layoutget_done,
+	.rpc_release = nfs4_layoutget_release,
+};
+
+static int _nfs4_proc_layoutget(struct nfs4_layoutget *lgp)
+{
+	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
+	struct rpc_task *task;
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_LAYOUTGET],
+		.rpc_argp = &lgp->args,
+		.rpc_resp = &lgp->res,
+	};
+	struct rpc_task_setup task_setup_data = {
+		.rpc_client = server->client,
+		.rpc_message = &msg,
+		.callback_ops = &nfs4_layoutget_call_ops,
+		.callback_data = lgp,
+		.flags = RPC_TASK_ASYNC,
+	};
+	int status = 0;
+
+	dprintk("--> %s\n", __func__);
+
+	lgp->res.layout.buf = (void *)__get_free_page(GFP_NOFS);
+	if (lgp->res.layout.buf == NULL) {
+		nfs4_layoutget_release(lgp);
+		return -ENOMEM;
+	}
+
+	lgp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE;
+	task = rpc_run_task(&task_setup_data);
+	if (IS_ERR(task))
+		return PTR_ERR(task);
+	status = nfs4_wait_for_completion_rpc_task(task);
+	if (status != 0)
+		goto out;
+	status = lgp->status;
+	if (status != 0)
+		goto out;
+	status = pnfs_layout_process(lgp);
+out:
+	rpc_put_task(task);
+	dprintk("<-- %s status=%d\n", __func__, status);
+	return status;
+}
+
+int nfs4_proc_layoutget(struct nfs4_layoutget *lgp)
+{
+	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
+	struct nfs4_exception exception = { };
+	int err;
+	do {
+		err = _nfs4_proc_layoutget(lgp);
+		switch (err) {
+		case -NFS4ERR_LAYOUTTRYLATER:
+		case -NFS4ERR_RECALLCONFLICT:
+			err = -NFS4ERR_DELAY;
+			/* Fall through */
+		default:
+			err = nfs4_handle_exception(server, err, &exception);
+		}
+	} while (exception.retry);
+	return err;
+}
+
+static int
+_nfs4_proc_getdeviceinfo(struct nfs_server *server, struct pnfs_device *pdev)
+{
+	struct nfs4_getdeviceinfo_args args = {
+		.pdev = pdev,
+	};
+	struct nfs4_getdeviceinfo_res res = {
+		.pdev = pdev,
+	};
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_GETDEVICEINFO],
+		.rpc_argp = &args,
+		.rpc_resp = &res,
+	};
+	int status;
+
+	dprintk("--> %s\n", __func__);
+	status = nfs4_call_sync(server, &msg, &args, &res, 0);
+	dprintk("<-- %s status=%d\n", __func__, status);
+
+	return status;
+}
+
+int nfs4_proc_getdeviceinfo(struct nfs_server *server, struct pnfs_device *pdev)
+{
+	struct nfs4_exception exception = { };
+	int err;
+
+	do {
+		err = nfs4_handle_exception(server,
+					_nfs4_proc_getdeviceinfo(server, pdev),
+					&exception);
+	} while (exception.retry);
+	return err;
+}
+EXPORT_SYMBOL_GPL(nfs4_proc_getdeviceinfo);
+
 #endif /* CONFIG_NFS_V4_1 */
 
 struct nfs4_state_recovery_ops nfs40_reboot_recovery_ops = {
fs/nfs/nfs4xdr.c +302 -0

350 lines not on this page · raw message

fs/nfs/pnfs.c +349 -27

461 lines not on this page · raw message

fs/nfs/pnfs.h +80 -1

123 lines not on this page · raw message

include/linux/nfs4.h +2 -0
index 2dde7c8..dcdd11c 100644
@@ -545,6 +545,8 @@ enum {
 	NFSPROC4_CLNT_SEQUENCE,
 	NFSPROC4_CLNT_GET_LEASE_TIME,
 	NFSPROC4_CLNT_RECLAIM_COMPLETE,
+	NFSPROC4_CLNT_LAYOUTGET,
+	NFSPROC4_CLNT_GETDEVICEINFO,
 };
 
 /* nfs41 types */
include/linux/nfs_fs_sb.h +1 -0
index e670a9c..7512886 100644
@@ -83,6 +83,7 @@ struct nfs_client {
 	u32			cl_exchange_flags;
 	struct nfs4_session	*cl_session; 	/* sharred session */
 	struct list_head	cl_layouts;
+	struct nfs4_deviceid_cache *cl_devid_cache; /* pNFS deviceid cache */
 #endif /* CONFIG_NFS_V4_1 */
 
 #ifdef CONFIG_NFS_FSCACHE
include/linux/nfs_xdr.h +49 -0

55 lines not on this page · raw message

Signature · 1 line
1.7.2.1
#25 lore raw

Re: [PATCH 11/12] RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

Quoted 9 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Add the ability to actually send LAYOUTGET and GETDEVICEINFO. This also adds in the machinery to handle layout state and the deviceid cache. Note that GETDEVICEINFO is not called directly by the generic layer. Instead it is called by the drivers while parsing the LAYOUTGET opaque data in response to an unknown device id embedded therein. Annoyingly, RFC 5661 only encodes device ids within the driver-specific opaque data.

Fred, In regard to device cache at least, this is a complete new code then what I knew before. I wish you could send a SQUASHME diff first of what has changed.

Quoted 1 line
Signed-off-by TBD - melding/reorganization of several patches

<snip>

Quoted 52 lines
fs/nfs/pnfs.c +52 -0
+/*
+ * Device ID cache. Currently supports one layout type per struct nfs_client.
+ * Add layout type to the lookup key to expand to support multiple types.
+ */
+int
+nfs4_alloc_init_deviceid_cache(struct nfs_client *clp,
+			 void (*free_callback)(struct nfs4_deviceid *))
+{
+	struct nfs4_deviceid_cache *c;
+
+	c = kzalloc(sizeof(struct nfs4_deviceid_cache), GFP_KERNEL);
+	if (!c)
+		return -ENOMEM;
+	spin_lock(&clp->cl_lock);
+	if (clp->cl_devid_cache != NULL) {
+		atomic_inc(&clp->cl_devid_cache->dc_ref);
+		dprintk("%s [kref [%d]]\n", __func__,
+			atomic_read(&clp->cl_devid_cache->dc_ref));
+		kfree(c);
+	} else {
+		/* kzalloc initializes hlists */
+		spin_lock_init(&c->dc_lock);
+		atomic_set(&c->dc_ref, 1);
+		c->dc_free_callback = free_callback;
+		clp->cl_devid_cache = c;
+		dprintk("%s [new]\n", __func__);
+	}
+	spin_unlock(&clp->cl_lock);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nfs4_alloc_init_deviceid_cache);
+
+void
+nfs4_init_deviceid_node(struct nfs4_deviceid *d)
+{
+	INIT_HLIST_NODE(&d->de_node);
+	atomic_set(&d->de_ref, 1);
+}
+EXPORT_SYMBOL_GPL(nfs4_init_deviceid_node);
+
+/*
+ * Called from pnfs_layoutdriver_type->free_lseg
+ * last layout segment reference frees deviceid
+ */
+void
+nfs4_put_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *devid)
+{
+	struct pnfs_deviceid *id = &devid->de_id;
+	struct nfs4_deviceid *d;
+	struct hlist_node *n;
+	long h = nfs4_deviceid_hash(id);
+

fs/nfs/pnfs.c out_unlock: · line 670

Quoted 10 lines
fs/nfs/pnfs.c +10 -0
+	dprintk("%s [%d]\n", __func__, atomic_read(&devid->de_ref));
+	if (!atomic_dec_and_lock(&devid->de_ref, &c->dc_lock))
+		return;
+
+	hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[h], de_node)
+		if (!memcmp(&d->de_id, id, sizeof(*id))) {
+			hlist_del_rcu(&d->de_node);
+			spin_unlock(&c->dc_lock);
+			synchronize_rcu();
+			c->dc_free_callback(devid);

What?!? free the devid on every put? surly something went hey-wire

This used to be kref(ed)

Quoted 41 lines
fs/nfs/pnfs.c +41 -0
+			return;
+		}
+	spin_unlock(&c->dc_lock);
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid);
+
+/* Find and reference a deviceid */
+struct nfs4_deviceid *
+nfs4_find_get_deviceid(struct nfs4_deviceid_cache *c, struct pnfs_deviceid *id)
+{
+	struct nfs4_deviceid *d;
+	struct hlist_node *n;
+	long hash = nfs4_deviceid_hash(id);
+
+	dprintk("--> %s hash %ld\n", __func__, hash);
+	rcu_read_lock();
+	hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+		if (!memcmp(&d->de_id, id, sizeof(*id))) {
+			if (!atomic_inc_not_zero(&d->de_ref)) {
+				goto fail;
+			} else {
+				rcu_read_unlock();
+				return d;
+			}
+		}
+	}
+fail:
+	rcu_read_unlock();
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nfs4_find_get_deviceid);
+
+/*
+ * Add a deviceid to the cache.
+ * GETDEVICEINFOs for same deviceid can race. If deviceid is found, discard new
+ */
+struct nfs4_deviceid *
+nfs4_add_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *new)
+{
+	struct nfs4_deviceid *d;
+	struct hlist_node *n;

fs/nfs/pnfs.c out_unlock: · line 721

Quoted 10 lines
fs/nfs/pnfs.c +10 -0
+	long hash = nfs4_deviceid_hash(&new->de_id);
+
+	dprintk("--> %s hash %ld\n", __func__, hash);
+	spin_lock(&c->dc_lock);
+	hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+		if (!memcmp(&d->de_id, &new->de_id, sizeof(new->de_id))) {
+			spin_unlock(&c->dc_lock);
+			dprintk("%s [discard]\n", __func__);
+			c->dc_free_callback(new);
+			return d;

I am totally confused and should be smacked on the head strong. As said above one alloc_lseg adds the first, second comes in, but same deviceid, finds the first and returns it (no refcounting). Then a first free_lseg nfs4_put_deviceid is called, so the second lseg holds free memory?

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+		}
+	}
+	hlist_add_head_rcu(&new->de_node, &c->dc_deviceids[hash]);
+	spin_unlock(&c->dc_lock);
+	dprintk("%s [new]\n", __func__);
+	return new;

fs/nfs/pnfs.c out_unlock: · line 737

Quoted 10 lines
fs/nfs/pnfs.c +10 -0
+}
+EXPORT_SYMBOL_GPL(nfs4_add_deviceid);
+
+void
+nfs4_put_deviceid_cache(struct nfs_client *clp)
+{
+	struct nfs4_deviceid_cache *local = clp->cl_devid_cache;
+
+	dprintk("--> %s cl_devid_cache %p\n", __func__, clp->cl_devid_cache);
+	if (atomic_dec_and_lock(&local->dc_ref, &clp->cl_lock)) {

Don't you want to make sure there are no hanging devices on this cache. Please note that an add or find does not take the cache reference only server_init does, right?

fs/nfs/pnfs.c out_unlock: · line 743

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+		clp->cl_devid_cache = NULL;
+		spin_unlock(&clp->cl_lock);
+		kfree(local);
+	}
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid_cache);

Note to self. Test code alot.

Fred:
I don't understand. A cache property I'm looking for and which I do not See in this code, perhaps because I'm clueless.
(And to be honest I have not understood it before as well, since I never
actually ran with that code I posted.)
Is:
- An _add_ to queue should magically hold two references to the item, so
- A matching _put_ does not deallocate the item but keeps it referenced once.
- A find/put pair does not remove the item from cache as well (+=1/-=1)
- [optional] At some future out-of-memory condition or when cache is to big, according to some specified max cache. The most unused items are dereferenced and if are not used (ref==0) are then removed.
- At driver shut-down all cached devices are dereferenced (and if not used which they should) are dealocated.

So in effect if [optional] code above is not yet implemented then a getdeviceinfo is never preformed twice for the same deviceid (Assuming no device recall). This is what I had in osd, and is cardinal for it's performance. Is that what we are aiming at?

Quoted 36 lines
fs/nfs/pnfs.h +6 -1
index 1c3eb02..b7de762 100644
@@ -32,11 +32,13 @@
 
 struct pnfs_layout_segment {
 	struct list_head fi_list;
-	u32 iomode;
+	struct pnfs_layout_range range;
 	struct kref kref;
 	struct pnfs_layout_hdr *layout;
 };
 
+#define NFS4_PNFS_DEVICEID4_SIZE 16
+
 #ifdef CONFIG_NFS_V4_1
 
 #define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
@@ -44,6 +46,7 @@ struct pnfs_layout_segment {
 enum {
 	NFS_LAYOUT_RO_FAILED = 0,	/* get ro layout failed stop trying */
 	NFS_LAYOUT_RW_FAILED,		/* get rw layout failed stop trying */
+	NFS_LAYOUT_STATEID_SET,		/* have a valid layout stateid */
 };
 
 /* Per-layout driver specific registration structure */
@@ -54,26 +57,102 @@ struct pnfs_layoutdriver_type {
 	struct module *owner;
 	int (*initialize_mountpoint) (struct nfs_server *);
 	int (*uninitialize_mountpoint) (struct nfs_server *);
+	struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_hdr *layoutid, struct nfs4_layoutget_res *lgr);
+	void (*free_lseg) (struct pnfs_layout_segment *lseg);
 };
 
 struct pnfs_layout_hdr {

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 74

Quoted 10 lines
fs/nfs/pnfs.h +3 -0
 	unsigned long		refcount;
 	struct list_head	layouts;   /* other client layouts */
 	struct list_head	segs;      /* layout segments list */
+	seqlock_t		seqlock;   /* Protects the stateid */
+	nfs4_stateid		stateid;
 	unsigned long		state;
 	struct inode		*inode;
 };
 
+struct pnfs_deviceid {

Please for my own sanity, could we change those names. I'll post a patch once we settle on final code. I want this to be nfs4_deviceid (And perhaps in that other header)

Quoted 26 lines
fs/nfs/pnfs.h +26 -0
+	char data[NFS4_PNFS_DEVICEID4_SIZE];
+};
+
+struct pnfs_device {
+	struct pnfs_deviceid dev_id;
+	unsigned int  layout_type;
+	unsigned int  mincount;
+	struct page **pages;
+	void          *area;
+	unsigned int  pgbase;
+	unsigned int  pglen;
+};
+
+/*
+ * Device ID RCU cache. A device ID is unique per client ID and layout type.
+ */
+#define NFS4_DEVICE_ID_HASH_BITS	5
+#define NFS4_DEVICE_ID_HASH_SIZE	(1 << NFS4_DEVICE_ID_HASH_BITS)
+#define NFS4_DEVICE_ID_HASH_MASK	(NFS4_DEVICE_ID_HASH_SIZE - 1)
+
+static inline u32
+nfs4_deviceid_hash(struct pnfs_deviceid *id)
+{
+	unsigned char *cptr = (unsigned char *)id->data;
+	unsigned int nbytes = NFS4_PNFS_DEVICEID4_SIZE;
+	u32 x = 0;

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 110

Quoted 10 lines
fs/nfs/pnfs.h +10 -0
+
+	while (nbytes--) {
+		x *= 37;
+		x += *cptr++;
+	}
+	return x & NFS4_DEVICE_ID_HASH_MASK;
+}
+
+/* Device ID cache node */
+struct nfs4_deviceid {

And this one rename to pnfs_deviceid_node

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 123

Quoted 13 lines
fs/nfs/pnfs.h +13 -0
+	struct hlist_node	de_node;
+	struct pnfs_deviceid	de_id;
+	atomic_t		de_ref;
+};
+
+struct nfs4_deviceid_cache {
+	spinlock_t		dc_lock;
+	atomic_t		dc_ref;
+	void			(*dc_free_callback)(struct nfs4_deviceid *);
+	struct hlist_head	dc_deviceids[NFS4_DEVICE_ID_HASH_SIZE];
+	struct hlist_head	dc_to_free;
+};
+

All these cache APIs below, the naming convention in this header is to call them pnfs_xxx. They don't have any use in general nfs4. (and are implemented in pnfs.c)

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 127

Quoted 4 lines
fs/nfs/pnfs.h +4 -0
+extern int nfs4_alloc_init_deviceid_cache(struct nfs_client *,
+				void (*free_callback)(struct nfs4_deviceid *));
+extern void nfs4_put_deviceid_cache(struct nfs_client *);
+extern void nfs4_init_deviceid_node(struct nfs4_deviceid *);

Why is above exported? Can't we just call it from _add_

Boaz

Quoted 116 lines
fs/nfs/pnfs.h +18 -0
+extern struct nfs4_deviceid *nfs4_find_get_deviceid(
+				struct nfs4_deviceid_cache *,
+				struct pnfs_deviceid *);
+extern struct nfs4_deviceid *nfs4_add_deviceid(struct nfs4_deviceid_cache *,
+				struct nfs4_deviceid *);
+extern void nfs4_put_deviceid(struct nfs4_deviceid_cache *c,
+			      struct nfs4_deviceid *devid);
+
 extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
 extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
 
+/* nfs4proc.c */
+extern int nfs4_proc_getdeviceinfo(struct nfs_server *server,
+				   struct pnfs_device *dev);
+extern int nfs4_proc_layoutget(struct nfs4_layoutget *lgp);
+
+/* pnfs.c */
 struct pnfs_layout_segment *
 pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
 		   enum pnfs_iomode access_type);
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
+int pnfs_layout_process(struct nfs4_layoutget *lgp);
 void pnfs_destroy_layout(struct nfs_inode *);
 void pnfs_destroy_all_layouts(struct nfs_client *);
+void put_layout_hdr(struct inode *inode);
+void pnfs_get_layout_stateid(nfs4_stateid *dst, struct pnfs_layout_hdr *lo,
+			     struct nfs4_state *open_state);
 
 
 static inline int lo_fail_bit(u32 iomode)
include/linux/nfs4.h +2 -0
index 2dde7c8..dcdd11c 100644
@@ -545,6 +545,8 @@ enum {
 	NFSPROC4_CLNT_SEQUENCE,
 	NFSPROC4_CLNT_GET_LEASE_TIME,
 	NFSPROC4_CLNT_RECLAIM_COMPLETE,
+	NFSPROC4_CLNT_LAYOUTGET,
+	NFSPROC4_CLNT_GETDEVICEINFO,
 };
 
 /* nfs41 types */
include/linux/nfs_fs_sb.h +1 -0
index e670a9c..7512886 100644
@@ -83,6 +83,7 @@ struct nfs_client {
 	u32			cl_exchange_flags;
 	struct nfs4_session	*cl_session; 	/* sharred session */
 	struct list_head	cl_layouts;
+	struct nfs4_deviceid_cache *cl_devid_cache; /* pNFS deviceid cache */
 #endif /* CONFIG_NFS_V4_1 */
 
 #ifdef CONFIG_NFS_FSCACHE
include/linux/nfs_xdr.h +49 -0
index 7d68a5c..4953b58 100644
@@ -186,6 +186,55 @@ struct nfs4_get_lease_time_res {
 	struct nfs4_sequence_res	lr_seq_res;
 };
 
+#define PNFS_LAYOUT_MAXSIZE 4096
+
+struct nfs4_layoutdriver_data {
+	__u32 len;
+	void *buf;
+};
+
+struct pnfs_layout_range {
+	u32 iomode;
+	u64 offset;
+	u64 length;
+};
+
+struct nfs4_layoutget_args {
+	__u32 type;
+	struct pnfs_layout_range range;
+	__u64 minlength;
+	__u32 maxcount;
+	struct inode *inode;
+	struct nfs_open_context *ctx;
+	struct nfs4_sequence_args seq_args;
+};
+
+struct nfs4_layoutget_res {
+	__u32 return_on_close;
+	struct pnfs_layout_range range;
+	__u32 type;
+	nfs4_stateid stateid;
+	struct nfs4_layoutdriver_data layout;
+	struct nfs4_sequence_res seq_res;
+};
+
+struct nfs4_layoutget {
+	struct nfs4_layoutget_args args;
+	struct nfs4_layoutget_res res;
+	struct pnfs_layout_segment **lsegpp;
+	int status;
+};
+
+struct nfs4_getdeviceinfo_args {
+	struct pnfs_device *pdev;
+	struct nfs4_sequence_args seq_args;
+};
+
+struct nfs4_getdeviceinfo_res {
+	struct pnfs_device *pdev;
+	struct nfs4_sequence_res seq_res;
+};
+
 /*
  * Arguments to the open call.
  */
#26 lore raw

Re: [PATCH 11/12] RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

Quoted 11 lines · Boaz Harrosh
Quoted 9 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Add the ability to actually send LAYOUTGET and GETDEVICEINFO.  This also adds in the machinery to handle layout state and the deviceid cache.  Note that GETDEVICEINFO is not called directly by the generic layer.  Instead it is called by the drivers while parsing the LAYOUTGET opaque data in response to an unknown device id embedded therein.  Annoyingly, RFC 5661 only encodes device ids within the driver-specific opaque data.

Quoted 3 lines

Fred, In regard to device cache at least, this is a complete new code then what I knew before. I wish you could send a SQUASHME diff first of what has changed.

Many of these changes were sent off-list to you in the initial format of original-code plus squashmes. In particular,
there was a patch labeled "SQUASHME pnfs_submit_1 fix pnfs_submit:
generic pnfs deviceid cache"

Quoted 60 lines
Quoted 1 line
Signed-off-by TBD - melding/reorganization of several patches

<snip>

Quoted 56 lines
fs/nfs/pnfs.c +56 -0
+/*
+ * Device ID cache. Currently supports one layout type per struct nfs_client.
+ * Add layout type to the lookup key to expand to support multiple types.
+ */
+int
+nfs4_alloc_init_deviceid_cache(struct nfs_client *clp,
+                      void (*free_callback)(struct nfs4_deviceid *))
+{
+     struct nfs4_deviceid_cache *c;
+
+     c = kzalloc(sizeof(struct nfs4_deviceid_cache), GFP_KERNEL);
+     if (!c)
+             return -ENOMEM;
+     spin_lock(&clp->cl_lock);
+     if (clp->cl_devid_cache != NULL) {
+             atomic_inc(&clp->cl_devid_cache->dc_ref);
+             dprintk("%s [kref [%d]]\n", __func__,
+                     atomic_read(&clp->cl_devid_cache->dc_ref));
+             kfree(c);
+     } else {
+             /* kzalloc initializes hlists */
+             spin_lock_init(&c->dc_lock);
+             atomic_set(&c->dc_ref, 1);
+             c->dc_free_callback = free_callback;
+             clp->cl_devid_cache = c;
+             dprintk("%s [new]\n", __func__);
+     }
+     spin_unlock(&clp->cl_lock);
+     return 0;
+}
+EXPORT_SYMBOL_GPL(nfs4_alloc_init_deviceid_cache);
+
+void
+nfs4_init_deviceid_node(struct nfs4_deviceid *d)
+{
+     INIT_HLIST_NODE(&d->de_node);
+     atomic_set(&d->de_ref, 1);
+}
+EXPORT_SYMBOL_GPL(nfs4_init_deviceid_node);
+
+/*
+ * Called from pnfs_layoutdriver_type->free_lseg
+ * last layout segment reference frees deviceid
+ */
+void
+nfs4_put_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *devid)
+{
+     struct pnfs_deviceid *id = &devid->de_id;
+     struct nfs4_deviceid *d;
+     struct hlist_node *n;
+     long h = nfs4_deviceid_hash(id);
+
+     dprintk("%s [%d]\n", __func__, atomic_read(&devid->de_ref));
+     if (!atomic_dec_and_lock(&devid->de_ref, &c->dc_lock))
+             return;
+
Quoted 9 lines
Quoted 6 lines

+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[h], de_node)
+             if (!memcmp(&d->de_id, id, sizeof(*id))) {
+                     hlist_del_rcu(&d->de_node);
+                     spin_unlock(&c->dc_lock);
+                     synchronize_rcu();
+                     c->dc_free_callback(devid);

What?!? free the devid on every put? surly something went hey-wire

Did you miss the atomic_dec_and_lock above?

Quoted 2 lines

This used to be kref(ed)

The original code required access to the internals of the kref, which was frowned upon.

Quoted 48 lines
Quoted 48 lines

+                     return;
+             }
+     spin_unlock(&c->dc_lock);

fs/nfs/pnfs.c +45 -0
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid);
+
+/* Find and reference a deviceid */
+struct nfs4_deviceid *
+nfs4_find_get_deviceid(struct nfs4_deviceid_cache *c, struct pnfs_deviceid *id)
+{
+     struct nfs4_deviceid *d;
+     struct hlist_node *n;
+     long hash = nfs4_deviceid_hash(id);
+
+     dprintk("--> %s hash %ld\n", __func__, hash);
+     rcu_read_lock();
+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+             if (!memcmp(&d->de_id, id, sizeof(*id))) {
+                     if (!atomic_inc_not_zero(&d->de_ref)) {
+                             goto fail;
+                     } else {
+                             rcu_read_unlock();
+                             return d;
+                     }
+             }
+     }
+fail:
+     rcu_read_unlock();
+     return NULL;
+}
+EXPORT_SYMBOL_GPL(nfs4_find_get_deviceid);
+
+/*
+ * Add a deviceid to the cache.
+ * GETDEVICEINFOs for same deviceid can race. If deviceid is found, discard new
+ */
+struct nfs4_deviceid *
+nfs4_add_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *new)
+{
+     struct nfs4_deviceid *d;
+     struct hlist_node *n;
+     long hash = nfs4_deviceid_hash(&new->de_id);
+
+     dprintk("--> %s hash %ld\n", __func__, hash);
+     spin_lock(&c->dc_lock);
+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+             if (!memcmp(&d->de_id, &new->de_id, sizeof(new->de_id))) {
+                     spin_unlock(&c->dc_lock);
Quoted 9 lines
Quoted 3 lines

+                     dprintk("%s [discard]\n", __func__);
+                     c->dc_free_callback(new);
+                     return d;

I am totally confused and should be smacked on the head strong. As said above one alloc_lseg adds the first, second comes in, but same deviceid, finds the first and returns it (no refcounting). Then a first free_lseg nfs4_put_deviceid is called, so the second lseg holds free memory?

You are right, there is a missing increment of the refcount here.

Part of the problem here is that the flow is obscurd because it is being called form the driver layer.
I think it might be better if nfs4_find_get_deviceid did the GETDEVICEINFO on a cache miss, instead of relying on the driver to do it.

Quoted 12 lines
Quoted 12 lines

+             }
+     }
+     hlist_add_head_rcu(&new->de_node, &c->dc_deviceids[hash]);
+     spin_unlock(&c->dc_lock);
+     dprintk("%s [new]\n", __func__);
+     return new;

fs/nfs/pnfs.c +6 -0
+}
+EXPORT_SYMBOL_GPL(nfs4_add_deviceid);
+
+void
+nfs4_put_deviceid_cache(struct nfs_client *clp)
+{
Quoted 9 lines
Quoted 4 lines

+     struct nfs4_deviceid_cache *local = clp->cl_devid_cache; +
+     dprintk("--> %s cl_devid_cache %p\n", __func__, clp->cl_devid_cache);
+     if (atomic_dec_and_lock(&local->dc_ref, &clp->cl_lock)) {

Don't you want to make sure there are no hanging devices on this cache. Please note that an add or find does not take the cache reference only server_init does, right?

If there are any hanging devices at this point, it is a pretty serious bug.

Quoted 9 lines

fs/nfs/pnfs.c out_unlock: · line 743

Quoted 6 lines

+             clp->cl_devid_cache = NULL;
+             spin_unlock(&clp->cl_lock);
+             kfree(local);
+     }

fs/nfs/pnfs.c +2 -0
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid_cache);

Note to self. Test code alot.

Quoted 15 lines

Fred:
 I don't understand. A cache property I'm looking for and which I do not See in this code, perhaps because I'm clueless.
 (And to be honest I have not understood it before as well, since I never actually ran with that code I posted.)
 Is:
 - An _add_ to queue should magically hold two references to the item, so
 - A matching _put_ does not deallocate the item but keeps it referenced once.
 - A find/put pair does not remove the item from cache as well (+=1/-=1)
 - [optional] At some future out-of-memory condition or when cache is to big, according to some specified max cache. The most unused items are dereferenced and if are not used (ref==0) are then removed.
 - At driver shut-down all cached devices are dereferenced (and if not used which they should) are dealocated.

I agree. The original deviceid code had an issue where nothing was ever deallocated until shutdown. The code as given here ties the deviceid lifetime to existing references. This is not ideal, because a brief loss of reference to a deviceid will cause an unnecessary GETDEVICEINFO. However, for the current submission, it has the advantage that it is simple and correct. Adding the machinery to do as you suggest above is indeed a (fairly high) priority goal, but was intentionally deferred.

Quoted 5 lines

So in effect if [optional] code above is not yet implemented then a getdeviceinfo is never preformed twice for the same deviceid (Assuming no device recall). This is what I had in osd, and is cardinal for it's performance. Is that what we are aiming at?

Yes, that is the goal. Right now, if a layout is still around that references a deviceid, a second GETDEVICEINFO will not be sent. However, if all layouts referencing a specific deviceid have been returned, then yes, a second GETDEVICEINFO will be called.

Quoted 100 lines
Quoted 46 lines
fs/nfs/pnfs.h +3 -1
index 1c3eb02..b7de762 100644
@@ -32,11 +32,13 @@
 
  struct pnfs_layout_segment {
       struct list_head fi_list;
-     u32 iomode;
+     struct pnfs_layout_range range;
       struct kref kref;
       struct pnfs_layout_hdr *layout;
  };
 
+#define NFS4_PNFS_DEVICEID4_SIZE 16
+
  #ifdef CONFIG_NFS_V4_1
 
  #define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
@@ -44,6 +46,7 @@ struct pnfs_layout_segment {
  enum {

      NFS_LAYOUT_RO_FAILED = 0,       /* get ro layout failed stop trying */ NFS_LAYOUT_RW_FAILED,           /* get rw layout failed stop trying */
+     NFS_LAYOUT_STATEID_SET,         /* have a valid layout stateid */ };

 /* Per-layout driver specific registration structure */

fs/nfs/pnfs.h +2 -0
@@ -54,26 +57,102 @@ struct pnfs_layoutdriver_type {
       struct module *owner;
       int (*initialize_mountpoint) (struct nfs_server *);
       int (*uninitialize_mountpoint) (struct nfs_server *);
+     struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_hdr *layoutid, struct nfs4_layoutget_res *lgr);
+     void (*free_lseg) (struct pnfs_layout_segment *lseg);
  };
 
  struct pnfs_layout_hdr {
       unsigned long           refcount;

      struct list_head        layouts;   /* other client layouts */ struct list_head        segs;      /* layout segments list */
+     seqlock_t               seqlock;   /* Protects the stateid */
+     nfs4_stateid            stateid;
      unsigned long           state;
      struct inode            *inode;
 };

Quoted 1 line

+struct pnfs_deviceid {

Please for my own sanity, could we change those names. I'll post a patch once we settle on final code. I want this to be nfs4_deviceid (And perhaps in that other header)

Quoted 26 lines

+     char data[NFS4_PNFS_DEVICEID4_SIZE];

fs/nfs/pnfs.h +25 -0
+};
+
+struct pnfs_device {
+     struct pnfs_deviceid dev_id;
+     unsigned int  layout_type;
+     unsigned int  mincount;
+     struct page **pages;
+     void          *area;
+     unsigned int  pgbase;
+     unsigned int  pglen;
+};
+
+/*
+ * Device ID RCU cache. A device ID is unique per client ID and layout type.
+ */
+#define NFS4_DEVICE_ID_HASH_BITS     5
+#define NFS4_DEVICE_ID_HASH_SIZE     (1 << NFS4_DEVICE_ID_HASH_BITS)
+#define NFS4_DEVICE_ID_HASH_MASK     (NFS4_DEVICE_ID_HASH_SIZE - 1)
+
+static inline u32
+nfs4_deviceid_hash(struct pnfs_deviceid *id)
+{
+     unsigned char *cptr = (unsigned char *)id->data;
+     unsigned int nbytes = NFS4_PNFS_DEVICEID4_SIZE;
+     u32 x = 0;

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 110

Quoted 10 lines
fs/nfs/pnfs.h +10 -0
+
+     while (nbytes--) {
+             x *= 37;
+             x += *cptr++;
+     }
+     return x & NFS4_DEVICE_ID_HASH_MASK;
+}
+
+/* Device ID cache node */
+struct nfs4_deviceid {

And this one rename to pnfs_deviceid_node

Quoted 9 lines

+     struct hlist_node       de_node;
+     struct pnfs_deviceid    de_id;
+     atomic_t                de_ref;

fs/nfs/pnfs.h +6 -0
+};
+
+struct nfs4_deviceid_cache {
+     spinlock_t              dc_lock;
+     atomic_t                dc_ref;
+     void                    (*dc_free_callback)(struct nfs4_deviceid *);
Quoted 8 lines

fs/nfs/pnfs.h

Quoted 4 lines

+     struct hlist_head       dc_deviceids[NFS4_DEVICE_ID_HASH_SIZE];
+     struct hlist_head       dc_to_free;

fs/nfs/pnfs.h +2 -0
+};
+

All these cache APIs below, the naming convention in this header is to call them pnfs_xxx. They don't have any use in general nfs4. (and are implemented in pnfs.c)

I'll defer to Andy regarding naming.

Quoted 8 lines

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 127

Quoted 5 lines
fs/nfs/pnfs.h +4 -0
+extern int nfs4_alloc_init_deviceid_cache(struct nfs_client *,
+                             void (*free_callback)(struct nfs4_deviceid *));
+extern void nfs4_put_deviceid_cache(struct nfs_client *);
+extern void nfs4_init_deviceid_node(struct nfs4_deviceid *);

Why is above exported? Can't we just call it from _add_

You are right. It is a remnant of older code organization.

Fred

Quoted 124 lines

Boaz

Quoted 106 lines
fs/nfs/pnfs.h +8 -0
+extern struct nfs4_deviceid *nfs4_find_get_deviceid(
+                             struct nfs4_deviceid_cache *,
+                             struct pnfs_deviceid *);
+extern struct nfs4_deviceid *nfs4_add_deviceid(struct nfs4_deviceid_cache *,
+                             struct nfs4_deviceid *);
+extern void nfs4_put_deviceid(struct nfs4_deviceid_cache *c,
+                           struct nfs4_deviceid *devid);
+

 extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *); extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);

fs/nfs/pnfs.h +6 -0
+/* nfs4proc.c */
+extern int nfs4_proc_getdeviceinfo(struct nfs_server *server,
+                                struct pnfs_device *dev);
+extern int nfs4_proc_layoutget(struct nfs4_layoutget *lgp);
+
+/* pnfs.c */

 struct pnfs_layout_segment *
 pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx, enum pnfs_iomode access_type);
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id); void unset_pnfs_layoutdriver(struct nfs_server *);
+int pnfs_layout_process(struct nfs4_layoutget *lgp); void pnfs_destroy_layout(struct nfs_inode *);
 void pnfs_destroy_all_layouts(struct nfs_client *);

fs/nfs/pnfs.h +3 -0
+void put_layout_hdr(struct inode *inode);
+void pnfs_get_layout_stateid(nfs4_stateid *dst, struct pnfs_layout_hdr *lo,
+                          struct nfs4_state *open_state);

 static inline int lo_fail_bit(u32 iomode)

include/linux/nfs4.h +2 -0
index 2dde7c8..dcdd11c 100644
@@ -545,6 +545,8 @@ enum {
       NFSPROC4_CLNT_SEQUENCE,
       NFSPROC4_CLNT_GET_LEASE_TIME,
       NFSPROC4_CLNT_RECLAIM_COMPLETE,
+     NFSPROC4_CLNT_LAYOUTGET,
+     NFSPROC4_CLNT_GETDEVICEINFO,
  };
 
  /* nfs41 types */
include/linux/nfs_fs_sb.h +0 -0
index e670a9c..7512886 100644
@@ -83,6 +83,7 @@ struct nfs_client {
       u32                     cl_exchange_flags;

      struct nfs4_session     *cl_session;    /* sharred session */ struct list_head        cl_layouts;
+     struct nfs4_deviceid_cache *cl_devid_cache; /* pNFS deviceid cache */ #endif /* CONFIG_NFS_V4_1 */

 #ifdef CONFIG_NFS_FSCACHE

include/linux/nfs_xdr.h +42 -0
index 7d68a5c..4953b58 100644
@@ -186,6 +186,55 @@ struct nfs4_get_lease_time_res {
       struct nfs4_sequence_res        lr_seq_res;
  };
 
+#define PNFS_LAYOUT_MAXSIZE 4096
+
+struct nfs4_layoutdriver_data {
+     __u32 len;
+     void *buf;
+};
+
+struct pnfs_layout_range {
+     u32 iomode;
+     u64 offset;
+     u64 length;
+};
+
+struct nfs4_layoutget_args {
+     __u32 type;
+     struct pnfs_layout_range range;
+     __u64 minlength;
+     __u32 maxcount;
+     struct inode *inode;
+     struct nfs_open_context *ctx;
+     struct nfs4_sequence_args seq_args;
+};
+
+struct nfs4_layoutget_res {
+     __u32 return_on_close;
+     struct pnfs_layout_range range;
+     __u32 type;
+     nfs4_stateid stateid;
+     struct nfs4_layoutdriver_data layout;
+     struct nfs4_sequence_res seq_res;
+};
+
+struct nfs4_layoutget {
+     struct nfs4_layoutget_args args;
+     struct nfs4_layoutget_res res;
+     struct pnfs_layout_segment **lsegpp;
+     int status;
+};
+
+struct nfs4_getdeviceinfo_args {
+     struct pnfs_device *pdev;
+     struct nfs4_sequence_args seq_args;

include/linux/nfs_xdr.h struct nfs4_get_lease_time_res { · line 240

Quoted 10 lines
include/linux/nfs_xdr.h +7 -0
+};
+
+struct nfs4_getdeviceinfo_res {
+     struct pnfs_device *pdev;
+     struct nfs4_sequence_res seq_res;
+};
+
  /*
   * Arguments to the open call.
   */

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#27 lore raw

Re: [PATCH 11/12] RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

Quoted 12 lines · Fred Isaman
Quoted 11 lines · Boaz Harrosh
Quoted 9 lines · Fred Isaman

From: The pNFS Team <[email protected]>

Add the ability to actually send LAYOUTGET and GETDEVICEINFO. This also adds in the machinery to handle layout state and the deviceid cache. Note that GETDEVICEINFO is not called directly by the generic layer. Instead it is called by the drivers while parsing the LAYOUTGET opaque data in response to an unknown device id embedded therein. Annoyingly, RFC 5661 only encodes device ids within the driver-specific opaque data.

Quoted 9 lines
Quoted 3 lines

Fred, In regard to device cache at least, this is a complete new code then what I knew before. I wish you could send a SQUASHME diff first of what has changed.

Many of these changes were sent off-list to you in the initial format of original-code plus squashmes. In particular,
there was a patch labeled "SQUASHME pnfs_submit_1 fix pnfs_submit:
generic pnfs deviceid cache"

Sorry for that then, That was confusing and me travelling to LSF got everything mixed up.

But you should have smacked me on the head for hacking the wrong code.

Quoted 71 lines
Quoted 61 lines
Quoted 2 lines
Signed-off-by TBD - melding/reorganization of several patches

<snip>

Quoted 56 lines
fs/nfs/pnfs.c +56 -0
+/*
+ * Device ID cache. Currently supports one layout type per struct nfs_client.
+ * Add layout type to the lookup key to expand to support multiple types.
+ */
+int
+nfs4_alloc_init_deviceid_cache(struct nfs_client *clp,
+                      void (*free_callback)(struct nfs4_deviceid *))
+{
+     struct nfs4_deviceid_cache *c;
+
+     c = kzalloc(sizeof(struct nfs4_deviceid_cache), GFP_KERNEL);
+     if (!c)
+             return -ENOMEM;
+     spin_lock(&clp->cl_lock);
+     if (clp->cl_devid_cache != NULL) {
+             atomic_inc(&clp->cl_devid_cache->dc_ref);
+             dprintk("%s [kref [%d]]\n", __func__,
+                     atomic_read(&clp->cl_devid_cache->dc_ref));
+             kfree(c);
+     } else {
+             /* kzalloc initializes hlists */
+             spin_lock_init(&c->dc_lock);
+             atomic_set(&c->dc_ref, 1);
+             c->dc_free_callback = free_callback;
+             clp->cl_devid_cache = c;
+             dprintk("%s [new]\n", __func__);
+     }
+     spin_unlock(&clp->cl_lock);
+     return 0;
+}
+EXPORT_SYMBOL_GPL(nfs4_alloc_init_deviceid_cache);
+
+void
+nfs4_init_deviceid_node(struct nfs4_deviceid *d)
+{
+     INIT_HLIST_NODE(&d->de_node);
+     atomic_set(&d->de_ref, 1);
+}
+EXPORT_SYMBOL_GPL(nfs4_init_deviceid_node);
+
+/*
+ * Called from pnfs_layoutdriver_type->free_lseg
+ * last layout segment reference frees deviceid
+ */
+void
+nfs4_put_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *devid)
+{
+     struct pnfs_deviceid *id = &devid->de_id;
+     struct nfs4_deviceid *d;
+     struct hlist_node *n;
+     long h = nfs4_deviceid_hash(id);
+
+     dprintk("%s [%d]\n", __func__, atomic_read(&devid->de_ref));
+     if (!atomic_dec_and_lock(&devid->de_ref, &c->dc_lock))
+             return;
+
Quoted 9 lines

fs/nfs/pnfs.c out_unlock: · line 670

Quoted 6 lines
fs/nfs/pnfs.c +6 -0
+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[h], de_node)
+             if (!memcmp(&d->de_id, id, sizeof(*id))) {
+                     hlist_del_rcu(&d->de_node);
+                     spin_unlock(&c->dc_lock);
+                     synchronize_rcu();
+                     c->dc_free_callback(devid);

What?!? free the devid on every put? surly something went hey-wire

Quoted 2 lines

Did you miss the atomic_dec_and_lock above?

Yes thanks that's what I missed, the early return.

Quoted 10 lines
Quoted 3 lines

This used to be kref(ed)

The original code required access to the internals of the kref, which was frowned upon.

Quoted 3 lines
Quoted 3 lines
fs/nfs/pnfs.c +3 -0
+                     return;
+             }
+     spin_unlock(&c->dc_lock);

BUG() not found then, right?

Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
fs/nfs/pnfs.c +37 -0
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid);
+
+/* Find and reference a deviceid */
+struct nfs4_deviceid *
+nfs4_find_get_deviceid(struct nfs4_deviceid_cache *c, struct pnfs_deviceid *id)
+{
+     struct nfs4_deviceid *d;
+     struct hlist_node *n;
+     long hash = nfs4_deviceid_hash(id);
+
+     dprintk("--> %s hash %ld\n", __func__, hash);
+     rcu_read_lock();
+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+             if (!memcmp(&d->de_id, id, sizeof(*id))) {
+                     if (!atomic_inc_not_zero(&d->de_ref)) {
+                             goto fail;
+                     } else {
+                             rcu_read_unlock();
+                             return d;
+                     }
+             }
+     }
+fail:
+     rcu_read_unlock();
+     return NULL;
+}
+EXPORT_SYMBOL_GPL(nfs4_find_get_deviceid);
+
+/*
+ * Add a deviceid to the cache.
+ * GETDEVICEINFOs for same deviceid can race. If deviceid is found, discard new
+ */
+struct nfs4_deviceid *
+nfs4_add_deviceid(struct nfs4_deviceid_cache *c, struct nfs4_deviceid *new)
+{
+     struct nfs4_deviceid *d;
Quoted 8 lines
Quoted 8 lines

fs/nfs/pnfs.c out_unlock: · line 718

Quoted 8 lines
fs/nfs/pnfs.c +8 -0
+     struct hlist_node *n;
+     long hash = nfs4_deviceid_hash(&new->de_id);
+
+     dprintk("--> %s hash %ld\n", __func__, hash);
+     spin_lock(&c->dc_lock);
+     hlist_for_each_entry_rcu(d, n, &c->dc_deviceids[hash], de_node) {
+             if (!memcmp(&d->de_id, &new->de_id, sizeof(new->de_id))) {
+                     spin_unlock(&c->dc_lock);
			  did_get(d); Right?
Quoted 10 lines
Quoted 9 lines

fs/nfs/pnfs.c out_unlock: · line 721

Quoted 3 lines
fs/nfs/pnfs.c +3 -0
+                     dprintk("%s [discard]\n", __func__);
+                     c->dc_free_callback(new);
+                     return d;

I am totally confused and should be smacked on the head strong. As said above one alloc_lseg adds the first, second comes in, but same deviceid, finds the first and returns it (no refcounting). Then a first free_lseg nfs4_put_deviceid is called, so the second lseg holds free memory?

Quoted 8 lines

You are right, there is a missing increment of the refcount here.

Part of the problem here is that the flow is obscurd because it is being called form the driver layer.
I think it might be better if nfs4_find_get_deviceid did the GETDEVICEINFO on a cache miss, instead of relying on the driver to do it.

That's fine. It means nothing the failing-find and _add_ are not locked so it changes nothing. The _add_ should be self-sustained regardless.

Quoted 3 lines
Quoted 3 lines

fs/nfs/pnfs.c out_unlock: · line 724

Quoted 3 lines
fs/nfs/pnfs.c +3 -0
+             }
+     }
+     hlist_add_head_rcu(&new->de_node, &c->dc_deviceids[hash]);

See below I would like to add here;

	  did_get(new);
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
fs/nfs/pnfs.c +9 -0
+     spin_unlock(&c->dc_lock);
+     dprintk("%s [new]\n", __func__);
+     return new;
+}
+EXPORT_SYMBOL_GPL(nfs4_add_deviceid);
+
+void
+nfs4_put_deviceid_cache(struct nfs_client *clp)
+{
Quoted 9 lines

fs/nfs/pnfs.c out_unlock: · line 737

Quoted 4 lines
fs/nfs/pnfs.c +4 -0
+     struct nfs4_deviceid_cache *local = clp->cl_devid_cache;
+
+     dprintk("--> %s cl_devid_cache %p\n", __func__, clp->cl_devid_cache);
+     if (atomic_dec_and_lock(&local->dc_ref, &clp->cl_lock)) {

Don't you want to make sure there are no hanging devices on this cache. Please note that an add or find does not take the cache reference only server_init does, right?

Quoted 2 lines

If there are any hanging devices at this point, it is a pretty serious bug.

Yes therefore a BUG_ON() is do.
But no! I would like for the code that decrements all did refs back in here. See below.

Quoted 26 lines
Quoted 10 lines

fs/nfs/pnfs.c out_unlock: · line 743

Quoted 7 lines
fs/nfs/pnfs.c +6 -0
+             clp->cl_devid_cache = NULL;
+             spin_unlock(&clp->cl_lock);
+             kfree(local);
+     }
+}
+EXPORT_SYMBOL_GPL(nfs4_put_deviceid_cache);

Note to self. Test code alot.

Quoted 15 lines

Fred:
I don't understand. A cache property I'm looking for and which I do not See in this code, perhaps because I'm clueless.
(And to be honest I have not understood it before as well, since I never actually ran with that code I posted.)
Is:
- An _add_ to queue should magically hold two references to the item, so
- A matching _put_ does not deallocate the item but keeps it referenced once.
- A find/put pair does not remove the item from cache as well (+=1/-=1)
- [optional] At some future out-of-memory condition or when cache is to big, according to some specified max cache. The most unused items are dereferenced and if are not used (ref==0) are then removed.
- At driver shut-down all cached devices are dereferenced (and if not used which they should) are dealocated.

Quoted 9 lines

I agree. The original deviceid code had an issue where nothing was ever deallocated until shutdown. The code as given here ties the deviceid lifetime to existing references. This is not ideal, because a brief loss of reference to a deviceid will cause an unnecessary GETDEVICEINFO. However, for the current submission, it has the advantage that it is simple and correct. Adding the machinery to do as you suggest above is indeed a (fairly high) priority goal, but was intentionally deferred.

I would like to please return it. That is deallocate it only at shut down. Otherwise we are for a lot of trouble. We are not yet in a situation of 1000nd of devices where it might matter. Even up to 100rd it is still fine to never de-allocate. (Currently on the planet Panasas is the only one with 1000nd of devices in a single installation)

Look at it this way. Before we mounted all the NFS servers in our domain prior to usage, and/or as part of auto-mount which never got unmounted until shotdown/unmount. So we do the same with pNFS only we have the privilege of doing a single mount and have everything dynamically logged in for us. So we are just as before.

For all pNFS filesystems today the first IOs will GETDEVICEINFO for the 10s of devices deploid and keep them allocated. If we free them then what will happen if we need to GETDEVICEINFO when all memory is dirty and we need it for cleaning that dirty memory. We don't have any forward progress provision for that yet.

The simple thing to do is keep them allocated for ever (until unmount) by adding a single did_get() call in the did_add() function. (And did_put() in cache deallocation). That's more simple then hardening the code.

And If we don't the pNFS performance will suck big time specially for that humongous files-layout get_device_info. Because it will be done for every get_layout. A regular bash script opens a file then closes it. Most of the time we are not parallel as we think.

Quoted 11 lines
Quoted 5 lines

So in effect if [optional] code above is not yet implemented then a getdeviceinfo is never preformed twice for the same deviceid (Assuming no device recall). This is what I had in osd, and is cardinal for it's performance. Is that what we are aiming at?

Yes, that is the goal. Right now, if a layout is still around that references a deviceid, a second GETDEVICEINFO will not be sent. However, if all layouts referencing a specific deviceid have been returned, then yes, a second GETDEVICEINFO will be called.

Then sure when we come to the situation that we need proper support for more then 100 machines in a cluster, then we can add the clean on add stage where if we are higher then x devices we start to replace entries.

What you guys think?

Boaz

Quoted 254 lines
Quoted 100 lines
Quoted 37 lines
fs/nfs/pnfs.h +6 -1
index 1c3eb02..b7de762 100644
@@ -32,11 +32,13 @@
 
 struct pnfs_layout_segment {
      struct list_head fi_list;
-     u32 iomode;
+     struct pnfs_layout_range range;
      struct kref kref;
      struct pnfs_layout_hdr *layout;
 };
 
+#define NFS4_PNFS_DEVICEID4_SIZE 16
+
 #ifdef CONFIG_NFS_V4_1
 
 #define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
@@ -44,6 +46,7 @@ struct pnfs_layout_segment {
 enum {
      NFS_LAYOUT_RO_FAILED = 0,       /* get ro layout failed stop trying */
      NFS_LAYOUT_RW_FAILED,           /* get rw layout failed stop trying */
+     NFS_LAYOUT_STATEID_SET,         /* have a valid layout stateid */
 };
 
 /* Per-layout driver specific registration structure */
@@ -54,26 +57,102 @@ struct pnfs_layoutdriver_type {
      struct module *owner;
      int (*initialize_mountpoint) (struct nfs_server *);
      int (*uninitialize_mountpoint) (struct nfs_server *);
+     struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_hdr *layoutid, struct nfs4_layoutget_res *lgr);
+     void (*free_lseg) (struct pnfs_layout_segment *lseg);
 };
 
 struct pnfs_layout_hdr {

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 74

Quoted 10 lines
fs/nfs/pnfs.h +3 -0
      unsigned long           refcount;
      struct list_head        layouts;   /* other client layouts */
      struct list_head        segs;      /* layout segments list */
+     seqlock_t               seqlock;   /* Protects the stateid */
+     nfs4_stateid            stateid;
      unsigned long           state;
      struct inode            *inode;
 };
 
+struct pnfs_deviceid {

Please for my own sanity, could we change those names. I'll post a patch once we settle on final code. I want this to be nfs4_deviceid (And perhaps in that other header)

Quoted 26 lines
fs/nfs/pnfs.h +26 -0
+     char data[NFS4_PNFS_DEVICEID4_SIZE];
+};
+
+struct pnfs_device {
+     struct pnfs_deviceid dev_id;
+     unsigned int  layout_type;
+     unsigned int  mincount;
+     struct page **pages;
+     void          *area;
+     unsigned int  pgbase;
+     unsigned int  pglen;
+};
+
+/*
+ * Device ID RCU cache. A device ID is unique per client ID and layout type.
+ */
+#define NFS4_DEVICE_ID_HASH_BITS     5
+#define NFS4_DEVICE_ID_HASH_SIZE     (1 << NFS4_DEVICE_ID_HASH_BITS)
+#define NFS4_DEVICE_ID_HASH_MASK     (NFS4_DEVICE_ID_HASH_SIZE - 1)
+
+static inline u32
+nfs4_deviceid_hash(struct pnfs_deviceid *id)
+{
+     unsigned char *cptr = (unsigned char *)id->data;
+     unsigned int nbytes = NFS4_PNFS_DEVICEID4_SIZE;
+     u32 x = 0;

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 110

Quoted 10 lines
fs/nfs/pnfs.h +10 -0
+
+     while (nbytes--) {
+             x *= 37;
+             x += *cptr++;
+     }
+     return x & NFS4_DEVICE_ID_HASH_MASK;
+}
+
+/* Device ID cache node */
+struct nfs4_deviceid {

And this one rename to pnfs_deviceid_node

Quoted 9 lines
fs/nfs/pnfs.h +9 -0
+     struct hlist_node       de_node;
+     struct pnfs_deviceid    de_id;
+     atomic_t                de_ref;
+};
+
+struct nfs4_deviceid_cache {
+     spinlock_t              dc_lock;
+     atomic_t                dc_ref;
+     void                    (*dc_free_callback)(struct nfs4_deviceid *);
Quoted 8 lines

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 123

Quoted 4 lines
fs/nfs/pnfs.h +4 -0
+     struct hlist_head       dc_deviceids[NFS4_DEVICE_ID_HASH_SIZE];
+     struct hlist_head       dc_to_free;
+};
+

All these cache APIs below, the naming convention in this header is to call them pnfs_xxx. They don't have any use in general nfs4. (and are implemented in pnfs.c)

I'll defer to Andy regarding naming.

Quoted 8 lines

fs/nfs/pnfs.h struct pnfs_layoutdriver_type { · line 127

Quoted 5 lines
fs/nfs/pnfs.h +4 -0
+extern int nfs4_alloc_init_deviceid_cache(struct nfs_client *,
+                             void (*free_callback)(struct nfs4_deviceid *));
+extern void nfs4_put_deviceid_cache(struct nfs_client *);
+extern void nfs4_init_deviceid_node(struct nfs4_deviceid *);

Why is above exported? Can't we just call it from _add_

You are right. It is a remnant of older code organization.

Fred

Quoted 115 lines

Boaz

Quoted 113 lines
fs/nfs/pnfs.h +18 -0
+extern struct nfs4_deviceid *nfs4_find_get_deviceid(
+                             struct nfs4_deviceid_cache *,
+                             struct pnfs_deviceid *);
+extern struct nfs4_deviceid *nfs4_add_deviceid(struct nfs4_deviceid_cache *,
+                             struct nfs4_deviceid *);
+extern void nfs4_put_deviceid(struct nfs4_deviceid_cache *c,
+                           struct nfs4_deviceid *devid);
+
 extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
 extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
 
+/* nfs4proc.c */
+extern int nfs4_proc_getdeviceinfo(struct nfs_server *server,
+                                struct pnfs_device *dev);
+extern int nfs4_proc_layoutget(struct nfs4_layoutget *lgp);
+
+/* pnfs.c */
 struct pnfs_layout_segment *
 pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
                 enum pnfs_iomode access_type);
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
+int pnfs_layout_process(struct nfs4_layoutget *lgp);
 void pnfs_destroy_layout(struct nfs_inode *);
 void pnfs_destroy_all_layouts(struct nfs_client *);
+void put_layout_hdr(struct inode *inode);
+void pnfs_get_layout_stateid(nfs4_stateid *dst, struct pnfs_layout_hdr *lo,
+                          struct nfs4_state *open_state);
 
 
 static inline int lo_fail_bit(u32 iomode)
include/linux/nfs4.h +2 -0
index 2dde7c8..dcdd11c 100644
@@ -545,6 +545,8 @@ enum {
      NFSPROC4_CLNT_SEQUENCE,
      NFSPROC4_CLNT_GET_LEASE_TIME,
      NFSPROC4_CLNT_RECLAIM_COMPLETE,
+     NFSPROC4_CLNT_LAYOUTGET,
+     NFSPROC4_CLNT_GETDEVICEINFO,
 };
 
 /* nfs41 types */
include/linux/nfs_fs_sb.h +1 -0
index e670a9c..7512886 100644
@@ -83,6 +83,7 @@ struct nfs_client {
      u32                     cl_exchange_flags;
      struct nfs4_session     *cl_session;    /* sharred session */
      struct list_head        cl_layouts;
+     struct nfs4_deviceid_cache *cl_devid_cache; /* pNFS deviceid cache */
 #endif /* CONFIG_NFS_V4_1 */
 
 #ifdef CONFIG_NFS_FSCACHE
include/linux/nfs_xdr.h +49 -0
index 7d68a5c..4953b58 100644
@@ -186,6 +186,55 @@ struct nfs4_get_lease_time_res {
      struct nfs4_sequence_res        lr_seq_res;
 };
 
+#define PNFS_LAYOUT_MAXSIZE 4096
+
+struct nfs4_layoutdriver_data {
+     __u32 len;
+     void *buf;
+};
+
+struct pnfs_layout_range {
+     u32 iomode;
+     u64 offset;
+     u64 length;
+};
+
+struct nfs4_layoutget_args {
+     __u32 type;
+     struct pnfs_layout_range range;
+     __u64 minlength;
+     __u32 maxcount;
+     struct inode *inode;
+     struct nfs_open_context *ctx;
+     struct nfs4_sequence_args seq_args;
+};
+
+struct nfs4_layoutget_res {
+     __u32 return_on_close;
+     struct pnfs_layout_range range;
+     __u32 type;
+     nfs4_stateid stateid;
+     struct nfs4_layoutdriver_data layout;
+     struct nfs4_sequence_res seq_res;
+};
+
+struct nfs4_layoutget {
+     struct nfs4_layoutget_args args;
+     struct nfs4_layoutget_res res;
+     struct pnfs_layout_segment **lsegpp;
+     int status;
+};
+
+struct nfs4_getdeviceinfo_args {
+     struct pnfs_device *pdev;
+     struct nfs4_sequence_args seq_args;
+};
+
+struct nfs4_getdeviceinfo_res {
+     struct pnfs_device *pdev;
+     struct nfs4_sequence_res seq_res;
+};
+
Quoted 9 lines

include/linux/nfs_xdr.h struct nfs4_get_lease_time_res { · line 240

Quoted 3 lines
include/linux/nfs_xdr.h +0 -0
 /*
  * Arguments to the open call.
  */

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw

Re: [PATCH 11/12] RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

Quoted 26 lines · Boaz Harrosh
Quoted 12 lines · Fred Isaman
Quoted 1 line · Boaz Harrosh

On 09/18/2010 05:17 AM, Fred Isaman wrote:

I agree. The original deviceid code had an issue where nothing was ever deallocated until shutdown. The code as given here ties the deviceid lifetime to existing references. This is not ideal, because a brief loss of reference to a deviceid will cause an unnecessary GETDEVICEINFO. However, for the current submission, it has the advantage that it is simple and correct. Adding the machinery to do as you suggest above is indeed a (fairly high) priority goal, but was intentionally deferred.

I would like to please return it. That is deallocate it only at shut down. Otherwise we are for a lot of trouble. We are not yet in a situation of 1000nd of devices where it might matter. Even up to 100rd it is still fine to never de-allocate. (Currently on the planet Panasas is the only one with 1000nd of devices in a single installation)

Look at it this way. Before we mounted all the NFS servers in our domain prior to usage, and/or as part of auto-mount which never got unmounted until shotdown/unmount. So we do the same with pNFS only we have the privilege of doing a single mount and have everything dynamically logged in for us. So we are just as before.

Quoted 5 lines

For all pNFS filesystems today the first IOs will GETDEVICEINFO for the 10s of devices deploid and keep them allocated. If we free them then what will happen if we need to GETDEVICEINFO when all memory is dirty and we need it for cleaning that dirty memory. We don't have any forward progress provision for that yet.

Devices are no different than layouts for this matter.
To flush your cache under low memory conditions using pnfs you'll need both a layout and the corresponding devices. So why do you want to keep the devices forever but not the layouts?

Quoted 11 lines

The simple thing to do is keep them allocated for ever (until unmount) by adding a single did_get() call in the did_add() function. (And did_put() in cache deallocation). That's more simple then hardening the code.

And If we don't the pNFS performance will suck big time specially for that humongous files-layout get_device_info. Because it will be done for every get_layout. A regular bash script opens a file then closes it. Most of the time we are not parallel as we think.

I'd like to see the actual numbers for a given benchmark. Keep in mind that typically for the files layout the server won't ask for return_on_close so the layout will actually be kept around (and the associated devices) while the inode is resident, right?

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

So in effect if [optional] code above is not yet implemented then a getdeviceinfo is never preformed twice for the same deviceid (Assuming no device recall). This is what I had in osd, and is cardinal for it's performance. Is that what we are aiming at?

Yes, that is the goal. Right now, if a layout is still around that references a deviceid, a second GETDEVICEINFO will not be sent. However, if all layouts referencing a specific deviceid have been returned, then yes, a second GETDEVICEINFO will be called.

Quoted 6 lines

Then sure when we come to the situation that we need proper support for more then 100 machines in a cluster, then we can add the clean on add stage where if we are higher then x devices we start to replace entries.

What you guys think?

It's hard to tell without the actual numbers.

Keeping a reference count on the device from the layout makes sure you'll have the devices to use the layout and that you don't issue multiple GETDEVICEINFOs in case a device is shared between more than one layout.

Ideally, a laundromat service could clean up unused client state once in a while so to keep it around for a short while so it can be reused if possible.

Benny

Quoted 3 lines

Boaz

#29 lore raw

Re: [PATCH 11/12] RFC: pnfs: add LAYOUTGET and GETDEVICEINFO infrastructure

Quoted 50 lines · Benny Halevy
Quoted 26 lines · Boaz Harrosh
Quoted 12 lines · Fred Isaman
Quoted 1 line · Boaz Harrosh

On 09/18/2010 05:17 AM, Fred Isaman wrote:

I agree.  The original deviceid code had an issue where nothing was ever deallocated until shutdown.  The code as given here ties the deviceid lifetime to existing references.  This is not ideal, because a brief loss of reference to a deviceid will cause an unnecessary GETDEVICEINFO.  However, for the current submission, it has the advantage that it is simple and correct.  Adding the machinery to do as you suggest above is indeed a (fairly high) priority goal, but was intentionally deferred.

I would like to please return it. That is deallocate it only at shut down. Otherwise we are for a lot of trouble. We are not yet in a situation of 1000nd of devices where it might matter. Even up to 100rd it is still fine to never de-allocate. (Currently on the planet Panasas is the only one with 1000nd of devices in a single installation)

Look at it this way. Before we mounted all the NFS servers in our domain prior to usage, and/or as part of auto-mount which never got unmounted until shotdown/unmount. So we do the same with pNFS only we have the privilege of doing a single mount and have everything dynamically logged in for us. So we are just as before.

Quoted 5 lines

For all pNFS filesystems today the first IOs will GETDEVICEINFO for the 10s of devices deploid and keep them allocated. If we free them then what will happen if we need to GETDEVICEINFO when all memory is dirty and we need it for cleaning that dirty memory. We don't have any forward progress provision for that yet.

Devices are no different than layouts for this matter.
To flush your cache under low memory conditions using pnfs you'll need both a layout and the corresponding devices. So why do you want to keep the devices forever but not the layouts?

Quoted 11 lines

The simple thing to do is keep them allocated for ever (until unmount) by adding a single did_get() call in the did_add() function. (And did_put() in cache deallocation). That's more simple then hardening the code.

And If we don't the pNFS performance will suck big time specially for that humongous files-layout get_device_info. Because it will be done for every get_layout. A regular bash script opens a file then closes it. Most of the time we are not parallel as we think.

Quoted 5 lines

I'd like to see the actual numbers for a given benchmark. Keep in mind that typically for the files layout the server won't ask for return_on_close so the layout will actually be kept around (and the associated devices) while the inode is resident, right?

True. The current code (once bugfixed) is simple, spec-compliant, and works reasonably well for filelayout. Why do improvements like maintaining cache lifetime past reference need to be in the initial submission? I agree they are important and will need to be done, but why now?

Fred

Quoted 40 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

So in effect if [optional] code above is not yet implemented then a getdeviceinfo is never preformed twice for the same deviceid (Assuming no device recall). This is what I had in osd, and is cardinal for it's performance. Is that what we are aiming at?

Yes, that is the goal.  Right now, if a layout is still around that references a deviceid, a second GETDEVICEINFO will not be sent. However, if all layouts referencing a specific deviceid have been returned, then yes, a second GETDEVICEINFO will be called.

Quoted 6 lines

Then sure when we come to the situation that we need proper support for more then 100 machines in a cluster, then we can add the clean on add stage where if we are higher then x devices we start to replace entries.

What you guys think?

It's hard to tell without the actual numbers.

Keeping a reference count on the device from the layout makes sure you'll have the devices to use the layout and that you don't issue multiple GETDEVICEINFOs in case a device is shared between more than one layout.

Ideally, a laundromat service could clean up unused client state once in a while so to keep it around for a short while so it can be reused if possible.

Benny

Quoted 3 lines

Boaz

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#30 lore raw

PATCH 12/12 RFC: pnfs: filelayout: add driver's LAYOUTGET and GETDEVICEINFO infrastructure

In reply to Fred Isaman

From: The pNFS Team <[email protected]>

Implement the driver's io_ops->alloc_lseg and free_lseg functions, which integrate into the deviceid cache and calls out to nfs4_proc_getdeviceinfo when necessary.

Signed-off-by TBD - melding/reorganization of several patches
---
 fs/nfs/Makefile            |    2 +-
 fs/nfs/client.c            |    1 +
 fs/nfs/nfs4filelayout.c    |  204 ++++++++++++++++++++-
 fs/nfs/nfs4filelayout.h    |   94 +++++++++
 fs/nfs/nfs4filelayoutdev.c |  451 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 750 insertions(+), 2 deletions(-)
 create mode 100644 fs/nfs/nfs4filelayout.h
 create mode 100644 fs/nfs/nfs4filelayoutdev.c
fs/nfs/Makefile +1 -1
index 631ffcb..b7dd199 100644
@@ -20,4 +20,4 @@ nfs-$(CONFIG_SYSCTL) += sysctl.o
 nfs-$(CONFIG_NFS_FSCACHE) += fscache.o fscache-index.o
 
 obj-$(CONFIG_NFS_V4_1) += nfs_layout_nfsv41_files.o
-nfs_layout_nfsv41_files-y := nfs4filelayout.o
+nfs_layout_nfsv41_files-y := nfs4filelayout.o nfs4filelayoutdev.o
fs/nfs/client.c +1 -0
index 6fc5c84..4a4ed94 100644
@@ -255,6 +255,7 @@ void nfs_put_client(struct nfs_client *clp)
 		nfs_free_client(clp);
 	}
 }
+EXPORT_SYMBOL_GPL(nfs_put_client);
 
 #if defined(CONFIG_IPV6) || defined(CONFIG_IPV6_MODULE)
 /*
fs/nfs/nfs4filelayout.c +203 -1

233 lines not on this page · raw message

fs/nfs/nfs4filelayout.h new file +94 -0

94 lines not on this page · raw message

fs/nfs/nfs4filelayoutdev.c new file +451 -0

451 lines not on this page · raw message

Signature · 1 line
1.7.2.1