Skip to contentSkip to conversation

pnfs_submit file layout client device management

PATCH In review [email protected] 3 participants 20 messages Apr 8, 2010 – Apr 12, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover pnfs_submit file layout client device management

Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches the file layout driver.

This set of patches implements several design changes enmerated at
http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.

- Only call GETDEVICEINFO from one place
- Only cache the first data server in the multipath_list4 array.
- Wait to connect to data server when needed for I/O
- Move data server cache to stand alone cache

Also some cleanup, and function interfaces are changed in preparation for the soon to come generic deviceid cache.

0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch 0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch 0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch 0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch 0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch 0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch 0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch 0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch 0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch 0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch 0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch 0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch 0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch 0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch

Testing:
CONFIG_NFS_V4_1 set:
pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
Connecatathon tests pass. pyNFS testclient tests pass.

CONFIG_NFS_V4_1 not set:
Connecatahon tests pass.

-->Andy

#1 lore raw

PATCH 1/14 SQUASHME: pnfs_submit: remove unused getdevicelist functions

From: Andy Adamson <[email protected]>

---
 fs/nfs/nfs4filelayout.h    |    4 ----
 fs/nfs/nfs4filelayoutdev.c |   27 ---------------------------
 2 files changed, 0 insertions(+), 31 deletions(-)
fs/nfs/nfs4filelayout.h +0 -4
index 6051195..d003809 100644
@@ -106,10 +106,6 @@ int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 			  loff_t offset,
 			  size_t count,
 			  struct nfs4_pnfs_dserver *dserver);
-int decode_and_add_devicelist(struct filelayout_mount_type *mt, struct pnfs_devicelist *devlist);
-int process_deviceid_list(struct filelayout_mount_type *mt,
-			  struct nfs_fh *fh,
-			  struct pnfs_devicelist *devlist);
 u32 filelayout_dserver_get_index(loff_t offset,
 				 struct nfs4_file_layout_dsaddr *di,
 				 struct nfs4_filelayout_segment *layout);
fs/nfs/nfs4filelayoutdev.c +0 -27
index 16dafc0..e557679 100644
@@ -662,33 +662,6 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
 	return dsaddr;
 }
 
-/* For each deviceid, if not already in the cache,
- * call getdeviceinfo and add the devices associated with
- * the deviceid to the list of available devices for this
- * mount point.
- * Must at some point be followed up with nfs4_pnfs_device_destroy.
- */
-int
-process_deviceid_list(struct filelayout_mount_type *mt,
-		      struct nfs_fh *fh,
-		      struct pnfs_devicelist *devlist)
-{
-	int i;
-
-	dprintk("--> %s: num_devs=%d\n", __func__, devlist->num_devs);
-
-	for (i = 0; i < devlist->num_devs; i++) {
-		if (!nfs4_file_layout_dsaddr_get(mt, &devlist->dev_id[i])) {
-			printk(KERN_WARNING
-			       "<-- %s: Error retrieving device %d\n",
-			       __func__, i);
-			return 1;
-		}
-	}
-	dprintk("<-- %s: success\n", __func__);
-	return 0;
-}
-
 /* Retrieve the information for dev_id, add it to the list
  * of available devices, and return it.
  */
Signature · 1 line
1.6.6
#2 lore raw

PATCH 2/14 SQUASHME pnfs-submit: call get_device_info from filelayout_check

From: Andy Adamson <[email protected]>

filelayout_check() validates a LAYOUTGET return including the deviceid. If the deviceid is not cached, GETDEVICEINFO is called.

Once cached, the deviceid will only be uncached upon umount or a return of all refering layouts. So, there is no need for a get_device_info call in the I/O paths.

Remove nfs4_file_layout_dsaddr_get.

---
 fs/nfs/nfs4filelayout.c    |   11 +++++++----
 fs/nfs/nfs4filelayout.h    |    6 ++++--
 fs/nfs/nfs4filelayoutdev.c |   27 +++------------------------
 3 files changed, 14 insertions(+), 30 deletions(-)
fs/nfs/nfs4filelayout.c +7 -4
index df6c1d0..ffc3ef2 100644
@@ -413,12 +413,15 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	struct nfs_server *nfss = NFS_SERVER(PNFS_INODE(lo));
 
 	dprintk("--> %s\n", __func__);
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(PNFS_INODE(lo)),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dprintk("%s NO device for dev_id %s\n",
+		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		if (dsaddr == NULL) {
+			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
-		goto out;
+			goto out;
+		}
 	}
 	if (fl->first_stripe_index < 0 ||
 	    fl->first_stripe_index > dsaddr->stripe_count) {
@@ -645,7 +648,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
 	stripesz = filelayout_get_stripesize(layoutid);
 	dprintk("%s stripesize %Zd\n", __func__, stripesz);
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(data->inode),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(data->inode)->hlist,
 					     &nfslay->dev_id);
 	if (dsaddr == NULL) {
 		data->pdata.pnfs_error = -EIO;
fs/nfs/nfs4filelayout.h +4 -2
index d003809..3821989 100644
@@ -121,7 +121,9 @@ u32 filelayout_dserver_get_index(loff_t offset,
 } while (0)
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *,
-			  struct pnfs_deviceid *);
+nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
+			   struct pnfs_deviceid *dev_id);
+struct nfs4_file_layout_dsaddr *
+get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
 
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
fs/nfs/nfs4filelayoutdev.c +3 -24
index e557679..ebb9845 100644
@@ -52,13 +52,6 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
 
-struct nfs4_file_layout_dsaddr *nfs4_file_layout_dsaddr_get(
-					struct filelayout_mount_type *mt,
-					struct pnfs_deviceid *dev_id);
-struct nfs4_file_layout_dsaddr *nfs4_pnfs_device_item_find(
-					struct nfs4_pnfs_dev_hlist *hlist,
-					struct pnfs_deviceid *dev_id);
-
 void
 print_ds_list(struct nfs4_multipath *multipath)
 {
@@ -665,7 +658,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
 /* Retrieve the information for dev_id, add it to the list
  * of available devices, and return it.
  */
-static struct nfs4_file_layout_dsaddr *
+struct nfs4_file_layout_dsaddr *
 get_device_info(struct filelayout_mount_type *mt,
 		struct pnfs_deviceid *dev_id)
 {
@@ -737,21 +730,6 @@ out_free:
 }
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *mt,
-			    struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	read_lock(&mt->hlist->dev_lock);
-	dsaddr = _device_lookup(mt->hlist, dev_id);
-	read_unlock(&mt->hlist->dev_lock);
-
-	if (dsaddr == NULL)
-		dsaddr = get_device_info(mt, dev_id);
-	return dsaddr;
-}
-
-struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id)
 {
@@ -798,7 +776,8 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	if (!layout)
 		return 1;
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(inode), &layout->dev_id);
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(inode)->hlist,
+					    &layout->dev_id);
 	if (dsaddr == NULL)
 		return 1;
 
Signature · 1 line
1.6.6
#3 lore raw

PATCH 3/14 SQUASHME pnfs_submit: remove multilist4 caching

From: Andy Adamson <[email protected]>

We only support the use of the first multipath4 data server.

Remove the multipath structures from struct nfs4_file_layout_dsaddr and regroup it's allocation to fit the new strucuture.

Refactor the associated debug print routines.

---
 fs/nfs/nfs4filelayout.c    |   44 ++++---------
 fs/nfs/nfs4filelayout.h    |   12 +---
 fs/nfs/nfs4filelayoutdev.c |  151 +++++++++++++++++++-------------------------
 3 files changed, 81 insertions(+), 126 deletions(-)
fs/nfs/nfs4filelayout.c +13 -31
index ffc3ef2..abdf691 100644
@@ -252,7 +252,6 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 	struct inode *inode = PNFS_INODE(layoutid);
 	struct nfs4_filelayout_segment *flseg;
 	struct nfs4_pnfs_dserver dserver;
-	struct nfs4_pnfs_ds *ds;
 	int status;
 
 	dprintk("--> %s ino %lu nr_pages %d pgbase %u req %Zu@%llu\n",
@@ -271,14 +270,12 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 		return PNFS_NOT_ATTEMPTED;
 	}
 
-	ds = dserver.multipath->ds_list[0];
-
 	dprintk("%s USE DS:ip %x %s\n", __func__,
-		htonl(ds->ds_ip_addr), ds->r_addr);
+		htonl(dserver.ds->ds_ip_addr), dserver.ds->r_addr);
 
 	/* just try the first data server for the index..*/
-	data->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
-	data->fldata.ds_nfs_client = ds->ds_clp;
+	data->fldata.pnfs_client = dserver.ds->ds_clp->cl_rpcclient;
+	data->fldata.ds_nfs_client = dserver.ds->ds_clp;
 	data->args.fh = dserver.fh;
 
 	/* Now get the file offset on the dserver
@@ -300,16 +297,6 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 	return PNFS_ATTEMPTED;
 }
 
-void
-print_ds(struct nfs4_pnfs_ds *ds)
-{
-	dprintk("        ds->ds_ip_addr %x\n", htonl(ds->ds_ip_addr));
-	dprintk("        ds->ds_port %hu\n", ntohs(ds->ds_port));
-	dprintk("        ds->ds_clp %p\n", ds->ds_clp);
-	dprintk("        ds->ds_count %d\n", atomic_read(&ds->ds_count));
-	dprintk("        %s\n", ds->r_addr);
-}
-
 /* Perform async writes. */
 static enum pnfs_try_status
 filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
@@ -324,7 +311,6 @@ filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
 	struct inode *inode = PNFS_INODE(layoutid);
 	struct nfs4_filelayout_segment *flseg = LSEG_LD_DATA(data->pdata.lseg);
 	struct nfs4_pnfs_dserver dserver;
-	struct nfs4_pnfs_ds *ds;
 	int status;
 
 	dprintk("--> %s ino %lu nr_pages %d pgbase %u req %Zu@%llu sync %d\n",
@@ -342,16 +328,13 @@ filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
 		return PNFS_NOT_ATTEMPTED;
 	}
 
-	/* use the first multipath data server */
-	ds = dserver.multipath->ds_list[0];
-
 	dprintk("%s ino %lu %Zu@%llu DS:%x:%hu %s\n",
 		__func__, inode->i_ino, count, offset,
-		htonl(ds->ds_ip_addr), ntohs(ds->ds_port),
-		ds->r_addr);
+		htonl(dserver.ds->ds_ip_addr), ntohs(dserver.ds->ds_port),
+		dserver.ds->r_addr);
 
-	data->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
-	data->fldata.ds_nfs_client = ds->ds_clp;
+	data->fldata.pnfs_client = dserver.ds->ds_clp->cl_rpcclient;
+	data->fldata.ds_nfs_client = dserver.ds->ds_clp;
 	data->args.fh = dserver.fh;
 
 	/* Get the file offset on the dserver. Set the write offset to
@@ -396,8 +379,8 @@ filelayout_free_layout(void *layoutid)
  * Make sure layout segment parameters are sane WRT the device.
  *
  * Notes:
- * 1) current code insists that # stripe index = # multipath devices which
- *    is wrong.
+ * 1) current code insists that # stripe index = # data servers in ds_list
+ *    which is wrong.
  * 2) pattern_offset is ignored and must == 0 which is wrong;
  * 3) the pattern_offset needs to be a mutliple of the stripe unit.
  * 4) stripe unit is multiple of page size
@@ -443,9 +426,9 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	}
 
 	/* XXX only support SPARSE packing. Don't support use MDS open fh */
-	if (!(fl->num_fh == 1 || fl->num_fh == dsaddr->multipath_count)) {
-		dprintk("%s num_fh %u not equal to 1 or multipath_count %u\n",
-			__func__, fl->num_fh, dsaddr->multipath_count);
+	if (!(fl->num_fh == 1 || fl->num_fh == dsaddr->ds_num)) {
+		dprintk("%s num_fh %u not equal to 1 or ds_num %u\n",
+			__func__, fl->num_fh, dsaddr->ds_num);
 		goto out;
 	}
 
@@ -714,8 +697,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
 		list_add(&dsdata->pages, &head2);
 		list_del_init(&head2);
 
-		/* Just try the first multipath data server */
-		ds = dserver.multipath->ds_list[0];
+		ds = dserver.ds;
 		dsdata->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
 		dsdata->fldata.ds_nfs_client = ds->ds_clp;
 		dsdata->args.fh = dserver.fh;
fs/nfs/nfs4filelayout.h +4 -8
index 3821989..0df1eaf 100644
@@ -42,18 +42,13 @@ struct nfs4_pnfs_ds {
 	char r_addr[29];
 };
 
-struct nfs4_multipath {
-	int 			num_ds;
-	struct nfs4_pnfs_ds	*ds_list[NFS4_PNFS_MAX_MULTI_DS];
-};
-
 struct nfs4_file_layout_dsaddr {
 	struct hlist_node	hash_node;   /* nfs4_pnfs_dev_hlist dev_list */
 	struct pnfs_deviceid	dev_id;
 	u32 			stripe_count;
 	u8			*stripe_indices;
-	u32			multipath_count;
-	struct nfs4_multipath	*multipath_list;
+	u32			ds_num;
+	struct nfs4_pnfs_ds	*ds_list[1];
 };
 
 struct nfs4_pnfs_dev_hlist {
@@ -69,7 +64,7 @@ struct nfs4_pnfs_dev_hlist {
 
 struct nfs4_pnfs_dserver {
 	struct nfs_fh        *fh;
-	struct nfs4_multipath *multipath;
+	struct nfs4_pnfs_ds	*ds;
 };
 
 struct nfs4_filelayout_segment {
@@ -97,6 +92,7 @@ struct filelayout_mount_type {
 
 extern struct pnfs_client_operations *pnfs_callback_ops;
 
+extern void print_ds(struct nfs4_pnfs_ds *ds);
 char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
 int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
 void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
fs/nfs/nfs4filelayoutdev.c +64 -87

265 lines not on this page · raw message

Signature · 1 line
1.6.6
#4 lore raw

PATCH 4/14 SQUASHME pnfs_submit: wait for I/O for data server connect

From: Andy Adamson <[email protected]>

Don't create (and maintain) a session to a data server until needed.

---
 fs/nfs/nfs4filelayoutdev.c |   34 ++++++++++++++++------------------
 1 files changed, 16 insertions(+), 18 deletions(-)
fs/nfs/nfs4filelayoutdev.c +16 -18
index ea25bd2..cda0bce 100644
@@ -458,10 +458,9 @@ nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
 static struct nfs4_pnfs_ds *
 decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 {
-	struct nfs_server *mds_srv = NFS_SB(mt->fl_sb);
 	struct nfs4_pnfs_ds *ds = NULL;
 	char r_addr[29]; /* max size of ip/port string */
-	int len, err;
+	int len;
 	u32 ip_addr, port;
 	int tmp[6];
 	uint32_t *p = *pp;
@@ -495,19 +494,6 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 
 	nfs4_pnfs_ds_add(mt, &ds, ip_addr, port, r_addr, len);
 
-	/* XXX: Don't connect to data servers here, because we
-	 * don't want any un-used (never used!) connections.
-	 * We should wait until I/O demands use of the data server.
-	 */
-	if (!ds->ds_clp) {
-		err = nfs4_pnfs_ds_create(mds_srv, ds);
-		if (err) {
-			printk(KERN_ERR "%s nfs4_pnfs_ds_create error %d\n",
-			       __func__, err);
-			goto out_err;
-		}
-	}
-
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
 out_err:
@@ -746,6 +732,7 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 {
 	struct nfs4_filelayout_segment *layout = LSEG_LD_DATA(lseg);
 	struct inode *inode = PNFS_INODE(lseg->layout);
+	struct nfs_server *mds_srv = NFS_SERVER(inode);
 	struct nfs4_file_layout_dsaddr *dsaddr;
 	u64 tmp, tmp2;
 	u32 stripe_idx, end_idx, ds_idx;
@@ -776,13 +763,24 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	BUG_ON(stripe_idx >= dsaddr->stripe_count);
 
 	ds_idx = dsaddr->stripe_indices[stripe_idx];
-	dserver->ds = dsaddr->ds_list[ds_idx];
-
-	if (dserver->ds == NULL) {
+	if (dsaddr->ds_list[ds_idx] == NULL) {
 		printk(KERN_ERR "%s: No data server for device id (%s)!! \n",
 			__func__, deviceid_fmt(&layout->dev_id));
 		return 1;
 	}
+
+	if (!dsaddr->ds_list[ds_idx]->ds_clp) {
+		int err;
+
+		err = nfs4_pnfs_ds_create(mds_srv, dsaddr->ds_list[ds_idx]);
+		if (err) {
+			printk(KERN_ERR "%s nfs4_pnfs_ds_create error %d\n",
+			       __func__, err);
+			return 1;
+		}
+	}
+	dserver->ds = dsaddr->ds_list[ds_idx];
+
 	if (layout->num_fh == 1)
 		dserver->fh = &layout->fh_array[0];
 	else
Signature · 1 line
1.6.6
#5 lore raw

PATCH 5/14 SQUASHME pnfs_submit: remove xdr macros

From: Andy Adamson <[email protected]>

As per the rest of the nfs client, remove the old xdr macros

---
 fs/nfs/nfs4filelayout.c    |   16 +++++++++-------
 fs/nfs/nfs4filelayout.h    |   11 -----------
 fs/nfs/nfs4filelayoutdev.c |   25 +++++++++++++------------
 3 files changed, 22 insertions(+), 30 deletions(-)
fs/nfs/nfs4filelayout.c +9 -7
index abdf691..20b3ae2 100644
@@ -459,8 +459,9 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 
 	dprintk("%s: set_layout_map Begin\n", __func__);
 
-	COPYMEM(&fl->dev_id, NFS4_PNFS_DEVICEID4_SIZE);
-	READ32(nfl_util);
+	memcpy(&fl->dev_id, p, NFS4_PNFS_DEVICEID4_SIZE);
+	p += XDR_QUADLEN(NFS4_PNFS_DEVICEID4_SIZE);
+	nfl_util = be32_to_cpup(p++);
 	if (nfl_util & NFL4_UFLG_COMMIT_THRU_MDS)
 		fl->commit_through_mds = 1;
 	if (nfl_util & NFL4_UFLG_DENSE)
@@ -477,9 +478,9 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 		flo->stripe_unit = fl->stripe_unit;
 	}
 
-	READ32(fl->first_stripe_index);
-	READ64(fl->pattern_offset);
-	READ32(fl->num_fh);
+	fl->first_stripe_index = be32_to_cpup(p++);
+	p = xdr_decode_hyper(p, &fl->pattern_offset);
+	fl->num_fh = be32_to_cpup(p++);
 
 	dprintk("%s: nfl_util 0x%X num_fh %u fsi %u po %llu dev_id %s\n",
 		__func__, nfl_util, fl->num_fh, fl->first_stripe_index,
@@ -499,7 +500,7 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 
 	for (i = 0; i < fl->num_fh; i++) {
 		/* fh */
-		READ32(fl->fh_array[i].size);
+		fl->fh_array[i].size = be32_to_cpup(p++);
 		if (sizeof(struct nfs_fh) < fl->fh_array[i].size) {
 			printk(KERN_ERR "Too big fh %d received %d\n",
 				i, fl->fh_array[i].size);
@@ -509,7 +510,8 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 			fl->num_fh = 0;
 			break;
 		}
-		COPYMEM(fl->fh_array[i].data, fl->fh_array[i].size);
+		memcpy(fl->fh_array[i].data, p, fl->fh_array[i].size);
+		p += XDR_QUADLEN(fl->fh_array[i].size);
 		dprintk("DEBUG: %s: fh len %d\n", __func__,
 					fl->fh_array[i].size);
 	}
fs/nfs/nfs4filelayout.h +0 -11
index 0df1eaf..04cca7e 100644
@@ -105,17 +105,6 @@ int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 u32 filelayout_dserver_get_index(loff_t offset,
 				 struct nfs4_file_layout_dsaddr *di,
 				 struct nfs4_filelayout_segment *layout);
-
-#define READ32(x)         (x) = ntohl(*p++)
-#define READ64(x)         do {			\
-	(x) = (u64)ntohl(*p++) << 32;		\
-	(x) |= ntohl(*p++);			\
-} while (0)
-#define COPYMEM(x,nbytes) do {			\
-	memcpy((x), p, nbytes);			\
-	p += XDR_QUADLEN(nbytes);		\
-} while (0)
-
 struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id);
fs/nfs/nfs4filelayoutdev.c +13 -12
index cda0bce..99a7ceb 100644
@@ -88,8 +88,8 @@ deviceid_fmt(const struct pnfs_deviceid *dev_id)
 	uint32_t *p = (uint32_t *)dev_id->data;
 	uint64_t major, minor;
 
-	READ64(major);
-	READ64(minor);
+	p = xdr_decode_hyper(p, &major);
+	p = xdr_decode_hyper(p, &minor);
 
 	sprintf(buf, "%08llu %08llu", major, minor);
 	return buf;
@@ -467,7 +467,7 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 
 	dprintk("%s enter\n", __func__);
 	/* check and skip r_netid */
-	READ32(len);
+	len = be32_to_cpup(p++);
 	/* "tcp" */
 	if (len != 3) {
 		printk("%s: ERROR: non TCP r_netid len %d\n",
@@ -476,15 +476,16 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 	}
 	/* Read the bytes into a temporary buffer */
 	/* XXX: should probably sanity check them */
-	READ32(tmp[0]);
+	tmp[0] = be32_to_cpup(p++);
 
-	READ32(len);
+	len = be32_to_cpup(p++);
 	if (len > 29) {
 		printk("%s: ERROR: Device ip/port too long (%d)\n",
 			__func__, len);
 		goto out_err;
 	}
-	COPYMEM(r_addr, len);
+	memcpy(r_addr, p, len);
+	p += XDR_QUADLEN(len);
 	*pp = p;
 	r_addr[len] = '\0';
 	sscanf(r_addr, "%d.%d.%d.%d.%d.%d", &tmp[0], &tmp[1],
@@ -513,7 +514,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
 	/* Get the stripe count (number of stripe index) */
-	READ32(cnt);
+	cnt = be32_to_cpup(p++);
 	dprintk("%s stripe count  %d\n", __func__, cnt);
 	if (cnt > NFS4_PNFS_MAX_STRIPE_CNT) {
 		printk(KERN_WARNING "%s: stripe count %d greater than "
@@ -525,7 +526,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	/* Check the multipath list count */
 	indicesp = p;
 	p += XDR_QUADLEN(cnt << 2);
-	READ32(num);
+	num = be32_to_cpup(p++);
 	dprintk("%s ds_num %u\n", __func__, num);
 	if (num > NFS4_PNFS_MAX_MULTI_CNT) {
 		printk(KERN_WARNING "%s: multipath count %d greater than "
@@ -552,7 +553,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	p = indicesp;
 	indexp = &dsaddr->stripe_indices[0];
 	for (i = 0; i < dsaddr->stripe_count; i++) {
-		READ32(dummy);
+		dummy = be32_to_cpup(p++);
 		*indexp = dummy; /* bound by NFS4_PNFS_MAX_MULTI_CNT */
 		indexp++;
 	}
@@ -562,7 +563,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	for (i = 0; i < dsaddr->ds_num; i++) {
 		int j;
 
-		READ32(dummy); /* multipath count */
+		dummy = be32_to_cpup(p++); /* multipath count */
 		if (dummy > 1) {
 			printk(KERN_WARNING
 			       "%s: Multipath count %d not supported, "
@@ -577,9 +578,9 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 			} else {
 				u32 len;
 				/* skip extra multipath */
-				READ32(len);
+				len = be32_to_cpup(p++);
 				p += XDR_QUADLEN(len);
-				READ32(len);
+				len = be32_to_cpup(p++);
 				p += XDR_QUADLEN(len);
 				continue;
 			}
Signature · 1 line
1.6.6
#6 lore raw

PATCH 6/14 SQUASHME pnfs_submit: remove unused nfs4_pnfs_device_put

From: Andy Adamson <[email protected]>

---
 fs/nfs/nfs4filelayoutdev.c |   21 ---------------------
 1 files changed, 0 insertions(+), 21 deletions(-)
fs/nfs/nfs4filelayoutdev.c +0 -21
index 99a7ceb..db6817d 100644
@@ -795,24 +795,3 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 
 	return 0;
 }
-
-/* Currently not used.
- * I have disabled checking the device count until we can think of a good way
- * to call nfs4_pnfs_device_put in a generic way from the pNFS client.
- * The only way I think think of is to put the nfs4_file_layout_dsaddr directly
- * in the nfs4_write/read_data structure, which breaks the clear line between
- * the pNFS client and layout drivers.  If I did do this, then I could call
- * an ioctl on the NFSv4 file layout driver to decrement the device count.
- */
-#if 0
-static void
-nfs4_pnfs_device_put(struct nfs_server *server,
-		     struct nfs4_pnfs_dev_hlist *hlist,
-		     struct nfs4_file_layout_dsaddr *dsaddr)
-{
-	dprintk("nfs4_pnfs_device_put: dev_id=%u\n", dsaddr->dev_id);
-	/* XXX Do we need to invoke this put_client? */
-	/* server->rpc_ops->put_client(dsaddr->clp); */
-	atomic_dec(&dsaddr->count);
-}
-#endif
Signature · 1 line
1.6.6
#7 lore raw

PATCH 7/14 SQUASHME pnfs_submit: change get_device_info parameter

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache.

---
 fs/nfs/nfs4filelayout.c    |    2 +-
 fs/nfs/nfs4filelayout.h    |    2 +-
 fs/nfs/nfs4filelayoutdev.c |    9 ++++-----
 3 files changed, 6 insertions(+), 7 deletions(-)
fs/nfs/nfs4filelayout.c +1 -1
index 20b3ae2..f222c3b 100644
@@ -399,7 +399,7 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		dsaddr = get_device_info(PNFS_INODE(lo), &fl->dev_id);
 		if (dsaddr == NULL) {
 			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
fs/nfs/nfs4filelayout.h +1 -1
index 04cca7e..560f7f6 100644
@@ -109,6 +109,6 @@ struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id);
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id);
 
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
fs/nfs/nfs4filelayoutdev.c +4 -5
index db6817d..9844a9c 100644
@@ -623,8 +623,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
  * of available devices, and return it.
  */
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt,
-		struct pnfs_deviceid *dev_id)
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id)
 {
 	struct pnfs_device *pdev = NULL;
 	int maxpages = NFS4_GETDEVINFO_MAXSIZE >> PAGE_SHIFT;
@@ -632,7 +631,7 @@ get_device_info(struct filelayout_mount_type *mt,
 	struct nfs4_file_layout_dsaddr *dsaddr = NULL;
 	int rc, i, j, minpages = 1;
 
-	dprintk("%s mt %p\n", __func__, mt);
+	dprintk("%s \n", __func__);
 	pdev = kzalloc(sizeof(struct pnfs_device), GFP_KERNEL);
 	if (pdev == NULL)
 		return NULL;
@@ -664,7 +663,7 @@ retry_once:
 	/* TODO: Update types when CB_NOTIFY_DEVICEID is available */
 	pdev->dev_notify_types = 0;
 
-	rc = pnfs_callback_ops->nfs_getdeviceinfo(mt->fl_sb, pdev);
+	rc = pnfs_callback_ops->nfs_getdeviceinfo(inode->i_sb, pdev);
 	/* Retry once with the returned mincount if a page was too small */
 	dprintk("%s getdevice info returns %d minpages %d\n", __func__, rc,
 		minpages);
@@ -682,7 +681,7 @@ retry_once:
 	/* Found new device, need to decode it and then add it to the
 	 * list of known devices for this mountpoint.
 	 */
-	dsaddr = decode_and_add_device(mt, pdev);
+	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
 out_free:
 	if (minpages > 1 && pdev->area != NULL)
 		vunmap(pdev->area);
Signature · 1 line
1.6.6
#8 lore raw

PATCH 8/14 SQUASHME pnfs_submit: change decode_and_add_device parameter

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache.

---
 fs/nfs/nfs4filelayoutdev.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
fs/nfs/nfs4filelayoutdev.c +5 -5
index 9844a9c..bb8a84a 100644
@@ -601,19 +601,19 @@ out_err:
  * Must at some point be followed up with nfs4_pnfs_device_destroy
  */
 static struct nfs4_file_layout_dsaddr*
-decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
+decode_and_add_device(struct inode *inode, struct pnfs_device *dev)
 {
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
-	dsaddr = decode_device(mt, dev);
+	dsaddr = decode_device(FILE_MT(inode), dev);
 	if (!dsaddr) {
 		printk(KERN_WARNING "%s: Could not decode device\n",
 			__func__);
-		nfs4_pnfs_device_destroy(dsaddr, mt->hlist);
+		nfs4_pnfs_device_destroy(dsaddr, FILE_MT(inode)->hlist);
 		return NULL;
 	}
 
-	if (nfs4_pnfs_device_add(mt, dsaddr))
+	if (nfs4_pnfs_device_add(FILE_MT(inode), dsaddr))
 		return NULL;
 
 	return dsaddr;
@@ -681,7 +681,7 @@ retry_once:
 	/* Found new device, need to decode it and then add it to the
 	 * list of known devices for this mountpoint.
 	 */
-	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
+	dsaddr = decode_and_add_device(inode, pdev);
 out_free:
 	if (minpages > 1 && pdev->area != NULL)
 		vunmap(pdev->area);
Signature · 1 line
1.6.6
#9 lore raw

PATCH 9/14 SQUASHME pnfs_submit: change decode_device parameter

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache

---
 fs/nfs/nfs4filelayoutdev.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)
fs/nfs/nfs4filelayoutdev.c +5 -4
index bb8a84a..506ede1 100644
@@ -505,7 +505,7 @@ out_err:
 /* Decode opaque device data and return the result
  */
 static struct nfs4_file_layout_dsaddr*
-decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
+decode_device(struct inode *ino, struct pnfs_device *pdev)
 {
 	int i, dummy;
 	u32 cnt, num;
@@ -572,7 +572,8 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 		}
 		for (j = 0; j < dummy; j++) {
 			if (j == 0) {
-				dsaddr->ds_list[i] = decode_and_add_ds(&p, mt);
+				dsaddr->ds_list[i] = decode_and_add_ds(&p,
+								FILE_MT(ino));
 				if (dsaddr->ds_list[i] == NULL)
 					goto out_err_free;
 			} else {
@@ -589,7 +590,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	return dsaddr;
 
 out_err_free:
-	nfs4_pnfs_device_destroy(dsaddr, mt->hlist);
+	nfs4_pnfs_device_destroy(dsaddr, FILE_MT(ino)->hlist);
 out_err:
 	dprintk("%s ERROR: returning NULL\n", __func__);
 	return NULL;
@@ -605,7 +606,7 @@ decode_and_add_device(struct inode *inode, struct pnfs_device *dev)
 {
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
-	dsaddr = decode_device(FILE_MT(inode), dev);
+	dsaddr = decode_device(inode, dev);
 	if (!dsaddr) {
 		printk(KERN_WARNING "%s: Could not decode device\n",
 			__func__);
Signature · 1 line
1.6.6
#10 lore raw

PATCH 10/14 SQUASHME pnfs_submit: change decode_and_add_ds parameter

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache

---
 fs/nfs/nfs4filelayoutdev.c |    7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)
fs/nfs/nfs4filelayoutdev.c +3 -4
index 506ede1..4882149 100644
@@ -456,7 +456,7 @@ nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
 }
 
 static struct nfs4_pnfs_ds *
-decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
+decode_and_add_ds(uint32_t **pp, struct inode *inode)
 {
 	struct nfs4_pnfs_ds *ds = NULL;
 	char r_addr[29]; /* max size of ip/port string */
@@ -493,7 +493,7 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 	ip_addr = htonl((tmp[0]<<24) | (tmp[1]<<16) | (tmp[2]<<8) | (tmp[3]));
 	port = htons((tmp[4] << 8) | (tmp[5]));
 
-	nfs4_pnfs_ds_add(mt, &ds, ip_addr, port, r_addr, len);
+	nfs4_pnfs_ds_add(FILE_MT(inode), &ds, ip_addr, port, r_addr, len);
 
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
@@ -572,8 +572,7 @@ decode_device(struct inode *ino, struct pnfs_device *pdev)
 		}
 		for (j = 0; j < dummy; j++) {
 			if (j == 0) {
-				dsaddr->ds_list[i] = decode_and_add_ds(&p,
-								FILE_MT(ino));
+				dsaddr->ds_list[i] = decode_and_add_ds(&p,ino);
 				if (dsaddr->ds_list[i] == NULL)
 					goto out_err_free;
 			} else {
Signature · 1 line
1.6.6
#11 lore raw

PATCH 11/14 SQUASHME pnfs_submit: change nfs4_pnfs_ds_add parameter

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache

---
 fs/nfs/nfs4filelayoutdev.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
fs/nfs/nfs4filelayoutdev.c +3 -3
index 4882149..2cbe931 100644
@@ -416,11 +416,11 @@ nfs4_pnfs_device_add(struct filelayout_mount_type *mt,
 }
 
 static void
-nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
+nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 		 u32 ip_addr, u32 port, char *r_addr, int len)
 {
 	struct nfs4_pnfs_ds *tmp_ds, *ds;
-	struct nfs4_pnfs_dev_hlist *hlist = mt->hlist;
+	struct nfs4_pnfs_dev_hlist *hlist = FILE_MT(inode)->hlist;
 
 	*dsp = NULL;
 
@@ -493,7 +493,7 @@ decode_and_add_ds(uint32_t **pp, struct inode *inode)
 	ip_addr = htonl((tmp[0]<<24) | (tmp[1]<<16) | (tmp[2]<<8) | (tmp[3]));
 	port = htons((tmp[4] << 8) | (tmp[5]));
 
-	nfs4_pnfs_ds_add(FILE_MT(inode), &ds, ip_addr, port, r_addr, len);
+	nfs4_pnfs_ds_add(inode, &ds, ip_addr, port, r_addr, len);
 
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
Signature · 1 line
1.6.6
#12 lore raw

PATCH 12/14 SQUASHME pnfs_submit: remove filelayout CB_NOTIFY_DEVICE support

From: Andy Adamson <[email protected]>

We won't support CB_NOTIFY_DEVICE in the first submission stage. Remove filelayout_device_delete and associated functionality.

---
 fs/nfs/nfs4filelayout.c    |   16 ----------------
 fs/nfs/nfs4filelayout.h    |    2 --
 fs/nfs/nfs4filelayoutdev.c |   18 ------------------
 3 files changed, 0 insertions(+), 36 deletions(-)
fs/nfs/nfs4filelayout.c +0 -16
index f222c3b..0530b59 100644
@@ -135,21 +135,6 @@ filelayout_uninitialize_mountpoint(struct pnfs_mount_type *mountid)
 	return 0;
 }
 
-int
-filelayout_device_delete(struct pnfs_mount_type *mountid,
-				struct pnfs_deviceid *dev_id)
-{
-	struct filelayout_mount_type *fl_mt = NULL;
-
-	if (mountid) {
-		fl_mt = (struct filelayout_mount_type *)mountid->mountid;
-
-		if (fl_mt != NULL)
-			nfs4_pnfs_dev_destroy(fl_mt->hlist, dev_id);
-	}
-	return 0;
-}
-
 /* This function is used by the layout driver to calculate the
  * offset of the file on the dserver based on whether the
  * layout type is STRIPE_DENSE or STRIPE_SPARSE
@@ -797,7 +782,6 @@ struct layoutdriver_io_operations filelayout_io_operations = {
 	.free_lseg               = filelayout_free_lseg,
 	.initialize_mountpoint   = filelayout_initialize_mountpoint,
 	.uninitialize_mountpoint = filelayout_uninitialize_mountpoint,
-	.device_delete           = filelayout_device_delete,
 };
 
 struct layoutdriver_policy_operations filelayout_policy_operations = {
fs/nfs/nfs4filelayout.h +0 -2
index 560f7f6..e988891 100644
@@ -96,8 +96,6 @@ extern void print_ds(struct nfs4_pnfs_ds *ds);
 char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
 int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
 void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
-void nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-				struct pnfs_deviceid *dev_id);
 int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 			  loff_t offset,
 			  size_t count,
fs/nfs/nfs4filelayoutdev.c +0 -18
index 2cbe931..2d98ca3 100644
@@ -368,24 +368,6 @@ nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist)
 	}
 }
 
-/* De-alloc a devices for a mount point. */
-void
-nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-			struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	if (hlist == NULL)
-		return;
-
-	dprintk("%s: dev_id=%s\n", __func__, deviceid_fmt(dev_id));
-
-	dsaddr = nfs4_pnfs_device_item_find(hlist, dev_id);
-	if (dsaddr)
-		/* nfs4_pnfs_device_destroy grabs hlist->dev_lock */
-		nfs4_pnfs_device_destroy(dsaddr, hlist);
-}
-
 /*
  * Add the device to the list of available devices for this mount point.
  * The * rpc client is created during first I/O.
Signature · 1 line
1.6.6
#13 lore raw

PATCH 13/14 SQUASHME pnfs_submit: stand alone data server cache

From: Andy Adamson <[email protected]>

Data servers can be associated with any device ID from any server. Multiple file layout meta data servers can share a data server. Implement a stand alone data server cache to share data servers and their session between deviceid's and meta data servers.

The data server cache is only changed at GETDEVICEINFO return or umount. Once in the cache and referenced by a deviceid structure, the use of the data server in the I/O path is via deviceid struct array indexes. Therefore, nether an hlist nor rwspin locks are needed.

---
 fs/nfs/nfs4filelayout.h    |    3 +-
 fs/nfs/nfs4filelayoutdev.c |   57 ++++++++++++++++---------------------------
 2 files changed, 22 insertions(+), 38 deletions(-)
fs/nfs/nfs4filelayout.h +1 -2
index e988891..809747f 100644
@@ -34,7 +34,7 @@ enum stripetype4 {
 
 /* Individual ip address */
 struct nfs4_pnfs_ds {
-	struct hlist_node 	ds_node;  /* nfs4_pnfs_dev_hlist dev_dslist */
+	struct list_head 	ds_node;  /* nfs4_pnfs_dev_hlist dev_dslist */
 	u32 			ds_ip_addr;
 	u32 			ds_port;
 	struct nfs_client	*ds_clp;
@@ -54,7 +54,6 @@ struct nfs4_file_layout_dsaddr {
 struct nfs4_pnfs_dev_hlist {
 	rwlock_t		dev_lock;
 	struct hlist_head	dev_list[NFS4_PNFS_DEV_HASH_SIZE];
-	struct hlist_head	dev_dslist[NFS4_PNFS_DEV_HASH_SIZE];
 };
 
 /*
fs/nfs/nfs4filelayoutdev.c +21 -36

126 lines not on this page · raw message

Signature · 1 line
1.6.6
#14 lore raw

PATCH 14/14 SQUASHME pnfs_submit: remove kfree from under spinlock

From: Andy Adamson <[email protected]>

---
 fs/nfs/nfs4filelayoutdev.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
fs/nfs/nfs4filelayoutdev.c +3 -3
index 50e2865..5c13dc5 100644
@@ -410,9 +410,7 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 				ds->ds_ip_addr);
 		list_add(&ds->ds_node, &nfs4_data_server_cache);
 		*dsp = ds;
-	}
-	if (tmp_ds != NULL) {
-		destroy_ds(ds);
+	} else {
 		atomic_inc(&tmp_ds->ds_count);
 		dprintk("%s data server found ip 0x%x, inc'ed ds_count to %d\n",
 				__func__, tmp_ds->ds_ip_addr,
@@ -420,6 +418,8 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 		*dsp = tmp_ds;
 	}
 	spin_unlock(&nfs4_ds_cache_lock);
+	if (tmp_ds != NULL)
+		destroy_ds(ds);
 }
 
 static struct nfs4_pnfs_ds *
Signature · 1 line
1.6.6
#15 lore raw

Re: [pnfs] [PATCH 12/14] SQUASHME pnfs_submit: remove filelayout CB_NOTIFY_DEVICE support

In reply to [email protected]

OK. I'll keep a patch reverting this patch (reintroducing this functionality) in the pnfs branch, on top of the pnfs-submit branch.

Benny

Quoted 88 lines · [email protected] wrote:

From: Andy Adamson <[email protected]>

We won't support CB_NOTIFY_DEVICE in the first submission stage. Remove filelayout_device_delete and associated functionality.

---
 fs/nfs/nfs4filelayout.c    |   16 ----------------
 fs/nfs/nfs4filelayout.h    |    2 --
 fs/nfs/nfs4filelayoutdev.c |   18 ------------------
 3 files changed, 0 insertions(+), 36 deletions(-)
fs/nfs/nfs4filelayout.c +0 -16
index f222c3b..0530b59 100644
@@ -135,21 +135,6 @@ filelayout_uninitialize_mountpoint(struct pnfs_mount_type *mountid)
 	return 0;
 }
 
-int
-filelayout_device_delete(struct pnfs_mount_type *mountid,
-				struct pnfs_deviceid *dev_id)
-{
-	struct filelayout_mount_type *fl_mt = NULL;
-
-	if (mountid) {
-		fl_mt = (struct filelayout_mount_type *)mountid->mountid;
-
-		if (fl_mt != NULL)
-			nfs4_pnfs_dev_destroy(fl_mt->hlist, dev_id);
-	}
-	return 0;
-}
-
 /* This function is used by the layout driver to calculate the
  * offset of the file on the dserver based on whether the
  * layout type is STRIPE_DENSE or STRIPE_SPARSE
@@ -797,7 +782,6 @@ struct layoutdriver_io_operations filelayout_io_operations = {
 	.free_lseg               = filelayout_free_lseg,
 	.initialize_mountpoint   = filelayout_initialize_mountpoint,
 	.uninitialize_mountpoint = filelayout_uninitialize_mountpoint,
-	.device_delete           = filelayout_device_delete,
 };
 
 struct layoutdriver_policy_operations filelayout_policy_operations = {
fs/nfs/nfs4filelayout.h +0 -2
index 560f7f6..e988891 100644
@@ -96,8 +96,6 @@ extern void print_ds(struct nfs4_pnfs_ds *ds);
 char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
 int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
 void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
-void nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-				struct pnfs_deviceid *dev_id);
 int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 			  loff_t offset,
 			  size_t count,
fs/nfs/nfs4filelayoutdev.c +0 -18
index 2cbe931..2d98ca3 100644
@@ -368,24 +368,6 @@ nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist)
 	}
 }
 
-/* De-alloc a devices for a mount point. */
-void
-nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-			struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	if (hlist == NULL)
-		return;
-
-	dprintk("%s: dev_id=%s\n", __func__, deviceid_fmt(dev_id));
-
-	dsaddr = nfs4_pnfs_device_item_find(hlist, dev_id);
-	if (dsaddr)
-		/* nfs4_pnfs_device_destroy grabs hlist->dev_lock */
-		nfs4_pnfs_device_destroy(dsaddr, hlist);
-}
-
 /*
  * Add the device to the list of available devices for this mount point.
  * The * rpc client is created during first I/O.
#16 lore raw

Re: [pnfs] [PATCH 07/14] SQUASHME pnfs_submit: change get_device_info parameter

In reply to [email protected]
Quoted 51 lines · On Apr. 08, 2010, 22:54 +0300, [email protected] wrote:

From: Andy Adamson <[email protected]>

In preparation to use the generic deviceid cache.

---
 fs/nfs/nfs4filelayout.c    |    2 +-
 fs/nfs/nfs4filelayout.h    |    2 +-
 fs/nfs/nfs4filelayoutdev.c |    9 ++++-----
 3 files changed, 6 insertions(+), 7 deletions(-)
fs/nfs/nfs4filelayout.c +1 -1
index 20b3ae2..f222c3b 100644
@@ -399,7 +399,7 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		dsaddr = get_device_info(PNFS_INODE(lo), &fl->dev_id);
 		if (dsaddr == NULL) {
 			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
fs/nfs/nfs4filelayout.h +1 -1
index 04cca7e..560f7f6 100644
@@ -109,6 +109,6 @@ struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id);
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id);
 
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
fs/nfs/nfs4filelayoutdev.c +1 -2
index db6817d..9844a9c 100644
@@ -623,8 +623,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
  * of available devices, and return it.
  */
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt,
-		struct pnfs_deviceid *dev_id)
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id)
 {
 	struct pnfs_device *pdev = NULL;
 	int maxpages = NFS4_GETDEVINFO_MAXSIZE >> PAGE_SHIFT;

fs/nfs/nfs4filelayoutdev.c get_device_info(struct filelayout_mount_type *mt, · line 634

Quoted 5 lines
fs/nfs/nfs4filelayoutdev.c +1 -1
@@ -632,7 +631,7 @@ get_device_info(struct filelayout_mount_type *mt,
 	struct nfs4_file_layout_dsaddr *dsaddr = NULL;
 	int rc, i, j, minpages = 1;
 
-	dprintk("%s mt %p\n", __func__, mt);
+	dprintk("%s \n", __func__);

nit: checkpatch rightfully complains about the stray space character before the newline. I'll just remove that upon committing, but there are other places
like that we need to clean up before submitting the final version...

Benny

Quoted 21 lines
fs/nfs/nfs4filelayoutdev.c +2 -2
 	pdev = kzalloc(sizeof(struct pnfs_device), GFP_KERNEL);
 	if (pdev == NULL)
 		return NULL;
@@ -664,7 +663,7 @@ retry_once:
 	/* TODO: Update types when CB_NOTIFY_DEVICEID is available */
 	pdev->dev_notify_types = 0;
 
-	rc = pnfs_callback_ops->nfs_getdeviceinfo(mt->fl_sb, pdev);
+	rc = pnfs_callback_ops->nfs_getdeviceinfo(inode->i_sb, pdev);
 	/* Retry once with the returned mincount if a page was too small */
 	dprintk("%s getdevice info returns %d minpages %d\n", __func__, rc,
 		minpages);
@@ -682,7 +681,7 @@ retry_once:
 	/* Found new device, need to decode it and then add it to the
 	 * list of known devices for this mountpoint.
 	 */
-	dsaddr = decode_and_add_device(mt, pdev);
+	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
 out_free:
 	if (minpages > 1 && pdev->area != NULL)
 		vunmap(pdev->area);
#17 lore raw

Re: [pnfs] [PATCH 02/14] SQUASHME pnfs-submit: call get_device_info from filelayout_check

In reply to [email protected]
Quoted 55 lines · On Apr. 08, 2010, 22:53 +0300, [email protected] wrote:

From: Andy Adamson <[email protected]>

filelayout_check() validates a LAYOUTGET return including the deviceid. If the deviceid is not cached, GETDEVICEINFO is called.

Once cached, the deviceid will only be uncached upon umount or a return of all refering layouts. So, there is no need for a get_device_info call in the I/O paths.

Remove nfs4_file_layout_dsaddr_get.

---
 fs/nfs/nfs4filelayout.c    |   11 +++++++----
 fs/nfs/nfs4filelayout.h    |    6 ++++--
 fs/nfs/nfs4filelayoutdev.c |   27 +++------------------------
 3 files changed, 14 insertions(+), 30 deletions(-)
fs/nfs/nfs4filelayout.c +7 -4
index df6c1d0..ffc3ef2 100644
@@ -413,12 +413,15 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	struct nfs_server *nfss = NFS_SERVER(PNFS_INODE(lo));
 
 	dprintk("--> %s\n", __func__);
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(PNFS_INODE(lo)),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dprintk("%s NO device for dev_id %s\n",
+		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		if (dsaddr == NULL) {
+			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
-		goto out;
+			goto out;
+		}
 	}
 	if (fl->first_stripe_index < 0 ||
 	    fl->first_stripe_index > dsaddr->stripe_count) {
@@ -645,7 +648,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
 	stripesz = filelayout_get_stripesize(layoutid);
 	dprintk("%s stripesize %Zd\n", __func__, stripesz);
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(data->inode),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(data->inode)->hlist,
 					     &nfslay->dev_id);
 	if (dsaddr == NULL) {
 		data->pdata.pnfs_error = -EIO;
fs/nfs/nfs4filelayout.h +0 -0
index d003809..3821989 100644

fs/nfs/nfs4filelayout.h u32 filelayout_dserver_get_index(loff_t offset, · line 128

Quoted 10 lines
fs/nfs/nfs4filelayout.h +4 -2
@@ -121,7 +121,9 @@ u32 filelayout_dserver_get_index(loff_t offset,
 } while (0)
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *,
-			  struct pnfs_deviceid *);
+nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
+			   struct pnfs_deviceid *dev_id);
+struct nfs4_file_layout_dsaddr *
+get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
 

Andy, this function name is too generic IMO.
better be something more specific to pnfs and file layout. (I'm committing this patchset as is so we can fix that later)

Benny

Quoted 60 lines
fs/nfs/nfs4filelayout.h +0 -0
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
fs/nfs/nfs4filelayoutdev.c +3 -24
index e557679..ebb9845 100644
@@ -52,13 +52,6 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
 
-struct nfs4_file_layout_dsaddr *nfs4_file_layout_dsaddr_get(
-					struct filelayout_mount_type *mt,
-					struct pnfs_deviceid *dev_id);
-struct nfs4_file_layout_dsaddr *nfs4_pnfs_device_item_find(
-					struct nfs4_pnfs_dev_hlist *hlist,
-					struct pnfs_deviceid *dev_id);
-
 void
 print_ds_list(struct nfs4_multipath *multipath)
 {
@@ -665,7 +658,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
 /* Retrieve the information for dev_id, add it to the list
  * of available devices, and return it.
  */
-static struct nfs4_file_layout_dsaddr *
+struct nfs4_file_layout_dsaddr *
 get_device_info(struct filelayout_mount_type *mt,
 		struct pnfs_deviceid *dev_id)
 {
@@ -737,21 +730,6 @@ out_free:
 }
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *mt,
-			    struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	read_lock(&mt->hlist->dev_lock);
-	dsaddr = _device_lookup(mt->hlist, dev_id);
-	read_unlock(&mt->hlist->dev_lock);
-
-	if (dsaddr == NULL)
-		dsaddr = get_device_info(mt, dev_id);
-	return dsaddr;
-}
-
-struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id)
 {
@@ -798,7 +776,8 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	if (!layout)
 		return 1;
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(inode), &layout->dev_id);
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(inode)->hlist,
+					    &layout->dev_id);
 	if (dsaddr == NULL)
 		return 1;
 
#18 lore raw
In reply to [email protected]
Quoted 6 lines · On Thu, Apr 08, 2010 at 03:53:56PM -0400, [email protected] wrote:

Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches the file layout driver.

This set of patches implements several design changes enmerated at
http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.

(Should be:

http://wiki.linux-nfs.org/wiki/index.php/Proposed_Device_Management_Design

.)

Quoted 39 lines

- Only call GETDEVICEINFO from one place
- Only cache the first data server in the multipath_list4 array.
- Wait to connect to data server when needed for I/O
- Move data server cache to stand alone cache

Also some cleanup, and function interfaces are changed in preparation for the soon to come generic deviceid cache.

0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch 0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch 0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch 0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch 0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch 0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch 0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch 0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch 0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch 0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch 0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch 0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch 0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch 0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch

Testing:
CONFIG_NFS_V4_1 set:
pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
Connecatathon tests pass. pyNFS testclient tests pass.

CONFIG_NFS_V4_1 not set:
Connecatahon tests pass.

-->Andy

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

#19 lore raw
In reply to [email protected]

Committed to a new branch in the 2.6.34-rc3 stream: pnfs-submit

Thanks!

Benny

Quoted 45 lines · On Apr. 08, 2010, 22:53 +0300, [email protected] wrote:

Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches the file layout driver.

This set of patches implements several design changes enmerated at
http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.

- Only call GETDEVICEINFO from one place
- Only cache the first data server in the multipath_list4 array.
- Wait to connect to data server when needed for I/O
- Move data server cache to stand alone cache

Also some cleanup, and function interfaces are changed in preparation for the soon to come generic deviceid cache.

0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch 0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch 0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch 0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch 0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch 0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch 0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch 0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch 0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch 0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch 0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch 0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch 0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch 0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch

Testing:
CONFIG_NFS_V4_1 set:
pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
Connecatathon tests pass. pyNFS testclient tests pass.

CONFIG_NFS_V4_1 not set:
Connecatahon tests pass.

-->Andy

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs