pnfs-obj: Important BUG fixing also for @Stable

Hi Trond

I'm sending in two patches that fixes a bad hung, memory leak, a WARN_ON and plain wrong protocol handling. In regard to objlayout error handling. These are also aimed at Stable@ for the 3.2 Kernel.
(Actually these bugs are in since 3.0, I might produce another version
for 3.[1,0] after these go in)

I have simulated errors at various layers, reads and writes, in combination of different tests, and they work as I expect. .i.e IO continues to MDS and oblayout is returned on error.

Please include these patches for 3.3 and they can go to Stable@ for 3.2 has well.

list of patches:
[PATCH 1/2] pnfs-obj: pNFS errors are communicated on iodata->pnfs_error

This is just a stupidity and neglect. Benny can you understand what happened here?

[PATCH 2/2] pnfs-obj: Must return layout on IO error

This patch is mainly for pnfs.c but I think it is very simple and totally not affecting anybody but objlayout. So it should be safe, and does fixes my main problem.
Fred please review?

Please Review

Thanks
Boaz

Some time along the way pNFS IO errors were switched to communicate with a special iodata->pnfs_error member instead of the regular RPC members. But objlayout was not switched over.

Fix that!
Without this fix any IO error is hanged, because IO is not switched to MDS and pages are never cleared or read.

---
 fs/nfs/objlayout/objlayout.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

index 72074e3..b3c2903 100644
@@ -254,6 +254,8 @@ objlayout_read_done(struct objlayout_io_res *oir, ssize_t status, bool sync)
 	oir->status = rdata->task.tk_status = status;
 	if (status >= 0)
 		rdata->res.count = status;
+	else
+		rdata->pnfs_error = status;
 	objlayout_iodone(oir);
 	/* must not use oir after this point */
 

@@ -334,6 +336,8 @@ objlayout_write_done(struct objlayout_io_res *oir, ssize_t status, bool sync)
 	if (status >= 0) {
 		wdata->res.count = status;
 		wdata->verf.committed = oir->committed;
+	} else {
+		wdata->pnfs_error = status;
 	}
 	objlayout_iodone(oir);
 	/* must not use oir after this point */

1.7.2.3

As mandated by the standard. In case of an IO error, a pNFS objects layout driver must return it's layout. This is because all device errors are reported to the server as part of the layout return buffer.

This is implemented the same way PNFS_LAYOUTRET_ON_SETATTR is done, through a bit flag on the pnfs_layoutdriver_type->flags member. The flag is set by the layout driver that wants a layout_return preformed at pnfs_ld_{write,read}_done in case of an error.
(Though I have not defined a wrapper like pnfs_ld_layoutret_on_setattr
because this code is never called outside of pnfs.c and pnfs IO paths)

Without this patch 3.[0-2] Kernels leak memory and have an annoying WARN_ON after every IO error utilizing the pnfs-obj driver.

---
 fs/nfs/objlayout/objio_osd.c |    3 ++-
 fs/nfs/pnfs.c                |   12 ++++++++++++
 fs/nfs/pnfs.h                |    1 +
 3 files changed, 15 insertions(+), 1 deletions(-)

index c807ab9..55d0128 100644
@@ -551,7 +551,8 @@ static const struct nfs_pageio_ops objio_pg_write_ops = {
 static struct pnfs_layoutdriver_type objlayout_type = {
 	.id = LAYOUT_OSD2_OBJECTS,
 	.name = "LAYOUT_OSD2_OBJECTS",
-	.flags                   = PNFS_LAYOUTRET_ON_SETATTR,
+	.flags                   = PNFS_LAYOUTRET_ON_SETATTR |
+				   PNFS_LAYOUTRET_ON_ERROR,
 
 	.alloc_layout_hdr        = objlayout_alloc_layout_hdr,
 	.free_layout_hdr         = objlayout_free_layout_hdr,

index 8e672a2..f881a63 100644
@@ -1178,6 +1178,15 @@ void pnfs_ld_write_done(struct nfs_write_data *data)
 		put_lseg(data->lseg);
 		data->lseg = NULL;
 		dprintk("pnfs write error = %d\n", data->pnfs_error);
+		if (NFS_SERVER(data->inode)->pnfs_curr_ld->flags &
+						PNFS_LAYOUTRET_ON_ERROR) {
+			/* Don't lo_commit on error, Server will needs to
+			 * preform a file recovery.
+			 */
+			clear_bit(NFS_INO_LAYOUTCOMMIT,
+				  &NFS_I(data->inode)->flags);
+			pnfs_return_layout(data->inode);
+		}
 	}
 	data->mds_ops->rpc_release(data);
 }

@@ -1267,6 +1276,9 @@ static void pnfs_ld_handle_read_error(struct nfs_read_data *data)
 	put_lseg(data->lseg);
 	data->lseg = NULL;
 	dprintk("pnfs write error = %d\n", data->pnfs_error);
+	if (NFS_SERVER(data->inode)->pnfs_curr_ld->flags &
+						PNFS_LAYOUTRET_ON_ERROR)
+		pnfs_return_layout(data->inode);
 
 	nfs_pageio_init_read_mds(&pgio, data->inode);
 

index 1509530..53d593a 100644
@@ -68,6 +68,7 @@ enum {
 enum layoutdriver_policy_flags {
 	/* Should the pNFS client commit and return the layout upon a setattr */
 	PNFS_LAYOUTRET_ON_SETATTR	= 1 << 0,
+	PNFS_LAYOUTRET_ON_ERROR		= 1 << 1,
 };
 
 struct nfs4_deviceid_node;

1.7.2.3

Fixes included in pnfs-all-3.2-2012-01-08

Thanks,

Benny