Skip to contentSkip to conversation

backported patches for pnfs-all-2.6.33

PATCH New Benny Halevy 1 participant 6 messages May 27, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover backported patches for pnfs-all-2.6.33

The following patches were backported to pnfs-all-2.6.33-2010-05-27:

commit 1d7e6862446faa85a6783c91a9da892ac93222d2

Author: Boaz Harrosh <[email protected]>
Date:   Thu May 27 17:52:24 2010 +0300

pnfs-obj: panlayout: Fix very old BUG_ONs on ol_state.status

OK This is definitely my stupidity when converting panfs_shim to the new objlayout_read/write_done() API. But that was very, very long time ago.

So I fixed the stale check on ol_state.status which is never set until much later inside the call to generic layer.

While at it I converted the BUG_ONs to WARN_ONs because it could be data corruption but otherwise it will not crush the Kernel. Better continue to be able to debug it better. (And added missing information to the WARN_ON)

Congratulation, I've successfully ran all tests over panfs-export/panfs_shim over real Panasas HW. With latest code. [ This is with the new panfs-export that is also compatible with
the std objects layout driver. Patches to that will follow]

TODO:
I should also simulate/cause some IO errors and see that errors are reported at layout_return

commit 422e0e67b2933a47afd2e426eee1ff8922de4135

Author: Tao Guo <[email protected]>
Date:   Tue May 25 16:33:52 2010 +0300

pnfs: unlock lo_lock before calling layoutdriver's setup_layoutcommit

So in blocklayoutdriver, we can use GFP_KERNEL to do memory allocation in bl_setup_layoutcommit().

The state protected by the lo_lock here is clear now, which is the layout_commit's position and state. .i.e write_begin/end_pos, nfsi->lo_cred, and the call to pnfs_get_layout_stateid.

commit a19e718c6be6b1e10627f1985483b9cff2bca53b

Author: Andy Adamson <[email protected]>
Date:   Mon May 24 14:28:46 2010 -0400

pnfs-submit: replace layoutcommit_ctx with rpc_cred

Taking a reference on the nfs_open_context to signal that a layoutcommit is needed is problematic because the last reference to the context triggers a close (nfs_release). But, if the layout holds a reference on the nfs_open_context, then close will not be triggered.

Since we only use the rpc credential from the layoutcommit_ctx, replace the layoutcommit_ctx with the rpc_cred.

Hold a reference on the rpc_cred until the layoutcommit rpc returns.

If the layoutdriver fails to setup the layoutcommit, clear the layoutcommit properties and put the credential.

[s/do_layoutcommit/layoutcommit_needed/]

commit d947cd6af78f5426bd6833d513df4aa95e7a116e

Author: Benny Halevy <[email protected]>
Date:   Wed Apr 14 19:05:13 2010 +0300
    pnfs: async layout return
    
    FIXME: currently there's only one path that requires async layout return
    to avoid blocking of the rpciod like the stack below.
    
    We really neee the pnfs state machine instead.
    
    rpciod/0      D 0000003d8c2d83b0     0  1029      2 0x00000000
    778ccdd0 60264f00 77a6a000 778ccb20 77a6ba20 6001375b 77a6ba20 77a6a000
           77a6a000 778cc8a0 77a6ba70 601b192f 77a6ba50 601b35a6 6ed537f0 77a6a000
           77a6bb00 63506fd8 7b1870c8 77a6bb10 77a6ba90 7b187100 63506fd8 77a6a000
    Call Trace:
    77a6b9f8:  [<6001375b>] _switch_to+0x5e/0xae
    77a6ba28:  [<601b192f>] schedule+0x1dd/0x21b
    77a6ba38:  [<601b35a6>] _raw_spin_unlock_irqrestore+0x18/0x1c
    77a6ba60:  [<7b1870c8>] rpc_wait_bit_killable+0x0/0x3c [sunrpc]
    77a6ba78:  [<7b187100>] rpc_wait_bit_killable+0x38/0x3c [sunrpc]
    77a6ba98:  [<601b1d9f>] __wait_on_bit+0x43/0x76
    77a6bae8:  [<601b1e43>] out_of_line_wait_on_bit+0x71/0x7c
    77a6baf8:  [<7b1870c8>] rpc_wait_bit_killable+0x0/0x3c [sunrpc]
    77a6bb20:  [<600430d6>] wake_bit_function+0x0/0x2e
    77a6bb68:  [<7b18709d>] __rpc_wait_for_completion_task+0x34/0x36 [sunrpc]
    77a6bb78:  [<7bf36048>] nfs4_wait_for_completion_rpc_task+0xb/0xd [nfs]
    77a6bb88:  [<7bf369ac>] pnfs4_proc_layoutreturn+0xae/0xe6 [nfs]
    77a6bbc8:  [<6007c8b1>] kmem_cache_alloc+0xaf/0xbe
    77a6bc08:  [<7bf4dc39>] _pnfs_return_layout+0x450/0x4d4 [nfs]
    77a6bc38:  [<7bf3cb79>] decode_attr_time+0x17/0x4d [nfs]
    77a6bc58:  [<7bf429d7>] decode_getfattr+0xaae/0xc02 [nfs]
    77a6bcb8:  [<7bf4601b>] __nfs4_close+0x15d/0x17a [nfs]
    77a6bd28:  [<7bf46053>] nfs4_close_state+0xb/0xd [nfs]
    77a6bd38:  [<7bf34183>] nfs4_close_context+0x26/0x28 [nfs]
    77a6bd48:  [<7bf2315c>] __put_nfs_open_context+0x79/0xa1 [nfs]
    77a6bd78:  [<7bf23227>] put_nfs_open_context+0xb/0xd [nfs]
    77a6bd88:  [<7bf4b763>] pnfs_layoutcommit_done+0xa5/0xad [nfs]
    77a6bdb8:  [<7bf3b12e>] pnfs_layoutcommit_rpc_done+0x39/0x56 [nfs]
    77a6bde8:  [<7b18712b>] rpc_exit_task+0x27/0x52 [sunrpc]
    77a6be08:  [<7b187817>] __rpc_execute+0x88/0x23a [sunrpc]
    77a6be30:  [<7b1879ee>] rpc_async_schedule+0x0/0x12 [sunrpc]
    77a6be48:  [<7b1879fe>] rpc_async_schedule+0x10/0x12 [sunrpc]
    77a6be58:  [<6003fc45>] worker_thread+0x114/0x1a5
    77a6be80:  [<600430a2>] autoremove_wake_function+0x0/0x34
    77a6beb8:  [<6003fb31>] worker_thread+0x0/0x1a5
    77a6bed8:  [<60042cf7>] kthread+0x8e/0x96
    77a6bf48:  [<60021429>] run_kernel_thread+0x41/0x4a
    77a6bf58:  [<60042c69>] kthread+0x0/0x96
    77a6bf98:  [<60021410>] run_kernel_thread+0x28/0x4a
    77a6bfc8:  [<600136d3>] new_thread_handler+0x71/0x9b
    
    [missing from FIXME: async layout return]
    Signed-off-by: Benny Halevy <[email protected]>

commit 7ccc11a731bc2f8021add9907b89654c9146bb32

Author: Zhang Jingwang <[email protected]>
Date:   Tue May 11 16:59:44 2010 +0800

pnfs-submit: send layoutcommit until IO completed when return layout

If we call layoutcommit before IO completed, then some extents we are writing may not be in the commit list. When we return layout, they may not get a chance to be committed to server.

#1 lore raw

PATCH 1/5 pnfs-submit: send layoutcommit until IO completed when return layout

From: Zhang Jingwang <[email protected]>

If we call layoutcommit before IO completed, then some extents we are writing may not be in the commit list. When we return layout, they may not get a chance to be committed to server.

---
 fs/nfs/pnfs.c |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)
fs/nfs/pnfs.c +10 -10
index b4aed4e..6819509 100644
@@ -731,16 +731,6 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 		arg.length = ~0;
 	}
 	if (type == RETURN_FILE) {
-		if (nfsi->layoutcommit_ctx) {
-			status = pnfs_layoutcommit_inode(ino, 1);
-			if (status) {
-				dprintk("%s: layoutcommit failed, status=%d. "
-					"Returning layout anyway\n",
-					__func__, status);
-				status = 0;
-			}
-		}
-
 		lo = get_lock_current_layout(nfsi);
 		if (lo && !has_layout_to_return(lo, &arg)) {
 			put_unlock_current_layout(lo);
@@ -768,6 +758,16 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 			wait_event(nfsi->lo_waitq,
 				!pnfs_return_layout_barrier(nfsi, &arg));
 		}
+
+		if (nfsi->layoutcommit_ctx) {
+			status = pnfs_layoutcommit_inode(ino, 1);
+			if (status) {
+				dprintk("%s: layoutcommit failed, status=%d. "
+					"Returning layout anyway\n",
+					__func__, status);
+				status = 0;
+			}
+		}
 	}
 send_return:
 	status = return_layout(ino, &arg, stateid, type, lo);
Signature · 1 line
1.6.6.1
#2 lore raw

PATCH 2/5 pnfs: async layout return

In reply to Benny Halevy

FIXME: currently there's only one path that requires async layout return to avoid blocking of the rpciod like the stack below.

We really neee the pnfs state machine instead.

rpciod/0 D 0000003d8c2d83b0 0 1029 2 0x00000000
778ccdd0 60264f00 77a6a000 778ccb20 77a6ba20 6001375b 77a6ba20 77a6a000

       77a6a000 778cc8a0 77a6ba70 601b192f 77a6ba50 601b35a6 6ed537f0 77a6a000
       77a6bb00 63506fd8 7b1870c8 77a6bb10 77a6ba90 7b187100 63506fd8 77a6a000
Call Trace:
77a6b9f8:  [<6001375b>] _switch_to+0x5e/0xae
77a6ba28:  [<601b192f>] schedule+0x1dd/0x21b
77a6ba38:  [<601b35a6>] _raw_spin_unlock_irqrestore+0x18/0x1c
77a6ba60:  [<7b1870c8>] rpc_wait_bit_killable+0x0/0x3c [sunrpc]
77a6ba78:  [<7b187100>] rpc_wait_bit_killable+0x38/0x3c [sunrpc]
77a6ba98:  [<601b1d9f>] __wait_on_bit+0x43/0x76
77a6bae8:  [<601b1e43>] out_of_line_wait_on_bit+0x71/0x7c
77a6baf8:  [<7b1870c8>] rpc_wait_bit_killable+0x0/0x3c [sunrpc]
77a6bb20:  [<600430d6>] wake_bit_function+0x0/0x2e
77a6bb68:  [<7b18709d>] __rpc_wait_for_completion_task+0x34/0x36 [sunrpc]
77a6bb78:  [<7bf36048>] nfs4_wait_for_completion_rpc_task+0xb/0xd [nfs]
77a6bb88:  [<7bf369ac>] pnfs4_proc_layoutreturn+0xae/0xe6 [nfs]
77a6bbc8:  [<6007c8b1>] kmem_cache_alloc+0xaf/0xbe
77a6bc08:  [<7bf4dc39>] _pnfs_return_layout+0x450/0x4d4 [nfs]
77a6bc38:  [<7bf3cb79>] decode_attr_time+0x17/0x4d [nfs]
77a6bc58:  [<7bf429d7>] decode_getfattr+0xaae/0xc02 [nfs]
77a6bcb8:  [<7bf4601b>] __nfs4_close+0x15d/0x17a [nfs]
77a6bd28:  [<7bf46053>] nfs4_close_state+0xb/0xd [nfs]
77a6bd38:  [<7bf34183>] nfs4_close_context+0x26/0x28 [nfs]
77a6bd48:  [<7bf2315c>] __put_nfs_open_context+0x79/0xa1 [nfs]
77a6bd78:  [<7bf23227>] put_nfs_open_context+0xb/0xd [nfs]
77a6bd88:  [<7bf4b763>] pnfs_layoutcommit_done+0xa5/0xad [nfs]
77a6bdb8:  [<7bf3b12e>] pnfs_layoutcommit_rpc_done+0x39/0x56 [nfs]
77a6bde8:  [<7b18712b>] rpc_exit_task+0x27/0x52 [sunrpc]
77a6be08:  [<7b187817>] __rpc_execute+0x88/0x23a [sunrpc]
77a6be30:  [<7b1879ee>] rpc_async_schedule+0x0/0x12 [sunrpc]
77a6be48:  [<7b1879fe>] rpc_async_schedule+0x10/0x12 [sunrpc]
77a6be58:  [<6003fc45>] worker_thread+0x114/0x1a5
77a6be80:  [<600430a2>] autoremove_wake_function+0x0/0x34
77a6beb8:  [<6003fb31>] worker_thread+0x0/0x1a5
77a6bed8:  [<60042cf7>] kthread+0x8e/0x96
77a6bf48:  [<60021429>] run_kernel_thread+0x41/0x4a
77a6bf58:  [<60042c69>] kthread+0x0/0x96
77a6bf98:  [<60021410>] run_kernel_thread+0x28/0x4a
77a6bfc8:  [<600136d3>] new_thread_handler+0x71/0x9b
[missing from FIXME: async layout return]
---
 fs/nfs/callback_proc.c |    7 ++++---
 fs/nfs/inode.c         |    2 +-
 fs/nfs/nfs4proc.c      |   17 ++++++++++-------
 fs/nfs/nfs4state.c     |    4 ++--
 fs/nfs/pnfs.c          |   21 +++++++++++++--------
 fs/nfs/pnfs.h          |    9 +++++----
 6 files changed, 35 insertions(+), 25 deletions(-)
fs/nfs/callback_proc.c +4 -3
index b3aa9e4..05e4592 100644
@@ -209,7 +209,7 @@ static int pnfs_recall_layout(void *data)
 
 	if (rl.cbl_recall_type == RETURN_FILE) {
 		status = pnfs_return_layout(inode, &rl.cbl_seg, &rl.cbl_stateid,
-					    RETURN_FILE);
+					    RETURN_FILE, true);
 		if (status)
 			dprintk("%s RETURN_FILE error: %d\n", __func__, status);
 		goto out;
@@ -221,12 +221,13 @@ static int pnfs_recall_layout(void *data)
 	/* FIXME: This loop is inefficient, running in O(|s_inodes|^2) */
 	while ((ino = nfs_layoutrecall_find_inode(clp, &rl)) != NULL) {
 		/* XXX need to check status on pnfs_return_layout */
-		pnfs_return_layout(ino, &rl.cbl_seg, NULL, RETURN_FILE);
+		pnfs_return_layout(ino, &rl.cbl_seg, NULL, RETURN_FILE, true);
 		iput(ino);
 	}
 
 	/* send final layoutreturn */
-	status = pnfs_return_layout(inode, &rl.cbl_seg, NULL, rl.cbl_recall_type);
+	status = pnfs_return_layout(inode, &rl.cbl_seg, NULL,
+				    rl.cbl_recall_type, true);
 	if (status)
 		printk(KERN_INFO "%s: ignoring pnfs_return_layout status=%d\n",
 				__func__, status);
fs/nfs/inode.c +1 -1
index a574492..8661036 100644
@@ -1388,7 +1388,7 @@ void nfs4_clear_inode(struct inode *inode)
 	/* First call standard NFS clear_inode() code */
 	nfs_clear_inode(inode);
 #ifdef CONFIG_PNFS
-	pnfs_return_layout(inode, NULL, NULL, RETURN_FILE);
+	pnfs_return_layout(inode, NULL, NULL, RETURN_FILE, true);
 #endif /* CONFIG_PNFS */
 }
 #endif
fs/nfs/nfs4proc.c +10 -7
index 162a119..3324a35 100644
@@ -1079,7 +1079,8 @@ static void pnfs4_layout_reclaim(struct nfs4_state *state)
 	/* FIXME: send gratuitous layout commits and return with the reclaim
 	 * flag during grace period
 	 */
-	pnfs_return_layout(state->inode, NULL, &state->open_stateid, RETURN_FILE);
+	pnfs_return_layout(state->inode, NULL, &state->open_stateid,
+			   RETURN_FILE, true);
 	pnfs_set_layout_stateid(&NFS_I(state->inode)->layout, &zero_stateid);
 #endif /* CONFIG_PNFS */
 }
@@ -2371,7 +2372,7 @@ pnfs4_proc_setattr(struct dentry *dentry, struct nfs_fattr *fattr,
 
 	if (pnfs_enabled_sb(server) && has_layout(nfsi) &&
 	    pnfs_ld_layoutret_on_setattr(server->pnfs_curr_ld))
-		pnfs_return_layout(inode, NULL, NULL, RETURN_FILE);
+		pnfs_return_layout(inode, NULL, NULL, RETURN_FILE, true);
 	return nfs4_proc_setattr(dentry, fattr, sattr);
 }
 #endif /* CONFIG_PNFS */
@@ -5725,7 +5726,7 @@ static const struct rpc_call_ops nfs4_pnfs_layoutreturn_call_ops = {
 	.rpc_release = nfs4_pnfs_layoutreturn_release,
 };
 
-int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp)
+int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait)
 {
 	struct inode *ino = lrp->args.inode;
 	struct nfs_server *server = NFS_SERVER(ino);
@@ -5742,7 +5743,7 @@ int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp)
 		.callback_data = lrp,
 		.flags = RPC_TASK_ASYNC,
 	};
-	int status;
+	int status = 0;
 
 	dprintk("--> %s\n", __func__);
 	lrp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE;
@@ -5751,9 +5752,11 @@ int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp)
 		status = PTR_ERR(task);
 		goto out;
 	}
-	status = nfs4_wait_for_completion_rpc_task(task);
-	if (status == 0)
-		status = task->tk_status;
+	if (wait) {
+		status = nfs4_wait_for_completion_rpc_task(task);
+		if (status == 0)
+			status = task->tk_status;
+	}
 	rpc_put_task(task);
 out:
 	dprintk("<-- %s\n", __func__);
fs/nfs/nfs4state.c +2 -2
index 30bff3b..a415e9d 100644
@@ -590,7 +590,7 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm
 		struct nfs_inode *nfsi = NFS_I(state->inode);
 
 		if (nfsi->layoutcommit_ctx)
-			pnfs_layoutcommit_inode(state->inode, 0);
+			pnfs_layoutcommit_inode(state->inode, wait);
 		if (has_layout(nfsi) && nfsi->layout.roc_iomode) {
 			struct nfs4_pnfs_layout_segment range;
 
@@ -598,7 +598,7 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm
 			range.offset = 0;
 			range.length = NFS4_MAX_UINT64;
 			pnfs_return_layout(state->inode, &range, NULL,
-					   RETURN_FILE);
+					   RETURN_FILE, wait);
 		}
 #endif /* CONFIG_PNFS */
 		nfs4_do_close(path, state, wait);
fs/nfs/pnfs.c +13 -8
index 6819509..51a4a9c 100644
@@ -680,7 +680,8 @@ pnfs_return_layout_barrier(struct nfs_inode *nfsi,
 static int
 return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 	      const nfs4_stateid *stateid, /* optional */
-	      enum pnfs_layoutreturn_type type, struct pnfs_layout_type *lo)
+	      enum pnfs_layoutreturn_type type, struct pnfs_layout_type *lo,
+	      bool wait)
 {
 	struct nfs4_pnfs_layoutreturn *lrp;
 	struct nfs_server *server = NFS_SERVER(ino);
@@ -705,7 +706,7 @@ return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 	else if (lo)
 		pnfs_get_layout_stateid(&lrp->args.stateid, lo);
 
-	status = pnfs4_proc_layoutreturn(lrp);
+	status = pnfs4_proc_layoutreturn(lrp, wait);
 out:
 	dprintk("<-- %s status: %d\n", __func__, status);
 	return status;
@@ -714,7 +715,8 @@ out:
 int
 _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 		    const nfs4_stateid *stateid, /* optional */
-		    enum pnfs_layoutreturn_type type)
+		    enum pnfs_layoutreturn_type type,
+		    bool wait)
 {
 	struct pnfs_layout_type *lo = NULL;
 	struct nfs_inode *nfsi = NFS_I(ino);
@@ -760,7 +762,7 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 		}
 
 		if (nfsi->layoutcommit_ctx) {
-			status = pnfs_layoutcommit_inode(ino, 1);
+			status = pnfs_layoutcommit_inode(ino, wait);
 			if (status) {
 				dprintk("%s: layoutcommit failed, status=%d. "
 					"Returning layout anyway\n",
@@ -770,7 +772,7 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 		}
 	}
 send_return:
-	status = return_layout(ino, &arg, stateid, type, lo);
+	status = return_layout(ino, &arg, stateid, type, lo, wait);
 out:
 	dprintk("<-- %s status: %d\n", __func__, status);
 	return status;
@@ -1697,7 +1699,8 @@ pnfs_writeback_done(struct nfs_write_data *data)
 			.length = data->args.count,
 		};
 		dprintk("%s: retrying\n", __func__);
-		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE);
+		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE,
+				    true);
 		pnfs_initiate_write(data, NFS_CLIENT(data->inode),
 				    pdata->call_ops, pdata->how);
 	}
@@ -1828,7 +1831,8 @@ pnfs_read_done(struct nfs_read_data *data)
 			.length = data->args.count,
 		};
 		dprintk("%s: retrying\n", __func__);
-		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE);
+		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE,
+				    true);
 		pnfs_initiate_read(data, NFS_CLIENT(data->inode),
 				   pdata->call_ops);
 	}
@@ -2054,7 +2058,8 @@ pnfs_commit_done(struct nfs_write_data *data)
 			.length = data->args.count,
 		};
 		dprintk("%s: retrying\n", __func__);
-		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE);
+		_pnfs_return_layout(data->inode, &range, NULL, RETURN_FILE,
+				    true);
 		pnfs_initiate_commit(data, NFS_CLIENT(data->inode),
 				     pdata->call_ops, pdata->how);
 	}
fs/nfs/pnfs.h +5 -4
index 1adf357..832ddf8 100644
@@ -29,7 +29,7 @@ extern int nfs4_pnfs_getdeviceinfo(struct super_block *sb,
 				   struct pnfs_device *dev);
 extern int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp);
 extern int pnfs4_proc_layoutcommit(struct pnfs_layoutcommit_data *data);
-extern int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp);
+extern int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait);
 
 /* pnfs.c */
 extern const nfs4_stateid zero_stateid;
@@ -40,7 +40,7 @@ int pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
 
 int _pnfs_return_layout(struct inode *, struct nfs4_pnfs_layout_segment *,
 			const nfs4_stateid *stateid, /* optional */
-			enum pnfs_layoutreturn_type);
+			enum pnfs_layoutreturn_type, bool wait);
 void set_pnfs_layoutdriver(struct super_block *sb, struct nfs_fh *fh, u32 id);
 void unmount_pnfs_layoutdriver(struct super_block *sb);
 int pnfs_use_read(struct inode *inode, ssize_t count);
@@ -244,14 +244,15 @@ static inline void pnfs_modify_new_request(struct nfs_page *req,
 static inline int pnfs_return_layout(struct inode *ino,
 				     struct nfs4_pnfs_layout_segment *lseg,
 				     const nfs4_stateid *stateid, /* optional */
-				     enum pnfs_layoutreturn_type type)
+				     enum pnfs_layoutreturn_type type,
+				     bool wait)
 {
 	struct nfs_inode *nfsi = NFS_I(ino);
 	struct nfs_server *nfss = NFS_SERVER(ino);
 
 	if (pnfs_enabled_sb(nfss) &&
 	    (type != RETURN_FILE || has_layout(nfsi)))
-		return _pnfs_return_layout(ino, lseg, stateid, type);
+		return _pnfs_return_layout(ino, lseg, stateid, type, wait);
 
 	return 0;
 }
Signature · 1 line
1.6.6.1
#3 lore raw

PATCH 3/5 pnfs-submit: replace layoutcommit_ctx with rpc_cred

In reply to Benny Halevy

From: Andy Adamson <[email protected]>

Taking a reference on the nfs_open_context to signal that a layoutcommit is needed is problematic because the last reference to the context triggers a close (nfs_release). But, if the layout holds a reference on the nfs_open_context, then close will not be triggered.

Since we only use the rpc credential from the layoutcommit_ctx, replace the layoutcommit_ctx with the rpc_cred.

Hold a reference on the rpc_cred until the layoutcommit rpc returns.

If the layoutdriver fails to setup the layoutcommit, clear the layoutcommit properties and put the credential.

[s/do_layoutcommit/layoutcommit_needed/]
---
 fs/nfs/inode.c            |    4 ++--
 fs/nfs/nfs4state.c        |    2 +-
 fs/nfs/pnfs.c             |   42 +++++++++++++++++-------------------------
 fs/nfs/write.c            |    2 +-
 include/linux/nfs4_pnfs.h |    6 ++++++
 include/linux/nfs_fs.h    |    3 +--
 include/linux/pnfs_xdr.h  |    1 -
 7 files changed, 28 insertions(+), 32 deletions(-)
fs/nfs/inode.c +2 -2
index 8661036..15c60fc 100644
@@ -1178,7 +1178,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
 	/*
 	 * file needs layout commit, server attributes may be stale
 	 */
-	if (nfsi->layoutcommit_ctx && nfsi->change_attr >= fattr->change_attr) {
+	if (layoutcommit_needed(nfsi) && nfsi->change_attr >= fattr->change_attr) {
 		dprintk("NFS: %s: layoutcommit is needed for file %s/%ld\n",
 			__func__, inode->i_sb->s_id, inode->i_ino);
 		return 0;
@@ -1399,7 +1399,7 @@ static void pnfs_alloc_init_inode(struct nfs_inode *nfsi)
 	nfsi->pnfs_layout_state = 0;
 	memset(&nfsi->layout.stateid, 0, NFS4_STATEID_SIZE);
 	nfsi->layout.roc_iomode = 0;
-	nfsi->layoutcommit_ctx = NULL;
+	nfsi->lo_cred = NULL;
 	nfsi->pnfs_write_begin_pos = 0;
 	nfsi->pnfs_write_end_pos = 0;
 #endif /* CONFIG_PNFS */
fs/nfs/nfs4state.c +1 -1
index a415e9d..9116e45 100644
@@ -589,7 +589,7 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm
 #ifdef CONFIG_PNFS
 		struct nfs_inode *nfsi = NFS_I(state->inode);
 
-		if (nfsi->layoutcommit_ctx)
+		if (layoutcommit_needed(nfsi))
 			pnfs_layoutcommit_inode(state->inode, wait);
 		if (has_layout(nfsi) && nfsi->layout.roc_iomode) {
 			struct nfs4_pnfs_layout_segment range;
fs/nfs/pnfs.c +17 -25

94 lines not on this page · raw message

fs/nfs/write.c +1 -1
index 4f01089..3a7679f 100644
@@ -1555,7 +1555,7 @@ long nfs_sync_mapping_wait(struct address_space *mapping, struct writeback_contr
 	} while (ret >= 0);
 	spin_unlock(&inode->i_lock);
 #ifdef CONFIG_PNFS
-	if (how == 0 && ret == 0 && NFS_I(inode)->layoutcommit_ctx) {
+	if (how == 0 && ret == 0 && layoutcommit_needed(NFS_I(inode))) {
 		dprintk("%s calling layoutcommit\n", __func__);
 		ret = pnfs_layoutcommit_inode(inode, 1);
 	}
include/linux/nfs4_pnfs.h +6 -0
index d997b69..514c6a3 100644
@@ -100,6 +100,12 @@ has_layout(struct nfs_inode *nfsi)
 	return nfsi->layout.ld_data != NULL;
 }
 
+static inline bool
+layoutcommit_needed(struct nfs_inode *nfsi)
+{
+	return nfsi->lo_cred != NULL;
+}
+
 struct pnfs_layout_segment {
 	struct list_head fi_list;
 	struct nfs4_pnfs_layout_segment range;
include/linux/nfs_fs.h +1 -2
index dd57a6a..aeb5e87 100644
@@ -205,11 +205,10 @@ struct nfs_inode {
 #define NFS_INO_RW_LAYOUT_FAILED 1 	/* get rw layout failed stop trying */
 #define NFS_INO_LAYOUT_ALLOC     2	/* bit lock for layout allocation */
 	time_t pnfs_layout_suspend;
+	struct rpc_cred		*lo_cred; /* layoutcommit credential */
 	wait_queue_head_t lo_waitq;
 	spinlock_t lo_lock;
 	struct pnfs_layout_type layout;
-	/* use rpc_creds in this open_context to send LAYOUTCOMMIT to MDS */
-	struct nfs_open_context *layoutcommit_ctx;
 	/* DH: These vars keep track of the maximum write range
 	 * so the values can be used for layoutcommit.
 	 */
include/linux/pnfs_xdr.h +0 -1
index 4f34aa8..ecfe5c2 100644
@@ -86,7 +86,6 @@ struct pnfs_layoutcommit_data {
 	bool is_sync;
 	struct rpc_cred *cred;
 	struct nfs_fattr fattr;
-	struct nfs_open_context *ctx;
 	struct pnfs_layoutcommit_arg args;
 	struct pnfs_layoutcommit_res res;
 	int status;
Signature · 1 line
1.6.6.1
#4 lore raw

PATCH 4/5 pnfs: unlock lo_lock before calling layoutdriver's setup_layoutcommit

In reply to Benny Halevy

From: Tao Guo <[email protected]>

So in blocklayoutdriver, we can use GFP_KERNEL to do memory allocation in bl_setup_layoutcommit().

The state protected by the lo_lock here is clear now, which is the layout_commit's position and state. .i.e write_begin/end_pos, nfsi->lo_cred, and the call to pnfs_get_layout_stateid.

---
 fs/nfs/pnfs.c |   66 +++++++++++++++++++++++++++++---------------------------
 1 files changed, 34 insertions(+), 32 deletions(-)
fs/nfs/pnfs.c +34 -32
index dafd66e..80dbb8e 100644
@@ -2161,15 +2161,22 @@ pnfs_layoutcommit_done(struct pnfs_layoutcommit_data *data)
  * Set up the argument/result storage required for the RPC call.
  */
 static int
-pnfs_layoutcommit_setup(struct pnfs_layoutcommit_data *data, int sync)
+pnfs_layoutcommit_setup(struct inode *inode,
+			struct pnfs_layoutcommit_data *data,
+			loff_t write_begin_pos, loff_t write_end_pos, int sync)
 {
-	struct nfs_inode *nfsi = NFS_I(data->args.inode);
-	struct nfs_server *nfss = NFS_SERVER(data->args.inode);
+	struct nfs_inode *nfsi = NFS_I(inode);
+	struct nfs_server *nfss = NFS_SERVER(inode);
 	int result = 0;
 
 	dprintk("%s Begin (sync:%d)\n", __func__, sync);
-	data->args.fh = NFS_FH(data->args.inode);
+
+	data->is_sync = sync;
+	data->args.inode = inode;
+	data->args.fh = NFS_FH(inode);
 	data->args.layout_type = nfss->pnfs_curr_ld->id;
+	data->res.fattr = &data->fattr;
+	nfs_fattr_init(&data->fattr);
 
 	/* TODO: Need to determine the correct values */
 	data->args.time_modify_changed = 0;
@@ -2177,26 +2184,19 @@ pnfs_layoutcommit_setup(struct pnfs_layoutcommit_data *data, int sync)
 	/* Set values from inode so it can be reset
 	 */
 	data->args.lseg.iomode = IOMODE_RW;
-	data->args.lseg.offset = nfsi->pnfs_write_begin_pos;
-	data->args.lseg.length = nfsi->pnfs_write_end_pos - nfsi->pnfs_write_begin_pos + 1;
-	data->args.lastbytewritten =  min(nfsi->pnfs_write_end_pos,
-					i_size_read(&nfsi->vfs_inode) - 1);
+	data->args.lseg.offset = write_begin_pos;
+	data->args.lseg.length = write_end_pos - write_begin_pos + 1;
+	data->args.lastbytewritten =  min(write_end_pos,
+					  i_size_read(inode) - 1);
 	data->args.bitmask = nfss->attr_bitmask;
 	data->res.server = nfss;
 
 	/* Call layout driver to set the arguments.
 	 */
-	if (nfss->pnfs_curr_ld->ld_io_ops->setup_layoutcommit) {
+	if (nfss->pnfs_curr_ld->ld_io_ops->setup_layoutcommit)
 		result = nfss->pnfs_curr_ld->ld_io_ops->setup_layoutcommit(
 				&nfsi->layout, &data->args);
-		if (result)
-			goto out;
-	}
-	pnfs_get_layout_stateid(&data->args.stateid, &nfsi->layout);
-	data->res.fattr = &data->fattr;
-	nfs_fattr_init(&data->fattr);
 
-out:
 	dprintk("%s End Status %d\n", __func__, result);
 	return result;
 }
@@ -2208,6 +2208,9 @@ pnfs_layoutcommit_inode(struct inode *inode, int sync)
 {
 	struct pnfs_layoutcommit_data *data;
 	struct nfs_inode *nfsi = NFS_I(inode);
+	loff_t write_begin_pos;
+	loff_t write_end_pos;
+
 	int status = 0;
 
 	dprintk("%s Begin (sync:%d)\n", __func__, sync);
@@ -2219,39 +2222,38 @@ pnfs_layoutcommit_inode(struct inode *inode, int sync)
 		return -ENOMEM;
 
 	spin_lock(&nfsi->lo_lock);
-	if (!layoutcommit_needed(nfsi))
-		goto out_unlock;
-
-	data->args.inode = inode;
-	data->cred = nfsi->lo_cred;
-
-	/* Set up layout commit args*/
-	status = pnfs_layoutcommit_setup(data, sync);
+	if (!layoutcommit_needed(nfsi)) {
+		spin_unlock(&nfsi->lo_lock);
+		goto out_free;
+	}
 
 	/* Clear layoutcommit properties in the inode so
 	 * new lc info can be generated
 	 */
+	write_begin_pos = nfsi->pnfs_write_begin_pos;
+	write_end_pos = nfsi->pnfs_write_end_pos;
+	data->cred = nfsi->lo_cred;
 	nfsi->pnfs_write_begin_pos = 0;
 	nfsi->pnfs_write_end_pos = 0;
 	nfsi->lo_cred = NULL;
+	pnfs_get_layout_stateid(&data->args.stateid, &nfsi->layout);
 
+	spin_unlock(&nfsi->lo_lock);
+
+	/* Set up layout commit args */
+	status = pnfs_layoutcommit_setup(inode, data, write_begin_pos,
+					 write_end_pos, sync);
 	if (status) {
 		/* The layout driver failed to setup the layoutcommit */
 		put_rpccred(data->cred);
-		goto out_unlock;
+		goto out_free;
 	}
-
-	/* release lock on pnfs layoutcommit attrs */
-	spin_unlock(&nfsi->lo_lock);
-
-	data->is_sync = sync;
 	status = pnfs4_proc_layoutcommit(data);
 out:
 	dprintk("%s end (err:%d)\n", __func__, status);
 	return status;
-out_unlock:
+out_free:
 	pnfs_layoutcommit_free(data);
-	spin_unlock(&nfsi->lo_lock);
 	goto out;
 }
 
Signature · 1 line
1.6.6.1
#5 lore raw

PATCH 5/5 pnfs-obj: panlayout: Fix very old BUG_ONs on ol_state.status

In reply to Benny Halevy

From: Boaz Harrosh <[email protected]>

OK This is definitely my stupidity when converting panfs_shim to the new objlayout_read/write_done() API. But that was very, very long time ago.

So I fixed the stale check on ol_state.status which is never set until much later inside the call to generic layer.

While at it I converted the BUG_ONs to WARN_ONs because it could be data corruption but otherwise it will not crush the Kernel. Better continue to be able to debug it better. (And added missing information to the WARN_ON)

Congratulation, I've successfully ran all tests over panfs-export/panfs_shim over real Panasas HW. With latest code. [ This is with the new panfs-export that is also compatible with
the std objects layout driver. Patches to that will follow]

TODO:
I should also simulate/cause some IO errors and see that errors are reported at layout_return

---
 fs/nfs/objlayout/panfs_shim.c |    9 +++------
 1 files changed, 3 insertions(+), 6 deletions(-)
fs/nfs/objlayout/panfs_shim.c +3 -6
index 414831e..7bb00aa 100644
@@ -421,9 +421,7 @@ panfs_shim_read_done(
 		rc = res_p->result;
 	if (rc == PAN_SUCCESS) {
 		status = res_p->length;
-		BUG_ON(state->ol_state.status < 0);
-		BUG_ON((pan_stor_len_t)state->ol_state.status !=
-		       state->u.read.res.length);
+		WARN_ON(status < 0);
 	} else {
 		status = -panfs_export_ops->convert_rc(rc);
 		dprintk("%s: pan_sam_read rc %d: status %Zd\n",
@@ -499,9 +497,8 @@ panfs_shim_write_done(
 	if (rc == PAN_SUCCESS) {
 		state->ol_state.committed = NFS_FILE_SYNC;
 		status = res_p->length;
-		BUG_ON(state->ol_state.status < 0);
-		BUG_ON((pan_stor_len_t)state->ol_state.status !=
-		       state->u.write.res.length);
+		WARN_ON(status < 0);
+
 		objlayout_add_delta_space_used(&state->ol_state,
 					       res_p->delta_capacity_used);
 	} else {
Signature · 1 line
1.6.6.1