Skip to contentSkip to conversation

pnfs-submit: cleanup layoutcommit verison 2

PATCH In review [email protected] 2 participants 7 messages Jun 3, 2010 – Jun 8, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover pnfs-submit: cleanup layoutcommit verison 2

Responded to comments.

This is against the pnfs-submit branch of the 2.6.34 tree. They will need to be applied against the 2.6.35-rc1 tree which I can do after comments.

A cleanup, and call the async error handler. 0001-SQUASHME-pnfs-submit-cleanup-layoutcommit-call.patch 0002-SQUASHME-pnfs-submit-handle-async-layoutcommit-erro.patch

These next two patches moves the pnfs_layoutcommit_inode call to nfs_write_inode, and it is the only call other than in layoutreturn. (removed calls in __nfs4_close, nfs_commit_inode, nfs_wb_sync).

0003-SQUASHME-pnfs-remove-ifdef-around-layoutcommit_neede.patch 0004-SQUASHME-pnfs-submit-move-layoutcommit-to-nfs_write.patch

pnfs_layoutcommit_inode is called after nfs_commit_unstable_pages() so that if LAYOUTCOMMIT fails, the unstable pages have been processed..

The error handlers (sync and async) call nfs4_map_errors, so unhandled errors (such as NFS4ERR_BADLAYOUT) get returned to nfs_write_ioode as -EIO.

We will add code to the error handlers for errors such as NFS4ERR_BADLAYOUT that require us to stop using and free the layout, and redo the I/O through the MDS.

Testing:
With CONFIG_NFS_V4_1 set
NFSv4.1/pnfs passed Connectathon against write enabled GFS2/pNFS. Note: there were exactly the same number of LAYOUTCOMMITS sent as were sent with pnfs_layoutcommit_inode being called from __nfs4_close (never happened), nfs_commit_inode and nfs_wb_sync.

V4.0 mount passes Connectathon tests

With CONFIG_NFS_V4_1 not set

V4.0 mount passes Connectathon tests

-->Andy

#1 lore raw

PATCH 1/4 SQUASHME pnfs-submit: cleanup layoutcommit call

From: Andy Adamson <[email protected]>

pnfs_layoutcommit_inode prints its status upon exit.

---
 fs/nfs/pnfs.c |   12 +++---------
 1 files changed, 3 insertions(+), 9 deletions(-)
fs/nfs/pnfs.c +3 -9
index 8cc4412..3e940ec 100644
@@ -766,15 +766,9 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range,
 				!pnfs_return_layout_barrier(nfsi, &arg));
 		}
 
-		if (layoutcommit_needed(nfsi)) {
-			status = pnfs_layoutcommit_inode(ino, wait);
-			if (status) {
-				dprintk("%s: layoutcommit failed, status=%d. "
-					"Returning layout anyway\n",
-					__func__, status);
-				status = 0;
-			}
-		}
+		if (layoutcommit_needed(nfsi))
+			/* Return layout even if layoutcommit fails */
+			pnfs_layoutcommit_inode(ino, wait);
 	}
 send_return:
 	status = return_layout(ino, &arg, stateid, type, lo, wait);
Signature · 1 line
1.6.2.5
#2 lore raw

PATCH 2/4 SQUASHME pnfs-submit: handle async layoutcommit errors

From: Andy Adamson <[email protected]>

nfs4_async_handle_error handles session level and grace,delay errors.

NOTE: Layout specific errors (NFS4ERR_BADIOMODE, NFS4ERR_BAD_LAYOUT, NFS4ERR_UNKNOWN_LAYOUTTYPE) need to be handled.

---
 fs/nfs/nfs4proc.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfs/nfs4proc.c +3 -0
index 8b375a7..19ac602 100644
@@ -5628,6 +5628,9 @@ pnfs_layoutcommit_rpc_done(struct rpc_task *task, void *calldata)
 	pnfs_layoutcommit_done(data);
 
 	nfs4_sequence_done(server, &data->res.seq_res, task->tk_status);
+
+	if (nfs4_async_handle_error(task, server, NULL) == -EAGAIN)
+		nfs_restart_rpc(task, server->nfs_client);
 }
 
 static void pnfs_layoutcommit_release(void *lcdata)
Signature · 1 line
1.6.2.5
#3 lore raw

PATCH 3/4 SQUASHME pnfs remove ifdef around layoutcommit_needed

From: Andy Adamson <[email protected]>

---
 fs/nfs/inode.c            |    2 --
 include/linux/nfs4_pnfs.h |    8 ++++++++
 2 files changed, 8 insertions(+), 2 deletions(-)
fs/nfs/inode.c +0 -2
index c93b756..086d46f 100644
@@ -1106,7 +1106,6 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
 			!test_bit(NFS_INO_MOUNTPOINT, &nfsi->flags))
 		server->fsid = fattr->fsid;
 
-#ifdef CONFIG_NFS_V4_1
 	/*
 	 * file needs layout commit, server attributes may be stale
 	 */
@@ -1115,7 +1114,6 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
 			__func__, inode->i_sb->s_id, inode->i_ino);
 		return 0;
 	}
-#endif /* CONFIG_NFS_V4_1 */
 	/*
 	 * Update the read time so we don't revalidate too often.
 	 */
include/linux/nfs4_pnfs.h +8 -0
index 84d2e95..c248ae9 100644
@@ -86,6 +86,14 @@ layoutcommit_needed(struct nfs_inode *nfsi)
 	return nfsi->lo_cred != NULL;
 }
 
+#else /* CONFIG_NFS_V4_1 */
+
+static inline bool
+layoutcommit_needed(struct nfs_inode *nfsi)
+{
+	return 0;
+}
+
 #endif /* CONFIG_NFS_V4_1 */
 
 struct pnfs_layout_segment {
Signature · 1 line
1.6.2.5
#4 lore raw

PATCH 4/4 SQUASHME pnfs-submit: move layoutcommit to nfs_write_inode

From: Andy Adamson <[email protected]>

The LAYOUTCOMMIT call indicates an update to the file meta data is needed, and should be called when clearing the I_DIRTY_SYNC state.

A call to LAYOUTCOMMIT in nfs_write_inode replaces the calls in nfs_wb_all, nfs_commit_inode, and __nfs4_close.

---
 fs/nfs/nfs4state.c |    2 --
 fs/nfs/pnfs.h      |    4 ++++
 fs/nfs/write.c     |   24 ++++++++++++------------
 3 files changed, 16 insertions(+), 14 deletions(-)
fs/nfs/nfs4state.c +0 -2
index d0dbdd4..e1207fa 100644
@@ -589,8 +589,6 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm
 #ifdef CONFIG_NFS_V4_1
 		struct nfs_inode *nfsi = NFS_I(state->inode);
 
-		if (layoutcommit_needed(nfsi))
-			pnfs_layoutcommit_inode(state->inode, wait);
 		if (has_layout(nfsi) && nfsi->layout.roc_iomode) {
 			struct nfs4_pnfs_layout_segment range;
 
fs/nfs/pnfs.h +4 -0
index c89be78..009393e 100644
@@ -256,6 +256,10 @@ static inline int pnfs_use_rpc(struct nfs_server *nfss)
 	return 1;
 }
 
+static inline int pnfs_layoutcommit_inode(struct inode *inode, int sync)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 
 #endif /* FS_NFS_PNFS_H */
fs/nfs/write.c +12 -12
index 0fd33cb..513b308 100644
@@ -1489,13 +1489,6 @@ static int nfs_commit_inode(struct inode *inode, int how)
 			wait_on_bit(&NFS_I(inode)->flags, NFS_INO_COMMIT,
 					nfs_wait_bit_killable,
 					TASK_KILLABLE);
-#ifdef CONFIG_NFS_V4_1
-		if (may_wait && layoutcommit_needed(NFS_I(inode))) {
-			error = pnfs_layoutcommit_inode(inode, 1);
-			if (error < 0)
-				return error;
-		}
-#endif /* CONFIG_NFS_V4_1 */
 	} else
 		nfs_commit_clear_lock(NFS_I(inode));
 out:
@@ -1545,7 +1538,18 @@ static int nfs_commit_unstable_pages(struct inode *inode, struct writeback_contr
 
 int nfs_write_inode(struct inode *inode, struct writeback_control *wbc)
 {
-	return nfs_commit_unstable_pages(inode, wbc);
+	int ret;
+	ret = nfs_commit_unstable_pages(inode, wbc);
+	if (ret >= 0 && layoutcommit_needed(NFS_I(inode))) {
+		int err, sync = wbc->sync_mode;
+
+		if (wbc->nonblocking || wbc->for_background)
+			sync = 0;
+		err = pnfs_layoutcommit_inode(inode, sync);
+		if (err < 0)
+			ret = err;
+	}
+	return ret;
 }
 
 /*
@@ -1562,10 +1566,6 @@ int nfs_wb_all(struct inode *inode)
 	};
 
 	ret = sync_inode(inode, &wbc);
-#ifdef CONFIG_NFS_V4_1
-	if (!ret && layoutcommit_needed(NFS_I(inode)))
-		ret = pnfs_layoutcommit_inode(inode, 1);
-#endif
 	return ret;
 }
 
Signature · 1 line
1.6.2.5
#5 lore raw

Re: [PATCH 4/4] SQUASHME pnfs-submit: move layoutcommit to nfs_write_inode

Quoted 9 lines · On 06/03/2010 07:25 PM, [email protected] wrote:

From: Andy Adamson <[email protected]>

The LAYOUTCOMMIT call indicates an update to the file meta data is needed, and should be called when clearing the I_DIRTY_SYNC state.

A call to LAYOUTCOMMIT in nfs_write_inode replaces the calls in nfs_wb_all, nfs_commit_inode, and __nfs4_close.

This patch no longer applies on 2.6.35-rc2 in write.c at nfs_commit_inode. But it was easy enough to fix. (Benny tell me if you want a tree with these in)

So I did a rough test "git clone linux" and cthon. It looks very good (few comments below)

Quoted 24 lines
---
 fs/nfs/nfs4state.c |    2 --
 fs/nfs/pnfs.h      |    4 ++++
 fs/nfs/write.c     |   24 ++++++++++++------------
 3 files changed, 16 insertions(+), 14 deletions(-)
fs/nfs/nfs4state.c +0 -2
index d0dbdd4..e1207fa 100644
@@ -589,8 +589,6 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm
 #ifdef CONFIG_NFS_V4_1
 		struct nfs_inode *nfsi = NFS_I(state->inode);
 
-		if (layoutcommit_needed(nfsi))
-			pnfs_layoutcommit_inode(state->inode, wait);
 		if (has_layout(nfsi) && nfsi->layout.roc_iomode) {
 			struct nfs4_pnfs_layout_segment range;
 
fs/nfs/pnfs.h +0 -0
index c89be78..009393e 100644

fs/nfs/pnfs.h static inline int pnfs_use_rpc(struct nfs_server *nfss) · line 264

Quoted 9 lines
fs/nfs/pnfs.h +4 -0
@@ -256,6 +256,10 @@ static inline int pnfs_use_rpc(struct nfs_server *nfss)
 	return 1;
 }
 
+static inline int pnfs_layoutcommit_inode(struct inode *inode, int sync)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 

Just for correctness this patch belongs to [patch 3/4], for it to compile independently.

If they are all squashed then that does not matter. Just as a note. You might as well combine them.

Quoted 40 lines
fs/nfs/pnfs.h +0 -0
 #endif /* FS_NFS_PNFS_H */
fs/nfs/write.c +12 -8
index 0fd33cb..513b308 100644
@@ -1489,13 +1489,6 @@ static int nfs_commit_inode(struct inode *inode, int how)
 			wait_on_bit(&NFS_I(inode)->flags, NFS_INO_COMMIT,
 					nfs_wait_bit_killable,
 					TASK_KILLABLE);
-#ifdef CONFIG_NFS_V4_1
-		if (may_wait && layoutcommit_needed(NFS_I(inode))) {
-			error = pnfs_layoutcommit_inode(inode, 1);
-			if (error < 0)
-				return error;
-		}
-#endif /* CONFIG_NFS_V4_1 */
 	} else
 		nfs_commit_clear_lock(NFS_I(inode));
 out:
@@ -1545,7 +1538,18 @@ static int nfs_commit_unstable_pages(struct inode *inode, struct writeback_contr
 
 int nfs_write_inode(struct inode *inode, struct writeback_control *wbc)
 {
-	return nfs_commit_unstable_pages(inode, wbc);
+	int ret;
+	ret = nfs_commit_unstable_pages(inode, wbc);
+	if (ret >= 0 && layoutcommit_needed(NFS_I(inode))) {
+		int err, sync = wbc->sync_mode;
+
+		if (wbc->nonblocking || wbc->for_background)
+			sync = 0;
+		err = pnfs_layoutcommit_inode(inode, sync);
+		if (err < 0)
+			ret = err;
+	}
+	return ret;
 }
 
 /*

fs/nfs/write.c int nfs_wb_all(struct inode *inode) · line 1571

Quoted 10 lines
fs/nfs/write.c +0 -4
@@ -1562,10 +1566,6 @@ int nfs_wb_all(struct inode *inode)
 	};
 
 	ret = sync_inode(inode, &wbc);
-#ifdef CONFIG_NFS_V4_1
-	if (!ret && layoutcommit_needed(NFS_I(inode)))
-		ret = pnfs_layoutcommit_inode(inode, 1);
-#endif
 	return ret;
 }
 

Grate stuff.
Boaz

#6 lore raw
In reply to [email protected]
Quoted 33 lines · On 06/03/2010 07:25 PM, [email protected] wrote:

Responded to comments.

This is against the pnfs-submit branch of the 2.6.34 tree. They will need to be applied against the 2.6.35-rc1 tree which I can do after comments.

A cleanup, and call the async error handler. 0001-SQUASHME-pnfs-submit-cleanup-layoutcommit-call.patch 0002-SQUASHME-pnfs-submit-handle-async-layoutcommit-erro.patch

These next two patches moves the pnfs_layoutcommit_inode call to nfs_write_inode, and it is the only call other than in layoutreturn. (removed calls in __nfs4_close, nfs_commit_inode, nfs_wb_sync).

0003-SQUASHME-pnfs-remove-ifdef-around-layoutcommit_neede.patch 0004-SQUASHME-pnfs-submit-move-layoutcommit-to-nfs_write.patch

pnfs_layoutcommit_inode is called after nfs_commit_unstable_pages() so that if LAYOUTCOMMIT fails, the unstable pages have been processed..

The error handlers (sync and async) call nfs4_map_errors, so unhandled errors (such as NFS4ERR_BADLAYOUT) get returned to nfs_write_ioode as -EIO.

We will add code to the error handlers for errors such as NFS4ERR_BADLAYOUT that require us to stop using and free the layout, and redo the I/O through the MDS.

Testing:
With CONFIG_NFS_V4_1 set
NFSv4.1/pnfs passed Connectathon against write enabled GFS2/pNFS. Note: there were exactly the same number of LAYOUTCOMMITS sent as were sent with pnfs_layoutcommit_inode being called from __nfs4_close (never happened), nfs_commit_inode and nfs_wb_sync.

Quoted 8 lines

V4.0 mount passes Connectathon tests

With CONFIG_NFS_V4_1 not set

V4.0 mount passes Connectathon tests

-->Andy

Hi Handy, I'll review this and Test on Sunday. Happy weekend.

Thanks
Boaz