pnfs-submit: cleanup layoutcommit verison 2
Subsystems (2)
cover pnfs-submit: cleanup layoutcommit verison 2
Responded to comments.
This is against the pnfs-submit branch of the 2.6.34 tree. They will need to be applied against the 2.6.35-rc1 tree which I can do after comments.
A cleanup, and call the async error handler. 0001-SQUASHME-pnfs-submit-cleanup-layoutcommit-call.patch 0002-SQUASHME-pnfs-submit-handle-async-layoutcommit-erro.patch
These next two patches moves the pnfs_layoutcommit_inode call to nfs_write_inode, and it is the only call other than in layoutreturn. (removed calls in __nfs4_close, nfs_commit_inode, nfs_wb_sync).
0003-SQUASHME-pnfs-remove-ifdef-around-layoutcommit_neede.patch 0004-SQUASHME-pnfs-submit-move-layoutcommit-to-nfs_write.patch
pnfs_layoutcommit_inode is called after nfs_commit_unstable_pages() so that if LAYOUTCOMMIT fails, the unstable pages have been processed..
The error handlers (sync and async) call nfs4_map_errors, so unhandled errors (such as NFS4ERR_BADLAYOUT) get returned to nfs_write_ioode as -EIO.
We will add code to the error handlers for errors such as NFS4ERR_BADLAYOUT that require us to stop using and free the layout, and redo the I/O through the MDS.
Testing:
With CONFIG_NFS_V4_1 set
NFSv4.1/pnfs passed Connectathon against write enabled GFS2/pNFS. Note: there were exactly the same number of LAYOUTCOMMITS sent as were sent with pnfs_layoutcommit_inode being called from __nfs4_close (never happened), nfs_commit_inode and nfs_wb_sync.
V4.0 mount passes Connectathon tests
With CONFIG_NFS_V4_1 not set
V4.0 mount passes Connectathon tests
-->Andy
PATCH 1/4 SQUASHME pnfs-submit: cleanup layoutcommit call
From: Andy Adamson <[email protected]>
pnfs_layoutcommit_inode prints its status upon exit.
--- fs/nfs/pnfs.c | 12 +++--------- 1 files changed, 3 insertions(+), 9 deletions(-)
fs/nfs/pnfs.c +3 -9
@@ -766,15 +766,9 @@ _pnfs_return_layout(struct inode *ino, struct nfs4_pnfs_layout_segment *range, !pnfs_return_layout_barrier(nfsi, &arg)); } -if (layoutcommit_needed (nfsi)) { - status =pnfs_layoutcommit_inode (ino, wait); -if (status) { -dprintk ("%s: layoutcommit failed, status=%d. " -"Returning layout anyway\n" , - __func__, status); - status =0 ; - } - } +if (layoutcommit_needed (nfsi)) +/* Return layout even if layoutcommit fails */ +pnfs_layoutcommit_inode (ino, wait); } send_return: status =return_layout (ino, &arg, stateid, type, lo, wait);
Signature · 1 line
1.6.2.5
PATCH 2/4 SQUASHME pnfs-submit: handle async layoutcommit errors
From: Andy Adamson <[email protected]>
nfs4_async_handle_error handles session level and grace,delay errors.
NOTE: Layout specific errors (NFS4ERR_BADIOMODE, NFS4ERR_BAD_LAYOUT, NFS4ERR_UNKNOWN_LAYOUTTYPE) need to be handled.
--- fs/nfs/nfs4proc.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfs/nfs4proc.c +3 -0
@@ -5628,6 +5628,9 @@ pnfs_layoutcommit_rpc_done(struct rpc_task *task, void *calldata)pnfs_layoutcommit_done (data);nfs4_sequence_done (server, &data->res.seq_res, task->tk_status); + +if (nfs4_async_handle_error (task, server,NULL ) == -EAGAIN ) +nfs_restart_rpc (task, server->nfs_client); }static void pnfs_layoutcommit_release (void *lcdata)
Signature · 1 line
1.6.2.5
PATCH 3/4 SQUASHME pnfs remove ifdef around layoutcommit_needed
From: Andy Adamson <[email protected]>
--- fs/nfs/inode.c | 2 -- include/linux/nfs4_pnfs.h | 8 ++++++++ 2 files changed, 8 insertions(+), 2 deletions(-)
fs/nfs/inode.c +0 -2
@@ -1106,7 +1106,6 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) !test_bit(NFS_INO_MOUNTPOINT, &nfsi->flags)) server->fsid = fattr->fsid; -#ifdef CONFIG_NFS_V4_1 /* * file needs layout commit, server attributes may be stale */
@@ -1115,7 +1114,6 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) __func__, inode->i_sb->s_id, inode->i_ino);return 0 ; } -#endif /* CONFIG_NFS_V4_1 */ /* * Update the read time so we don't revalidate too often. */
include/linux/nfs4_pnfs.h +8 -0
@@ -86,6 +86,14 @@ layoutcommit_needed(struct nfs_inode *nfsi)return nfsi->lo_cred !=NULL ; } +#else /* CONFIG_NFS_V4_1 */ + +static inline bool +layoutcommit_needed (struct nfs_inode *nfsi) +{ +return 0 ; +} +#endif /* CONFIG_NFS_V4_1 */ struct pnfs_layout_segment {
Signature · 1 line
1.6.2.5
PATCH 4/4 SQUASHME pnfs-submit: move layoutcommit to nfs_write_inode
From: Andy Adamson <[email protected]>
The LAYOUTCOMMIT call indicates an update to the file meta data is needed, and should be called when clearing the I_DIRTY_SYNC state.
A call to LAYOUTCOMMIT in nfs_write_inode replaces the calls in nfs_wb_all, nfs_commit_inode, and __nfs4_close.
--- fs/nfs/nfs4state.c | 2 -- fs/nfs/pnfs.h | 4 ++++ fs/nfs/write.c | 24 ++++++++++++------------ 3 files changed, 16 insertions(+), 14 deletions(-)
fs/nfs/nfs4state.c +0 -2
@@ -589,8 +589,6 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm#ifdef CONFIG_NFS_V4_1 struct nfs_inode *nfsi =NFS_I (state->inode); -if (layoutcommit_needed (nfsi)) -pnfs_layoutcommit_inode (state->inode, wait);if (has_layout (nfsi) && nfsi->layout.roc_iomode) {struct nfs4_pnfs_layout_segment range;
fs/nfs/pnfs.h +4 -0
@@ -256,6 +256,10 @@ static inline int pnfs_use_rpc(struct nfs_server *nfss)return 1 ; } +static inline int pnfs_layoutcommit_inode (struct inode *inode,int sync) +{ +return 0 ; +}#endif /* CONFIG_NFS_V4_1 */ #endif /* FS_NFS_PNFS_H */
fs/nfs/write.c +12 -12
@@ -1489,13 +1489,6 @@ static int nfs_commit_inode(struct inode *inode, int how)wait_on_bit (&NFS_I (inode)->flags,NFS_INO_COMMIT , nfs_wait_bit_killable,TASK_KILLABLE ); -#ifdef CONFIG_NFS_V4_1 -if (may_wait &&layoutcommit_needed (NFS_I (inode))) { - error =pnfs_layoutcommit_inode (inode,1 ); -if (error <0 ) -return error; - } -#endif /* CONFIG_NFS_V4_1 */ }else nfs_commit_clear_lock (NFS_I (inode)); out:
@@ -1545,7 +1538,18 @@ static int nfs_commit_unstable_pages(struct inode *inode, struct writeback_contrint nfs_write_inode (struct inode *inode,struct writeback_control *wbc) { -return nfs_commit_unstable_pages (inode, wbc); +int ret; + ret =nfs_commit_unstable_pages (inode, wbc); +if (ret >=0 &&layoutcommit_needed (NFS_I (inode))) { +int err, sync = wbc->sync_mode; + +if (wbc->nonblocking || wbc->for_background) + sync =0 ; + err =pnfs_layoutcommit_inode (inode, sync); +if (err <0 ) + ret = err; + } +return ret; }/*
@@ -1562,10 +1566,6 @@ int nfs_wb_all(struct inode *inode) }; ret =sync_inode (inode, &wbc); -#ifdef CONFIG_NFS_V4_1 -if (!ret &&layoutcommit_needed (NFS_I (inode))) - ret =pnfs_layoutcommit_inode (inode,1 ); -#endif return ret; }
Signature · 1 line
1.6.2.5
Re: [PATCH 4/4] SQUASHME pnfs-submit: move layoutcommit to nfs_write_inode
Quoted 9 lines · On 06/03/2010 07:25 PM, [email protected] wrote:
From: Andy Adamson <[email protected]>
The LAYOUTCOMMIT call indicates an update to the file meta data is needed, and should be called when clearing the
I_DIRTY_SYNCstate.A call to LAYOUTCOMMIT in
nfs_write_inodereplaces the calls innfs_wb_all,nfs_commit_inode, and__nfs4_close.Signed-off-by Andy Adamson <[email protected]>
This patch no longer applies on 2.6.35-rc2 in write.c at nfs_commit_inode. But it was easy enough to fix. (Benny tell me if you want a tree with these in)
So I did a rough test "git clone linux" and cthon. It looks very good (few comments below)
Quoted 24 lines
--- fs/nfs/nfs4state.c | 2 -- fs/nfs/pnfs.h | 4 ++++ fs/nfs/write.c | 24 ++++++++++++------------ 3 files changed, 16 insertions(+), 14 deletions(-)fs/nfs/nfs4state.c +0
-2@@ -589,8 +589,6 @@ static void __nfs4_close(struct path *path, struct nfs4_state *state, fmode_t fm#ifdef CONFIG_NFS_V4_1 struct nfs_inode *nfsi =NFS_I (state->inode); -if (layoutcommit_needed (nfsi)) -pnfs_layoutcommit_inode (state->inode, wait);if (has_layout (nfsi) && nfsi->layout.roc_iomode) {struct nfs4_pnfs_layout_segment range;fs/nfs/pnfs.h +0
-0
fs/nfs/pnfs.h static inline int pnfs_use_rpc(struct nfs_server *nfss) · line 264
Quoted 9 lines
fs/nfs/pnfs.h +4
-0@@ -256,6 +256,10 @@ static inline int pnfs_use_rpc(struct nfs_server *nfss) return 1; } +static inline int pnfs_layoutcommit_inode (struct inode *inode,int sync) +{ +return 0 ; +}#endif /* CONFIG_NFS_V4_1 */
Just for correctness this patch belongs to [patch 3/4], for it to compile independently.
If they are all squashed then that does not matter. Just as a note. You might as well combine them.
Quoted 40 lines
fs/nfs/pnfs.h +0
-0#endif /* FS_NFS_PNFS_H */ fs/nfs/write.c +12
-8@@ -1489,13 +1489,6 @@ static int nfs_commit_inode(struct inode *inode, int how)wait_on_bit (&NFS_I (inode)->flags,NFS_INO_COMMIT , nfs_wait_bit_killable,TASK_KILLABLE ); -#ifdef CONFIG_NFS_V4_1 -if (may_wait &&layoutcommit_needed (NFS_I (inode))) { - error =pnfs_layoutcommit_inode (inode,1 ); -if (error <0 ) -return error; - } -#endif /* CONFIG_NFS_V4_1 */ }else nfs_commit_clear_lock (NFS_I (inode)); out:@@ -1545,7 +1538,18 @@ static int nfs_commit_unstable_pages(struct inode *inode, struct writeback_contrint nfs_write_inode (struct inode *inode,struct writeback_control *wbc) { -return nfs_commit_unstable_pages (inode, wbc); +int ret; + ret =nfs_commit_unstable_pages (inode, wbc); +if (ret >=0 &&layoutcommit_needed (NFS_I (inode))) { +int err, sync = wbc->sync_mode; + +if (wbc->nonblocking || wbc->for_background) + sync =0 ; + err =pnfs_layoutcommit_inode (inode, sync); +if (err <0 ) + ret = err; + } +return ret; }/*
fs/nfs/write.c int nfs_wb_all(struct inode *inode) · line 1571
Quoted 10 lines
fs/nfs/write.c +0
-4@@ -1562,10 +1566,6 @@ int nfs_wb_all(struct inode *inode) }; ret =sync_inode (inode, &wbc); -#ifdef CONFIG_NFS_V4_1 -if (!ret &&layoutcommit_needed (NFS_I (inode))) - ret =pnfs_layoutcommit_inode (inode,1 ); -#endif return ret; }
Grate stuff.
Boaz
Quoted 33 lines · On 06/03/2010 07:25 PM, [email protected] wrote:
Responded to comments.
This is against the pnfs-submit branch of the 2.6.34 tree. They will need to be applied against the 2.6.35-rc1 tree which I can do after comments.
A cleanup, and call the async error handler. 0001-SQUASHME-pnfs-submit-cleanup-layoutcommit-call.patch 0002-SQUASHME-pnfs-submit-handle-async-layoutcommit-erro.patch
These next two patches moves the
pnfs_layoutcommit_inodecall tonfs_write_inode, and it is the only call other than in layoutreturn. (removed calls in__nfs4_close,nfs_commit_inode,nfs_wb_sync).0003-SQUASHME-pnfs-remove-ifdef-around-layoutcommit_neede.patch 0004-SQUASHME-pnfs-submit-move-layoutcommit-to-nfs_write.patch
pnfs_layoutcommit_inodeis called afternfs_commit_unstable_pages()so that if LAYOUTCOMMIT fails, the unstable pages have been processed..The error handlers (sync and async) call
nfs4_map_errors, so unhandled errors (such asNFS4ERR_BADLAYOUT) get returned tonfs_write_ioodeas-EIO.We will add code to the error handlers for errors such as
NFS4ERR_BADLAYOUTthat require us to stop using and free the layout, and redo the I/O through the MDS.Testing:
WithCONFIG_NFS_V4_1set
NFSv4.1/pnfs passed Connectathon against write enabled GFS2/pNFS. Note: there were exactly the same number of LAYOUTCOMMITS sent as were sent withpnfs_layoutcommit_inodebeing called from__nfs4_close(never happened),nfs_commit_inodeandnfs_wb_sync.
Quoted 8 lines
V4.0 mount passes Connectathon tests
With
CONFIG_NFS_V4_1not setV4.0 mount passes Connectathon tests
-->Andy
Hi Handy, I'll review this and Test on Sunday. Happy weekend.
Thanks
Boaz