xfs: candidate fixes for 3.12-rc4
cover xfs: candidate fixes for 3.12-rc4
Hi folks,
The first two patches are regressions and hence -rc4 candidates - the first patch fixes a lockdep false positive as a result of adding the third quota inode, and the second fixes a on-disk format interpretation problem in the v4 superblock dtype-in-dirent feature.
The third patch fixes the AGF vs AGI lock inversion problem I reported a short while ago, and the last reduces the overhead of atomic version change accounting. These two aren't critical fixes, but would be nice to have out there for 3.12.
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/4 xfs: lockdep needs to know about 3 dquot-deep nesting
From: Dave Chinner <[email protected]>
Michael Semon reported that xfs/299 generated this lockdep warning:
=============================================
[ INFO: possible recursive locking detected ]
3.12.0-rc2+ #2 Not tainted
---------------------------------------------
touch/21072 is trying to acquire lock:
(&xfs_dquot_other_class){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64
but task is already holding lock:
(&xfs_dquot_other_class){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
---- lock(&xfs_dquot_other_class);
lock(&xfs_dquot_other_class);
*** DEADLOCK ***
May be due to missing lock nesting notation
7 locks held by touch/21072:
#0: (sb_writers#10){++++.+}, at: [<c11185b6>] mnt_want_write+0x1e/0x3e
#1: (&type->i_mutex_dir_key#4){+.+.+.}, at: [<c11078ee>] do_last+0x245/0xe40
#2: (sb_internal#2){++++.+}, at: [<c122c9e0>] xfs_trans_alloc+0x1f/0x35
#3: (&(&ip->i_lock)->mr_lock/1){+.+...}, at: [<c126cd1b>] xfs_ilock+0x100/0x1f1
#4: (&(&ip->i_lock)->mr_lock){++++-.}, at: [<c126cf52>] xfs_ilock_nowait+0x105/0x22f
#5: (&dqp->q_qlock){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64
#6: (&xfs_dquot_other_class){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64The lockdep annotation for dquot lock nesting only understands locking for user and "other" dquots, not user, group and quota dquots. Fix the annotations to match the locking heirarchy we now have.
--- fs/xfs/xfs_dquot.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
fs/xfs/xfs_dquot.c +16 -3
@@ -64,7 +64,8 @@ int xfs_dqerror_mod = 33;struct kmem_zone *xfs_qm_dqtrxzone;static struct kmem_zone *xfs_qm_dqzone; -static struct lock_class_key xfs_dquot_other_class; +static struct lock_class_key xfs_dquot_group_class; +static struct lock_class_key xfs_dquot_project_class;/* * This is called to free all the memory associated with a dquot
@@ -703,8 +704,20 @@ xfs_qm_dqread(* Make sure group quotas have a different lock class than user * quotas. */ -if (!(type &XFS_DQ_USER )) -lockdep_set_class (&dqp->q_qlock, &xfs_dquot_other_class); +switch (type) { +case XFS_DQ_USER : +/* uses the default lock class */ +break ; +case XFS_DQ_GROUP : +lockdep_set_class (&dqp->q_qlock, &xfs_dquot_group_class); +break ; +case XFS_DQ_PROJ : +lockdep_set_class (&dqp->q_qlock, &xfs_dquot_project_class); +break ; +default : +ASSERT (0 ); +break ; + }XFS_STATS_INC (xs_qm_dquot);
Signature · 6 lines
1.8.3.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/4] xfs: lockdep needs to know about 3 dquot-deep nesting
Quoted 35 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Michael Semon reported that xfs/299 generated this lockdep warning:
=============================================
[ INFO: possible recursive locking detected ]
3.12.0-rc2+ #2 Not tainted
---------------------------------------------
touch/21072 is trying to acquire lock:
(&xfs_dquot_other_class){+.+...}, at: [<c12902fb>]xfs_trans_dqlockedjoin+0x57/0x64but task is already holding lock:
(&xfs_dquot_other_class){+.+...}, at: [<c12902fb>]xfs_trans_dqlockedjoin+0x57/0x64other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----
lock(&xfs_dquot_other_class);
lock(&xfs_dquot_other_class);*** DEADLOCK ***
May be due to missing lock nesting notation
7 locks held by touch/21072:
#0: (sb_writers#10){++++.+}, at: [<c11185b6>] mnt_want_write+0x1e/0x3e #1: (&type->i_mutex_dir_key#4){+.+.+.}, at: [<c11078ee>] do_last+0x245/0xe40 #2: (sb_internal#2){++++.+}, at: [<c122c9e0>] xfs_trans_alloc+0x1f/0x35 #3: (&(&ip->i_lock)->mr_lock/1){+.+...}, at: [<c126cd1b>] xfs_ilock+0x100/0x1f1 #4: (&(&ip->i_lock)->mr_lock){++++-.}, at: [<c126cf52>] xfs_ilock_nowait+0x105/0x22f #5: (&dqp->q_qlock){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64 #6: (&xfs_dquot_other_class){+.+...}, at: [<c12902fb>] xfs_trans_dqlockedjoin+0x57/0x64
Quoted 7 lines
The lockdep annotation for dquot lock nesting only understands locking for user and "other" dquots, not user, group and quota dquots. Fix the annotations to match the locking heirarchy we now have.
Reported-by Michael L. Semon <[email protected]>Signed-off-by Dave Chinner <[email protected]>
Looks good.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/4 xfs: dirent dtype presence is dependent on directory magic numbers
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The determination of whether a directory entry contains a dtype field originally was dependent on the filesystem having CRCs enabled. This meant that the format for dtype beign enabled could be determined by checking the directory block magic number rather than doing a feature bit check. This was useful in that it meant that we didn't need to pass a struct xfs_mount around to functions that were already supplied with a directory block header.
Unfortunately, the introduction of dtype fields into the v4 structure via a feature bit meant this "use the directory block magic number" method of discriminating the dirent entry sizes is broken. Hence we need to convert the places that use magic number checks to use feature bit checks so that they work correctly and not by chance.
The current code works on v4 filesystems only because the dirent size roundup covers the extra byte needed by the dtype field in the places where this problem occurs.
--- fs/xfs/xfs_dir2_block.c | 6 +++--- fs/xfs/xfs_dir2_format.h | 51 +++++++++++++++++++---------------------------- fs/xfs/xfs_dir2_readdir.c | 4 ++-- fs/xfs/xfs_dir2_sf.c | 6 +++--- 4 files changed, 28 insertions(+), 39 deletions(-)
fs/xfs/xfs_dir2_block.c +3 -3
@@ -1158,7 +1158,7 @@ xfs_dir2_sf_to_block(/* * Create entry for . */ - dep =xfs_dir3_data_dot_entry_p (hdr); + dep =xfs_dir3_data_dot_entry_p (mp, hdr); dep->inumber =cpu_to_be64 (dp->i_ino); dep->namelen =1 ; dep->name[0 ] ='.' ;
@@ -1172,7 +1172,7 @@ xfs_dir2_sf_to_block(/* * Create entry for .. */ - dep =xfs_dir3_data_dotdot_entry_p (hdr); + dep =xfs_dir3_data_dotdot_entry_p (mp, hdr); dep->inumber =cpu_to_be64 (xfs_dir2_sf_get_parent_ino (sfp)); dep->namelen =2 ; dep->name[0 ] = dep->name[1 ] ='.' ;
@@ -1183,7 +1183,7 @@ xfs_dir2_sf_to_block( blp[1 ].hashval =cpu_to_be32 (xfs_dir_hash_dotdot); blp[1 ].address =cpu_to_be32 (xfs_dir2_byte_to_dataptr (mp, (char *)dep - (char *)hdr)); - offset =xfs_dir3_data_first_offset (hdr); + offset =xfs_dir3_data_first_offset (mp);/* * Loop over existing entries, stuff them in. */
fs/xfs/xfs_dir2_format.h +20 -31
89 lines not on this page · raw message
fs/xfs/xfs_dir2_readdir.c +2 -2
@@ -119,9 +119,9 @@ xfs_dir2_sf_getdents(* mp->m_dirdatablk. */ dot_offset =xfs_dir2_db_off_to_dataptr (mp, mp->m_dirdatablk, -XFS_DIR3_DATA_DOT_OFFSET (mp)); +xfs_dir3_data_dot_offset (mp)); dotdot_offset =xfs_dir2_db_off_to_dataptr (mp, mp->m_dirdatablk, -XFS_DIR3_DATA_DOTDOT_OFFSET (mp)); +xfs_dir3_data_dotdot_offset (mp));/* * Put . entry unless we're starting past it.
fs/xfs/xfs_dir2_sf.c +3 -3
@@ -557,7 +557,7 @@ xfs_dir2_sf_addname_hard(* to insert the new entry. * If it's going to end up at the end then oldsfep will point there. */ - for (offset = XFS_DIR3_DATA_FIRST_OFFSET(mp), + for (offset = xfs_dir3_data_first_offset(mp), oldsfep = xfs_dir2_sf_firstentry(oldsfp), add_datasize = xfs_dir3_data_entsize(mp, args->namelen), eof = (char *)oldsfep == &buf[old_isize];
@@ -640,7 +640,7 @@ xfs_dir2_sf_addname_pick( sfp = (xfs_dir2_sf_hdr_t *)dp->i_df.if_u1.if_data; size =xfs_dir3_data_entsize (mp, args->namelen); - offset =XFS_DIR3_DATA_FIRST_OFFSET (mp); + offset =xfs_dir3_data_first_offset (mp); sfep =xfs_dir2_sf_firstentry (sfp); holefit =0 ;/*
@@ -713,7 +713,7 @@ xfs_dir2_sf_check( mp = dp->i_mount; sfp = (xfs_dir2_sf_hdr_t *)dp->i_df.if_u1.if_data; - offset =XFS_DIR3_DATA_FIRST_OFFSET (mp); + offset =xfs_dir3_data_first_offset (mp); ino =xfs_dir2_sf_get_parent_ino (sfp); i8count = ino >XFS_DIR2_MAX_SHORT_INUM ;
Signature · 6 lines
1.8.3.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfs: dirent dtype presence is dependent on directory magic numbers
Quoted 17 lines · Dave Chinner
From: Dave Chinner <[email protected]>
The determination of whether a directory entry contains a dtype field originally was dependent on the filesystem having CRCs enabled. This meant that the format for dtype beign enabled could be determined by checking the directory block magic number rather than doing a feature bit check. This was useful in that it meant that we didn't need to pass a struct
xfs_mountaround to functions that were already supplied with a directory block header.Unfortunately, the introduction of dtype fields into the v4 structure via a feature bit meant this "use the directory block magic number" method of discriminating the dirent entry sizes is broken. Hence we need to convert the places that use magic number checks to use feature bit checks so that they work correctly and not by chance.
Quoted 5 lines
The current code works on v4 filesystems only because the dirent size roundup covers the extra byte needed by the dtype field in the places where this problem occurs.
Signed-off-by Dave Chinner <[email protected]>
Looks fine to me.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfs: dirent dtype presence is dependent on directory magic numbers
In reply to Dave ChinnerThis has been committed.
Thanks
--Rich
commit 41315687d9db9b50876401e7b0ee20dd77cfc712
Author: Dave Chinner <[email protected]> Date: Mon Sep 30 03:15:19 2013 +0000
xfs: dirent dtype presence is dependent on directory magic numbers
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfs: dirent dtype presence is dependent on directory magic numbers
Quoted 10 lines · Rich Johnston
This has been committed.
Thanks
--Richcommit
41315687d9db9b50876401e7b0ee20dd77cfc712Author: Dave Chinner <[email protected]> Date: Mon Sep 30 03:15:19 2013 +0000xfs: dirent dtype presence is dependent on directory magic numbers
Committed where, exactly?
This is a kernel patch that has already been committed and been pushed upstream to linus as commit 6d31349....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfs: dirent dtype presence is dependent on directory magic numbers
sorry oversight on reply the patch had the similar name
Re: [PATCH 07/32] xfs: dirent dtype presence is dependent on directory magic numbers
Quoted 12 lines · D \ave Chinner
Quoted 10 lines · Rich Johnston
This has been committed.
Thanks
--Richcommit
41315687d9db9b50876401e7b0ee20dd77cfc712Author: Dave Chinner <[email protected]> Date: Mon Sep 30 03:15:19 2013 +0000xfs: dirent dtype presence is dependent on directory magic numbers
Quoted 9 lines
Committed where, exactly?
This is a kernel patch that has already been committed and been pushed upstream to linus as commit 6d31349....
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/4 xfs: xfs_remove deadlocks due to inverted AGF vs AGI lock ordering
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Removing an inode from the namespace involves removing the directory entry and dropping the link count on the inode. Removing the directory entry can result in locking an AGF (directory blocks were freed) and removing a link count can result in placing the inode on an unlinked list which results in locking an AGI.
The big problem here is that we have an ordering constraint on AGF and AGI locking - inode allocation locks the AGI, then can allocate a new extent for new inodes, locking the AGF after the AGI. Similarly, freeing the inode removes the inode from the unlinked list, requiring that we lock the AGI first, and then freeing the inode can result in an inode chunk being freed and hence freeing disk space requiring that we lock an AGF.
Hence the ordering that is imposed by other parts of the code is AGI before AGF. This means we cannot remove the directory entry before we drop the inode reference count and put it on the unlinked list as this results in a lock order of AGF then AGI, and this can deadlock against inode allocation and freeing. Therefore we must drop the link counts before we remove the directory entry.
This is still safe from a transactional point of view - it is not until we get to xfs_bmap_finish() that we have the possibility of multiple transactions in this operation. Hence as long as we remove the directory entry and drop the link count in the first transaction of the remove operation, there are no transactional constraints on the ordering here.
Change the ordering of the operations in the xfs_remove() function to align the ordering of AGI and AGF locking to match that of the rest of the code.
--- fs/xfs/xfs_inode.c | 72 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 44 insertions(+), 28 deletions(-)
fs/xfs/xfs_inode.c +44 -28
@@ -2370,6 +2370,33 @@ xfs_iunpin_wait(__xfs_iunpin_wait (ip); } +/* +* Removing an inode from the namespace involves removing the directory entry +* and dropping the link count on the inode. Removing the directory entry can +* result in locking an AGF (directory blocks were freed) and removing a link +* count can result in placing the inode on an unlinked list which results in +* locking an AGI. +* +* The big problem here is that we have an ordering constraint on AGF and AGI +* locking - inode allocation locks the AGI, then can allocate a new extent for +* new inodes, locking the AGF after the AGI. Similarly, freeing the inode +* removes the inode from the unlinked list, requiring that we lock the AGI +* first, and then freeing the inode can result in an inode chunk being freed +* and hence freeing disk space requiring that we lock an AGF. +* +* Hence the ordering that is imposed by other parts of the code is AGI before +* AGF. This means we cannot remove the directory entry before we drop the inode +* reference count and put it on the unlinked list as this results in a lock +* order of AGF then AGI, and this can deadlock against inode allocation and +* freeing. Therefore we must drop the link counts before we remove the +* directory entry. +* +* This is still safe from a transactional point of view - it is not until we +* get to xfs_bmap_finish() that we have the possibility of multiple +* transactions in this operation. Hence as long as we remove the directory +* entry and drop the link count in the first transaction of the remove +* operation, there are no transactional constraints on the ordering here. +*/ int xfs_remove( xfs_inode_t *dp,
@@ -2439,6 +2466,7 @@ xfs_remove(/* * If we're removing a directory perform some additional validation. */ + cancel_flags |=XFS_TRANS_ABORT ;if (is_dir) {ASSERT (ip->i_d.di_nlink >=2 );if (ip->i_d.di_nlink !=2 ) {
@@ -2449,31 +2477,16 @@ xfs_remove( error =XFS_ERROR (ENOTEMPTY ); goto out_trans_cancel; } - } -xfs_bmap_init (&free_list, &first_block); - error =xfs_dir_removename (tp, dp, name, ip->i_ino, - &first_block, &free_list, resblks); -if (error) { -ASSERT (error !=ENOENT ); - goto out_bmap_cancel; - } -xfs_trans_ichgtime (tp, dp,XFS_ICHGTIME_MOD |XFS_ICHGTIME_CHG ); - -if (is_dir) { -/* -* Drop the link from ip's "..". -*/ +/* Drop the link from ip's "..". */ error =xfs_droplink (tp, dp);if (error) - goto out_bmap_cancel; + goto out_trans_cancel; -/* -* Drop the "." link from ip to self. -*/ +/* Drop the "." link from ip to self. */ error =xfs_droplink (tp, ip);if (error) - goto out_bmap_cancel; + goto out_trans_cancel; }else {/* * When removing a non-directory we need to log the parent
@@ -2482,20 +2495,24 @@ xfs_remove(*/ xfs_trans_log_inode (tp, dp,XFS_ILOG_CORE ); } +xfs_trans_ichgtime (tp, dp,XFS_ICHGTIME_MOD |XFS_ICHGTIME_CHG ); -/* -* Drop the link from dp to ip. -*/ +/* Drop the link from dp to ip. */ error =xfs_droplink (tp, ip);if (error) - goto out_bmap_cancel; + goto out_trans_cancel; -/* -* Determine if this is the last link while -* we are in the transaction. -*/ +/* Determine if this is the last link while the inode is locked */ link_zero = (ip->i_d.di_nlink ==0 ); +xfs_bmap_init (&free_list, &first_block); + error =xfs_dir_removename (tp, dp, name, ip->i_ino, + &first_block, &free_list, resblks); +if (error) { +ASSERT (error !=ENOENT ); + goto out_bmap_cancel; + } +/* * If this is a synchronous mount, make sure that the * remove transaction goes to disk before returning to
@@ -2525,7 +2542,6 @@ xfs_remove( out_bmap_cancel:xfs_bmap_cancel (&free_list); - cancel_flags |=XFS_TRANS_ABORT ; out_trans_cancel:xfs_trans_cancel (tp, cancel_flags); std_return:
Signature · 6 lines
1.8.3.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/4] xfs: xfs_remove deadlocks due to inverted AGF vs AGI lock ordering
Hi Dave,
Quoted 30 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Removing an inode from the namespace involves removing the directory entry and dropping the link count on the inode. Removing the directory entry can result in locking an AGF (directory blocks were freed) and removing a link count can result in placing the inode on an unlinked list which results in locking an AGI.
The big problem here is that we have an ordering constraint on AGF and AGI locking - inode allocation locks the AGI, then can allocate a new extent for new inodes, locking the AGF after the AGI. Similarly, freeing the inode removes the inode from the unlinked list, requiring that we lock the AGI first, and then freeing the inode can result in an inode chunk being freed and hence freeing disk space requiring that we lock an AGF.
Hence the ordering that is imposed by other parts of the code is AGI before AGF. This means we cannot remove the directory entry before we drop the inode reference count and put it on the unlinked list as this results in a lock order of AGF then AGI, and this can deadlock against inode allocation and freeing. Therefore we must drop the link counts before we remove the directory entry.
This is still safe from a transactional point of view - it is not until we get to
xfs_bmap_finish()that we have the possibility of multiple transactions in this operation. Hence as long as we remove the directory entry and drop the link count in the first transaction of the remove operation, there are no transactional constraints on the ordering here.
Quoted 5 lines
Change the ordering of the operations in the
xfs_remove()function to align the ordering of AGI and AGF locking to match that of the rest of the code.Signed-off-by Dave Chinner <[email protected]>
Hmmm.. I'm not quite comfortable with this one yet... It'll probably be better after some more review. Did you happen to have a test case to go with this?
Thanks,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/4] xfs: xfs_remove deadlocks due to inverted AGF vs AGI lock ordering
Quoted 37 lines · Ben Myers
Hi Dave,
Quoted 34 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Removing an inode from the namespace involves removing the directory entry and dropping the link count on the inode. Removing the directory entry can result in locking an AGF (directory blocks were freed) and removing a link count can result in placing the inode on an unlinked list which results in locking an AGI.
The big problem here is that we have an ordering constraint on AGF and AGI locking - inode allocation locks the AGI, then can allocate a new extent for new inodes, locking the AGF after the AGI. Similarly, freeing the inode removes the inode from the unlinked list, requiring that we lock the AGI first, and then freeing the inode can result in an inode chunk being freed and hence freeing disk space requiring that we lock an AGF.
Hence the ordering that is imposed by other parts of the code is AGI before AGF. This means we cannot remove the directory entry before we drop the inode reference count and put it on the unlinked list as this results in a lock order of AGF then AGI, and this can deadlock against inode allocation and freeing. Therefore we must drop the link counts before we remove the directory entry.
This is still safe from a transactional point of view - it is not until we get to
xfs_bmap_finish()that we have the possibility of multiple transactions in this operation. Hence as long as we remove the directory entry and drop the link count in the first transaction of the remove operation, there are no transactional constraints on the ordering here.Change the ordering of the operations in the
xfs_remove()function to align the ordering of AGI and AGF locking to match that of the rest of the code.
Quoted 5 lines
Quoted 1 line
Signed-off-by Dave Chinner <[email protected]>Hmmm.. I'm not quite comfortable with this one yet... It'll probably be better after some more review. Did you happen to have a test case to go with this?
Two test cases.
The first that I saw involved a 100TB filesystem and 16-way creates running concurrently with 16-way unlinks, as reported here:
http://oss.sgi.com/archives/xfs/2013-09/msg00359.html
That's not reliable, though - in general it takes hours of running a couple of hundred threads doing sequential file create and unlinks on a 32p VM. i.e. adding a half-hour long 16-way fsmark workload every couple of minutes to the load that is aready running on the filesystem. It runs at about 350,000 file creates/s and 150,000 unlinks/s, so it takes a load that is not really feasible for the majority of QA environments. I haven't had this workload deadlock since I wrote this patch.
The other system I have that reliably trips over this is at the other end of the scale, and it's generic/269 w/ MKFS_OPTIONS="-b size=1024 -m crc=1" that deadlocks on a single CPU VM with 1GB RAM. This patch does make generic/269 complete on that machine - it just changes the deadlock that is being hit.
i.e. it appears that we have introduced an ENOSPC AGF vs AGF lock ordering deadlock at some point, but I haven't got a handle on that yet to be able to even speculate on what the cause might be. I'm in the process of refining tracepoints to narrow down the exactly order of operations that is causing problems...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/4 xfs: open code inc_inode_iversion when logging an inode
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Michael L Semon reported that generic/069 runtime increased on v5 superblocks by 100% compared to v4 superblocks. his perf-based analysis pointed directly at the timestamp updates being done by the write path in this workload. The append writers are doing 4-byte writes, so there are lots of timestamp updates occurring.
The thing is, they aren't being triggered by timestamp changes - they are being triggered by the inode change counter needing to be updated. That is, every write(2) system call needs to bump the inode version count, and it does that through the timestamp update mechanism. Hence for v5 filesystems, test generic/069 is running 3 orders of magnitude more timestmap update transactions on v5 filesystems due to the fact it does a huge number of *4 byte* write(2) calls.
This isn't a real world scenario we really need to address - anyone doing such sequential IO should be using fwrite(3), not write(2). i.e. fwrite(3) buffers the writes in userspace to minimise the number of write(2) syscalls, and the problem goes away.
However, there is a small change we can make to improve the situation - removing the expensive lock operation on the change counter update. All inode version counter changes in XFS occur under the ip->i_ilock during a transaction, and therefore we don't actually need the spin lock that provides exclusive access to it through inc_inode_iversion().
Hence avoid the lock and just open code the increment ourselves when logging the inode.
--- fs/xfs/xfs_trans_inode.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/xfs/xfs_trans_inode.c +1 -2
@@ -118,8 +118,7 @@ xfs_trans_log_inode(*/ if (!(ip->i_itemp->ili_item.li_desc->lid_flags &XFS_LID_DIRTY ) &&IS_I_VERSION (VFS_I (ip))) { -inode_inc_iversion (VFS_I (ip)); - ip->i_d.di_changecount =VFS_I (ip)->i_version; + ip->i_d.di_changecount = ++VFS_I (ip)->i_version; flags |=XFS_ILOG_CORE ; }
Signature · 6 lines
1.8.3.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfs: open code inc_inode_iversion when logging an inode
Quoted 43 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Michael L Semon reported that generic/069 runtime increased on v5 superblocks by 100% compared to v4 superblocks. his perf-based analysis pointed directly at the timestamp updates being done by the write path in this workload. The append writers are doing 4-byte writes, so there are lots of timestamp updates occurring.
The thing is, they aren't being triggered by timestamp changes - they are being triggered by the inode change counter needing to be updated. That is, every write(2) system call needs to bump the inode version count, and it does that through the timestamp update mechanism. Hence for v5 filesystems, test generic/069 is running 3 orders of magnitude more timestmap update transactions on v5 filesystems due to the fact it does a huge number of *4 byte* write(2) calls.
This isn't a real world scenario we really need to address - anyone doing such sequential IO should be using fwrite(3), not write(2). i.e. fwrite(3) buffers the writes in userspace to minimise the number of write(2) syscalls, and the problem goes away.
However, there is a small change we can make to improve the situation - removing the expensive lock operation on the change counter update. All inode version counter changes in XFS occur under the
ip->i_ilockduring a transaction, and therefore we don't actually need the spin lock that provides exclusive access to it throughinc_inode_iversion().Hence avoid the lock and just open code the increment ourselves when logging the inode.
Reported-by Michael L. Semon <[email protected]>Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_trans_inode.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)fs/xfs/xfs_trans_inode.c +0
-0
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 122
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +0
-2@@ -118,8 +118,7 @@ xfs_trans_log_inode(*/ if (!(ip->i_itemp->ili_item.li_desc->lid_flags &XFS_LID_DIRTY ) &&IS_I_VERSION (VFS_I (ip))) { -inode_inc_iversion (VFS_I (ip)); - ip->i_d.di_changecount =VFS_I (ip)->i_version;
comment about the reason for the open-code might be good, too?
otherwise some semantic patcher might "fix" it for you again later...
-Eric
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 124
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +1
-0+ ip->i_d.di_changecount = ++VFS_I (ip)->i_version; flags |=XFS_ILOG_CORE ; }
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfs: open code inc_inode_iversion when logging an inode
Hi Gents,
Quoted 54 lines · Eric Sandeen
Quoted 43 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Michael L Semon reported that generic/069 runtime increased on v5 superblocks by 100% compared to v4 superblocks. his perf-based analysis pointed directly at the timestamp updates being done by the write path in this workload. The append writers are doing 4-byte writes, so there are lots of timestamp updates occurring.
The thing is, they aren't being triggered by timestamp changes - they are being triggered by the inode change counter needing to be updated. That is, every write(2) system call needs to bump the inode version count, and it does that through the timestamp update mechanism. Hence for v5 filesystems, test generic/069 is running 3 orders of magnitude more timestmap update transactions on v5 filesystems due to the fact it does a huge number of *4 byte* write(2) calls.
This isn't a real world scenario we really need to address - anyone doing such sequential IO should be using fwrite(3), not write(2). i.e. fwrite(3) buffers the writes in userspace to minimise the number of write(2) syscalls, and the problem goes away.
However, there is a small change we can make to improve the situation - removing the expensive lock operation on the change counter update. All inode version counter changes in XFS occur under the
ip->i_ilockduring a transaction, and therefore we don't actually need the spin lock that provides exclusive access to it throughinc_inode_iversion().Hence avoid the lock and just open code the increment ourselves when logging the inode.
Reported-by Michael L. Semon <[email protected]>Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_trans_inode.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)fs/xfs/xfs_trans_inode.c +0
-0fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 122
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +0
-2@@ -118,8 +118,7 @@ xfs_trans_log_inode(*/ if (!(ip->i_itemp->ili_item.li_desc->lid_flags &XFS_LID_DIRTY ) &&IS_I_VERSION (VFS_I (ip))) { -inode_inc_iversion (VFS_I (ip)); - ip->i_d.di_changecount =VFS_I (ip)->i_version;comment about the reason for the open-code might be good, too?
otherwise some semantic patcher might "fix" it for you again later...
Quoted 7 lines
-Eric
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 124
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +1
-0+ ip->i_d.di_changecount = ++VFS_I (ip)->i_version; flags |=XFS_ILOG_CORE ; }
Adding a comment strikes me as a good idea too... But isn't that lock there for a reason? I suspect that will break i_version like i_size on 32 bit systems. Jean added this function, hopefully he can shed some light.
Thanks,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfs: open code inc_inode_iversion when logging an inode
Quoted 9 lines · Ben Myers
Quoted 8 lines · Eric Sandeen
Quoted 7 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Michael L Semon reported that generic/069 runtime increased on v5 superblocks by 100% compared to v4 superblocks. his perf-based analysis pointed directly at the timestamp updates being done by the write path in this workload. The append writers are doing 4-byte writes, so there are lots of timestamp updates occurring.
...
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
fs/xfs/xfs_trans_inode.c +0
-0
Quoted 8 lines
Quoted 8 lines
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 122
Quoted 6 lines
fs/xfs/xfs_trans_inode.c +0
-2@@ -118,8 +118,7 @@ xfs_trans_log_inode(*/ if (!(ip->i_itemp->ili_item.li_desc->lid_flags &XFS_LID_DIRTY ) &&IS_I_VERSION (VFS_I (ip))) { -inode_inc_iversion (VFS_I (ip)); - ip->i_d.di_changecount =VFS_I (ip)->i_version;comment about the reason for the open-code might be good, too?
Sure, I can add that.
Quoted 4 lines
Quoted 4 lines
otherwise some semantic patcher might "fix" it for you again later...
-Eric
Quoted 9 lines
Quoted 5 lines
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 124
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +1
-0+ ip->i_d.di_changecount = ++VFS_I (ip)->i_version; flags |=XFS_ILOG_CORE ; }Adding a comment strikes me as a good idea too... But isn't that lock there for a reason? I suspect that will break
i_versionlikei_sizeon 32 bit systems. Jean added this function, hopefully he can shed some light.
I can't see how there's a 32 bit issue here - i_version is always read unlocked, and so if you're worried about a 32 bit system doing 2 32 bit reads to read the 64 bit value and seeing values on different sides of the increment, then we've already got that problem *everywhere*. i.e. the only place that i_version is protected by i_lock is in inode_inc_iversion() - nowhere else is that lock used at all when reading or writing i_version.
A quick grep points out that ext2/3/4 directory code all update and read i_version without using the i_lock - they are all serialised by the directory locks that are held. Ceph, exofs, ocfs2, ecryptfs, affs, fat, etc all do similar things with inode->i_version.
So if the intention is to make i_version safe on 32 bit systems, then it's failed. The only thing it does in inode_inc_iversion is serialise other updates that aren't done under some exclusive inode locks, and all the XFS updates are done either under the i_mutex and/or the i_ilock, so I don't think there is any problem with racing occurring here...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfs: open code inc_inode_iversion when logging an inode
Hi Dave,
Quoted 40 lines · Dave Chinner
Quoted 9 lines · Ben Myers
Quoted 8 lines · Eric Sandeen
Quoted 7 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Michael L Semon reported that generic/069 runtime increased on v5 superblocks by 100% compared to v4 superblocks. his perf-based analysis pointed directly at the timestamp updates being done by the write path in this workload. The append writers are doing 4-byte writes, so there are lots of timestamp updates occurring.
...
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
fs/xfs/xfs_trans_inode.c +0
-0Quoted 8 lines
Quoted 8 lines
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 122
Quoted 6 lines
fs/xfs/xfs_trans_inode.c +0
-2@@ -118,8 +118,7 @@ xfs_trans_log_inode(*/ if (!(ip->i_itemp->ili_item.li_desc->lid_flags &XFS_LID_DIRTY ) &&IS_I_VERSION (VFS_I (ip))) { -inode_inc_iversion (VFS_I (ip)); - ip->i_d.di_changecount =VFS_I (ip)->i_version;comment about the reason for the open-code might be good, too?
Sure, I can add that.
Quoted 4 lines
Quoted 4 lines
otherwise some semantic patcher might "fix" it for you again later...
-Eric
Quoted 9 lines
Quoted 5 lines
fs/xfs/xfs_trans_inode.c xfs_trans_log_inode( · line 124
Quoted 5 lines
fs/xfs/xfs_trans_inode.c +1
-0+ ip->i_d.di_changecount = ++VFS_I (ip)->i_version; flags |=XFS_ILOG_CORE ; }Adding a comment strikes me as a good idea too... But isn't that lock there for a reason? I suspect that will break
i_versionlikei_sizeon 32 bit systems. Jean added this function, hopefully he can shed some light.
Quoted 5 lines
I can't see how there's a 32 bit issue here -
i_versionis always read unlocked, and so if you're worried about a 32 bit system doing 2 32 bit reads to read the 64 bit value and seeing values on different sides of the increment, then we've already got that problem *everywhere*.
I think if we had the 32 bit issues with i_size, the same is likely true here. You're not making it any worse, AFAICT.
Quoted 3 lines
i.e. the only place that
i_versionis
protected byi_lockis ininode_inc_iversion()- nowhere else is that lock used at all when reading or writingi_version.
Seems like if nobody is taking the i_lock when reading i_version, it's not really providing the protection that was intended. Weird.
Quoted 7 lines
A quick grep points out that ext2/3/4 directory code all update and read
i_versionwithout using thei_lock- they are all serialised by the directory locks that are held. Ceph, exofs, ocfs2, ecryptfs, affs, fat, etc all do similar things withinode->i_version.So if the intention is to make
i_versionsafe on 32 bit systems, then it's failed.
Agreed.
Quoted 5 lines
The only thing it does in
inode_inc_iversionis serialise other updates that aren't done under some exclusive inode locks, and all the XFS updates are done either under thei_mutexand/or thei_ilock, so I don't think there is any problem with racing occurring here...
I'll take another look at it with that in mind.
Thanks,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Dave Chinner
Hi folks,
The first two patches are regressions and hence -rc4 candidates - the first patch fixes a lockdep false positive as a result of adding the third quota inode, and the second fixes a on-disk format interpretation problem in the v4 superblock dtype-in-dirent feature.
Applied 1 and 2.
Thanks,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs