working on extent locks for i_mutex
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.
A while ago I had done some investigation on where i_mutex is currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!
Allison
List of ext4 functions that lock i_mutex:ext4_sync_fileext4_fallocateext4_move_extents via two helper routines:
mext_inode_double_lock and mext_inode_double_unlockext4_ioctl (for the EXT4_IOC_SETFLAGS ioctl) ext4_quota_writeext4_llseekext4_end_io_workext4_ind_direct_IO (only while calling ext4_flush_completed_IO)
Functions called by vfs with i_mutex locked:ext4_setattrext4_da_writepagesext4_rmdirext4_unlinkext4_symlinkext4_linkext4_renameext4_get_block
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.
What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 24 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Quoted 3 lines
Cheers,
Dave.
Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!
Allison Henderson
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Quoted 5 lines
Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!
Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in lib/ was what I was thinking....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 32 lines · Dave Chinner
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Quoted 5 lines
Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!
Quoted 7 lines
Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in
lib/was what I was thinking....Cheers,
Dave.
Alrighty, that sounds good to me. I will aim to keep it as general purpose as I can. I am going to start some proto typing and will post back when I get something working. Thx for the feedback all! :)
Allison
Quoted 37 lines · Allison Henderson
Quoted 36 lines · Dave Chinner
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Quoted 5 lines
Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!
Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in
lib/was what I was thinking....
Quoted 8 lines
Quoted 3 lines
Cheers,
Dave.
Alrighty, that sounds good to me. I will aim to keep it as general purpose as I can. I am going to start some proto typing and will post back when I get something working. Thx for the feedback all! :)
Hi Allison,
For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.
I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with dioread_nolock.
To understand the effect of lock contention, I define a new function called ext4_file_aio_write() that calls __generic_file_aio_write() without acquiring i_mutex lock. Meanwhile, I remove DIO_LOCKING flag when __blockdev_direct_IO() is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that the i_mutex heavily impacts the performance. Hopefully the result is useful for you. :-)
I post the result in here.
config file:
[global]
filesize=64G
size=64G
bs=16k
ioengine=psync
direct=1
filename=/mnt/ext4/benchmark
runtime=600group_reporting
thread
[randrw]
numjobs=32
rw=randrw
rwmixread=90
result:
iops 1 (r/w) 2 3
ext4 5584/622 5726/636 5719/636
ext4+dioread_nolock 7105/789 7117/793 7129/795
ext4+dio_nolock 8920/992 8956/995 8976/997
xfs 8726/971 8962/994 8975/998
bandwidth 1 (r/w) 2 3 KB/s
ext4 89359/9955.3 91621/10186 91519/10185
ext4+dioread_nolock 113691/12635 113882/12692 114066/12728
ext4+dio_nolock 142731/15888 143301/15930 143617/15959
xfs 139627/15537 143400/15914 143603/15980
latency 1 (r/w) 2 3 usec
ext4 5163.28/5048.31 5037.81/4914.82 5041.49/4932.81
ext4+dioread_nolock 1220.04/29510.5 1213.67/29418.9 1208.77/29361.49
ext4+dio_nolock 3226.61/3194.35 3214.59/3178.09 3207.34/3173.78
xfs 3299.87/3266.32 3213.73/3182.20 3208.16/3178.10
Regards,
Zheng
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Zheng Liu
For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.
I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with
dioread_nolock.
Quoted 6 lines
To understand the effect of lock contention, I define a new function called
ext4_file_aio_write()that calls__generic_file_aio_write()without acquiringi_mutexlock. Meanwhile, I removeDIO_LOCKINGflag when__blockdev_direct_IO()is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that thei_mutexheavily impacts the performance. Hopefully the result is useful for you. :-)
For the record, I have a patchset that, while not affecting i_mutex (or locking in general), does allow AIO append writes to actually be done asynchronously. (Currently they're forced to be done synchronously.) It makes a big difference in performance for that particular case, even for spinning media. Performance roughly doubled when testing with fio against a regular two-terabyte drive; the performance improvement against SSD would have to be much greater.
One day soon I'll accumulate enough spare time to port the patchset forward to the latest kernel and submit it here.
Signature · 2 lines
Frank Mayhar [email protected]
Quoted 24 lines · Frank Mayhar
Quoted 8 lines · Zheng Liu
For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.
I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with
dioread_nolock.Quoted 6 lines
To understand the effect of lock contention, I define a new function called
ext4_file_aio_write()that calls__generic_file_aio_write()without acquiringi_mutexlock. Meanwhile, I removeDIO_LOCKINGflag when__blockdev_direct_IO()is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that thei_mutexheavily impacts the performance. Hopefully the result is useful for you. :-)For the record, I have a patchset that, while not affecting
i_mutex(or locking in general), does allow AIO append writes to actually be done asynchronously. (Currently they're forced to be done synchronously.) It makes a big difference in performance for that particular case, even for spinning media. Performance roughly doubled when testing with fio against a regular two-terabyte drive; the performance improvement against SSD would have to be much greater.
Quoted 2 lines
One day soon I'll accumulate enough spare time to port the patchset forward to the latest kernel and submit it here.
Interesting. I think it might help us to improve this issue. So could you please post your test case and result in detail? Thank you. :-)
Regards,
Zheng
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)
Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.
btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)
Thanks
Tao
Quoted 19 lines · Tao Ma
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)
Quoted 8 lines
Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.
No, it doesn't have range locks. If has separate locks for IO exclusion vs metadata modification (i_iolock vs i_ilock). Both are rwsems, the ilock nests inside and protects the extent list and other metadata.
What I want to do is replace the i_iolock with a read/write range lock so that we can do sane cache coherent concurrent IO to separate ranges of the file. We can't do concurrent modifications to the extent tree, so we have no need for changing the i_ilock (metadata) lock to range locks.
Quoted 3 lines
btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)
flock is way more complex than simple read/write range locks and has fixed semantics and lots of scope for difficult to find regressions, so I wouldn't even bother trying to support them...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 34 lines · Dave Chinner
Quoted 19 lines · Tao Ma
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson
Hi All,
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.What locking API are you looking at? If you are looking at an something like:
read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)write_range_unlock(lock, off, len)Quoted 8 lines
Quoted 5 lines
and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.
Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.
No, it doesn't have range locks. If has separate locks for IO exclusion vs metadata modification (
i_iolockvsi_ilock). Both are rwsems, the ilock nests inside and protects the extent list and other metadata.
Quoted 5 lines
What I want to do is replace the
i_iolockwith a read/write range lock so that we can do sane cache coherent concurrent IO to separate ranges of the file. We can't do concurrent modifications to the extent tree, so we have no need for changing thei_ilock(metadata) lock to range locks.
OK, I see. Thanks for the information.
Quoted 9 lines
Quoted 5 lines
btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)
flock is way more complex than simple read/write range locks and has fixed semantics and lots of scope for difficult to find regressions, so I wouldn't even bother trying to support them...
fair enough. :)
Thanks
Tao
Quoted 3 lines · Allison Henderson
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.A while ago I had done some investigation on where
i_mutexis currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!
The in-ext4 users appear to all be file IO related, while the VFS functions are mostly directory related, though I don't see any mention of the callers of ext4_mkdir() or ext4_mknod() or ext4_create()?
For Lustre we developed a patch that allows parallel metadata operations on directories (e.g. concurrent lookup, mkdir, rmdir, create, unlink in a single directory) which would replace i_mutex for the namespace operations. Patch:
though this in itself isn't enough to allow the VFS to do parallel directory operations. We're of course also interested in parallel file IO operations through the VFS for the client, though this hasn't been a focus of ours since we typically have a large number of clients doing IO concurrently.
Quoted 22 lines
List of ext4 functions that lock
i_mutex:ext4_sync_fileext4_fallocateext4_move_extentsvia two helper routines:
mext_inode_double_lockandmext_inode_double_unlockext4_ioctl(for theEXT4_IOC_SETFLAGSioctl)ext4_quota_writeext4_llseekext4_end_io_workext4_ind_direct_IO(only while callingext4_flush_completed_IO)Functions called by vfs with
i_mutexlocked:ext4_setattrext4_da_writepagesext4_rmdirext4_unlinkext4_symlinkext4_linkext4_renameext4_get_block
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
Quoted 8 lines · Andreas Dilger
Quoted 3 lines · Allison Henderson
I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace
i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.A while ago I had done some investigation on where
i_mutexis currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!The in-ext4 users appear to all be file IO related, while the VFS functions are mostly directory related, though I don't see any mention of the callers of
ext4_mkdir()orext4_mknod()orext4_create()?
Hmm, those ones didnt turn up when I was looking for i_mutex locking. Though it would make sense that they would be locked. Did I over look it in a helper function somewhere?
Quoted 5 lines
For Lustre we developed a patch that allows parallel metadata operations on directories (e.g. concurrent lookup, mkdir, rmdir, create, unlink in a single directory) which would replace
i_mutexfor the namespace operations. Patch:
Quoted 7 lines
though this in itself isn't enough to allow the VFS to do parallel directory operations. We're of course also interested in parallel file IO operations through the VFS for the client, though this hasn't been a focus of ours since we typically have a large number of clients doing IO concurrently.
I see, I will take a look at it, maybe there will be some things I can borrow from it. Thx!
Quoted 34 lines
Quoted 22 lines
List of ext4 functions that lock
i_mutex:ext4_sync_fileext4_fallocateext4_move_extentsvia two helper routines:
mext_inode_double_lockandmext_inode_double_unlockext4_ioctl(for theEXT4_IOC_SETFLAGSioctl)ext4_quota_writeext4_llseekext4_end_io_workext4_ind_direct_IO(only while callingext4_flush_completed_IO)Functions called by vfs with
i_mutexlocked:ext4_setattrext4_da_writepagesext4_rmdirext4_unlinkext4_symlinkext4_linkext4_renameext4_get_blockQuoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlCheers, Andreas