Skip to contentSkip to conversation

working on extent locks for i_mutex

DISCUSSION Allison Henderson 6 participants 13 messages Oct 7, 2011 – Jan 20, 2012

lore
Share & tools
lore
#0 lore raw

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

A while ago I had done some investigation on where i_mutex is currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!

Allison

List of ext4 functions that lock i_mutex:
ext4_sync_file
ext4_fallocate
ext4_move_extents via two helper routines:
mext_inode_double_lock and mext_inode_double_unlock
ext4_ioctl (for the EXT4_IOC_SETFLAGS ioctl) ext4_quota_write
ext4_llseek
ext4_end_io_work
ext4_ind_direct_IO (only while calling ext4_flush_completed_IO)

Functions called by vfs with i_mutex locked:
ext4_setattr
ext4_da_writepages
ext4_rmdir
ext4_unlink
ext4_symlink
ext4_link
ext4_rename
ext4_get_block

#1 lore raw
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 24 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Quoted 3 lines

Cheers,

Dave.

Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!

Allison Henderson

#3 lore raw
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Quoted 5 lines

Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!

Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in lib/ was what I was thinking....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 32 lines · Dave Chinner
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Quoted 5 lines

Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!

Quoted 7 lines

Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in lib/ was what I was thinking....

Cheers,

Dave.

Alrighty, that sounds good to me. I will aim to keep it as general purpose as I can. I am going to start some proto typing and will post back when I get something working. Thx for the feedback all! :)

Allison

#5 lore raw
Quoted 37 lines · Allison Henderson
Quoted 36 lines · Dave Chinner
Quoted 25 lines · Allison Henderson
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Quoted 5 lines

Yes that is generally the idea I was thinking about doing, but at the time, I was not thinking outside the scope of ext4. You are thinking maybe it should be in vfs layer so that it's something that all the filesystems will use? That seems to be the impression I'm getting from folks. Thx!

Yes, that's what I'm suggesting. Not so much a vfs layer function, but a library (range locks could be useful outside filesystems) so locating it in lib/ was what I was thinking....

Quoted 8 lines
Quoted 3 lines

Cheers,

Dave.

Alrighty, that sounds good to me. I will aim to keep it as general purpose as I can. I am going to start some proto typing and will post back when I get something working. Thx for the feedback all! :)

Hi Allison,

For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.

I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with dioread_nolock.

To understand the effect of lock contention, I define a new function called ext4_file_aio_write() that calls __generic_file_aio_write() without acquiring i_mutex lock. Meanwhile, I remove DIO_LOCKING flag when __blockdev_direct_IO() is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that the i_mutex heavily impacts the performance. Hopefully the result is useful for you. :-)

I post the result in here.

config file:
[global]
filesize=64G
size=64G
bs=16k
ioengine=psync
direct=1
filename=/mnt/ext4/benchmark
runtime=600
group_reporting
thread

[randrw]
numjobs=32
rw=randrw
rwmixread=90

result:

iops 1 (r/w) 2 3
ext4 5584/622 5726/636 5719/636
ext4+dioread_nolock 7105/789 7117/793 7129/795
ext4+dio_nolock 8920/992 8956/995 8976/997
xfs 8726/971 8962/994 8975/998

bandwidth 1 (r/w) 2 3 KB/s
ext4 89359/9955.3 91621/10186 91519/10185
ext4+dioread_nolock 113691/12635 113882/12692 114066/12728
ext4+dio_nolock 142731/15888 143301/15930 143617/15959
xfs 139627/15537 143400/15914 143603/15980

latency 1 (r/w) 2 3 usec
ext4 5163.28/5048.31 5037.81/4914.82 5041.49/4932.81
ext4+dioread_nolock 1220.04/29510.5 1213.67/29418.9 1208.77/29361.49
ext4+dio_nolock 3226.61/3194.35 3214.59/3178.09 3207.34/3173.78
xfs 3299.87/3266.32 3213.73/3182.20 3208.16/3178.10

Regards,
Zheng

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 8 lines · Zheng Liu

For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.

I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with dioread_nolock.

Quoted 6 lines

To understand the effect of lock contention, I define a new function called ext4_file_aio_write() that calls __generic_file_aio_write() without acquiring i_mutex lock. Meanwhile, I remove DIO_LOCKING flag when __blockdev_direct_IO() is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that the i_mutex heavily impacts the performance. Hopefully the result is useful for you. :-)

For the record, I have a patchset that, while not affecting i_mutex (or locking in general), does allow AIO append writes to actually be done asynchronously. (Currently they're forced to be done synchronously.) It makes a big difference in performance for that particular case, even for spinning media. Performance roughly doubled when testing with fio against a regular two-terabyte drive; the performance improvement against SSD would have to be much greater.

One day soon I'll accumulate enough spare time to port the patchset forward to the latest kernel and submit it here.

Signature · 2 lines
Frank Mayhar
[email protected]
#7 lore raw
Quoted 24 lines · Frank Mayhar
Quoted 8 lines · Zheng Liu

For this project, do you have a schedule? Would you like to share to me? This lock contention heavily impacts the performance of direct IO in our production environment. So we hope to improve it ASAP.

I have done some direct IO benchmarks to compare ext4 with xfs using fio in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and ext4 with dioread_nolock.

Quoted 6 lines

To understand the effect of lock contention, I define a new function called ext4_file_aio_write() that calls __generic_file_aio_write() without acquiring i_mutex lock. Meanwhile, I remove DIO_LOCKING flag when __blockdev_direct_IO() is called and do the similar benchmarks. The result shows that the performance in ext4 is almost the same to the xfs. Thus, it proves that the i_mutex heavily impacts the performance. Hopefully the result is useful for you. :-)

For the record, I have a patchset that, while not affecting i_mutex (or locking in general), does allow AIO append writes to actually be done asynchronously. (Currently they're forced to be done synchronously.) It makes a big difference in performance for that particular case, even for spinning media. Performance roughly doubled when testing with fio against a regular two-terabyte drive; the performance improvement against SSD would have to be much greater.

Quoted 2 lines

One day soon I'll accumulate enough spare time to port the patchset forward to the latest kernel and submit it here.

Interesting. I think it might help us to improve this issue. So could you please post your test case and result in detail? Thank you. :-)

Regards,
Zheng

#8 lore raw
In reply to Dave Chinner
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.

btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)

Thanks
Tao

#9 lore raw
Quoted 19 lines · Tao Ma
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 8 lines
Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.

No, it doesn't have range locks. If has separate locks for IO exclusion vs metadata modification (i_iolock vs i_ilock). Both are rwsems, the ilock nests inside and protects the extent list and other metadata.

What I want to do is replace the i_iolock with a read/write range lock so that we can do sane cache coherent concurrent IO to separate ranges of the file. We can't do concurrent modifications to the extent tree, so we have no need for changing the i_ilock (metadata) lock to range locks.

Quoted 3 lines

btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)

flock is way more complex than simple read/write range locks and has fixed semantics and lots of scope for difficult to find regressions, so I wouldn't even bother trying to support them...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw
Quoted 34 lines · Dave Chinner
Quoted 19 lines · Tao Ma
Quoted 18 lines · Dave Chinner
Quoted 8 lines · Allison Henderson

Hi All,

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

Quoted 8 lines
Quoted 5 lines

and implementing with an rbtree or a btree for tracking, then I definitely have a use for it in XFS - replacing the current rwsem that is used for the iolock. Range locks like this are the only thing we need to allow concurrent buffered writes to the same file to maintain the per-write exclusion that posix requires.

Interesting, so xfs already have these range lock, right? If yes, any possibility that the code can be reused in ext4 since we have the same thing in mind but don't have any resource to work on it by now.

No, it doesn't have range locks. If has separate locks for IO exclusion vs metadata modification (i_iolock vs i_ilock). Both are rwsems, the ilock nests inside and protects the extent list and other metadata.

Quoted 5 lines

What I want to do is replace the i_iolock with a read/write range lock so that we can do sane cache coherent concurrent IO to separate ranges of the file. We can't do concurrent modifications to the extent tree, so we have no need for changing the i_ilock (metadata) lock to range locks.

OK, I see. Thanks for the information.

Quoted 9 lines
Quoted 5 lines

btw, IIRC flock(2) uses a list to indicate the range lock, so if we can make these pieces of codes common, at least there are 3 places that can benefit from it. ;)

flock is way more complex than simple read/write range locks and has fixed semantics and lots of scope for difficult to find regressions, so I wouldn't even bother trying to support them...

fair enough. :)

Thanks
Tao

#11 lore raw
In reply to Allison Henderson
Quoted 3 lines · Allison Henderson

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

A while ago I had done some investigation on where i_mutex is currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!

The in-ext4 users appear to all be file IO related, while the VFS functions are mostly directory related, though I don't see any mention of the callers of ext4_mkdir() or ext4_mknod() or ext4_create()?

For Lustre we developed a patch that allows parallel metadata operations on directories (e.g. concurrent lookup, mkdir, rmdir, create, unlink in a single directory) which would replace i_mutex for the namespace operations. Patch:

http://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=ldiskfs/kernel_patches/patches/ext4_pdirop-rhel6.patch;hb=HEAD

though this in itself isn't enough to allow the VFS to do parallel directory operations. We're of course also interested in parallel file IO operations through the VFS for the client, though this hasn't been a focus of ours since we typically have a large number of clients doing IO concurrently.

Quoted 22 lines

List of ext4 functions that lock i_mutex:
ext4_sync_file
ext4_fallocate
ext4_move_extents via two helper routines:
mext_inode_double_lock and mext_inode_double_unlock
ext4_ioctl (for the EXT4_IOC_SETFLAGS ioctl) ext4_quota_write
ext4_llseek
ext4_end_io_work
ext4_ind_direct_IO (only while calling ext4_flush_completed_IO)

Functions called by vfs with i_mutex locked:
ext4_setattr
ext4_da_writepages
ext4_rmdir
ext4_unlink
ext4_symlink
ext4_link
ext4_rename
ext4_get_block

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Cheers, Andreas

#12 lore raw
Quoted 8 lines · Andreas Dilger
Quoted 3 lines · Allison Henderson

I know this is an old topic, but I am poking it again because I've had some work items wrap up, and Im planning on picking up on this one again. I am thinking about implementing extent locks to replace i_mutex. So I just wanted to touch base with folks and see what people are working on because I know there were some folks out there that were thing about doing similar solutions.

A while ago I had done some investigation on where i_mutex is currently used, so I did a review and updated my list. Only one thing had been removed, but I will leave the list here since it was a while ago. Let me know if anyone has been working on similar concept. Thx!

The in-ext4 users appear to all be file IO related, while the VFS functions are mostly directory related, though I don't see any mention of the callers of ext4_mkdir() or ext4_mknod() or ext4_create()?

Hmm, those ones didnt turn up when I was looking for i_mutex locking. Though it would make sense that they would be locked. Did I over look it in a helper function somewhere?

Quoted 5 lines

For Lustre we developed a patch that allows parallel metadata operations on directories (e.g. concurrent lookup, mkdir, rmdir, create, unlink in a single directory) which would replace i_mutex for the namespace operations. Patch:

Quoted 7 lines

http://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=ldiskfs/kernel_patches/patches/ext4_pdirop-rhel6.patch;hb=HEAD

though this in itself isn't enough to allow the VFS to do parallel directory operations. We're of course also interested in parallel file IO operations through the VFS for the client, though this hasn't been a focus of ours since we typically have a large number of clients doing IO concurrently.

I see, I will take a look at it, maybe there will be some things I can borrow from it. Thx!

Quoted 34 lines
Quoted 22 lines

List of ext4 functions that lock i_mutex:
ext4_sync_file
ext4_fallocate
ext4_move_extents via two helper routines:
mext_inode_double_lock and mext_inode_double_unlock
ext4_ioctl (for the EXT4_IOC_SETFLAGS ioctl) ext4_quota_write
ext4_llseek
ext4_end_io_work
ext4_ind_direct_IO (only while calling ext4_flush_completed_IO)

Functions called by vfs with i_mutex locked:
ext4_setattr
ext4_da_writepages
ext4_rmdir
ext4_unlink
ext4_symlink
ext4_link
ext4_rename
ext4_get_block

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Cheers, Andreas