xfs: properly serialise fallocate against AIO+DIO
PATCH v2 1/1 xfs: properly serialise fallocate against AIO+DIO
From: Dave Chinner <[email protected]>
AIO+DIO can extend the file size on IO completion, and it holds no inode locks while the IO is in flight. Therefore, a race condition exists in file size updates if we do something like this:
aio-thread fallocate-thread
lock inode
submit IO beyond inode->i_size
unlock inode
.....
lock inode break layouts if (off + len > inode->i_size) new_size = off + len ..... inode_dio_wait() <blocks>
.....
completesinode->i_size updatedinode_dio_done()
....
<wakes> <does stuff no long beyond EOF> if (new_size) xfs_vn_setattr(inode, new_size)
Yup, that attempt to extend the file size in the fallocate code turns into a truncate - it removes the whatever the aio write allocated and put to disk, and reduced the inode size back down to where the fallocate operation ends.
Fundamentally, xfs_file_fallocate() not compatible with racing AIO+DIO completions, so we need to move the inode_dio_wait() call up to where the lock the inode and break the layouts.
Secondly, storing the inode size and then using it unchecked without holding the ILOCK is not safe; we can only do such a thing if we've locked out and drained all IO and other modification operations, which we don't do initially in xfs_file_fallocate.
It should be noted that some of the fallocate operations are compound operations - they are made up of multiple manipulations that may zero data, and so we may need to flush and invalidate the file multiple times during an operation. However, we only need to lock out IO and other space manipulation operations once, as that lockout is maintained until the entire fallocate operation has been completed.
--- fs/xfs/xfs_bmap_util.c | 8 +------- fs/xfs/xfs_file.c | 23 +++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 1 + 3 files changed, 25 insertions(+), 7 deletions(-)
fs/xfs/xfs_bmap_util.c +1 -7
@@ -1040,6 +1040,7 @@ xfs_unmap_extent( goto out_unlock; } +/* Caller must first wait for the completion of any pending DIOs if required. */ int xfs_flush_unmap_range( struct xfs_inode *ip,
@@ -1051,9 +1052,6 @@ xfs_flush_unmap_range(xfs_off_t rounding, start, end;int error; -/* wait for the completion of any pending DIOs */ -inode_dio_wait (inode); - rounding =max_t (xfs_off_t,1 << mp->m_sb.sb_blocklog,PAGE_SIZE ); start =round_down (offset, rounding); end =round_up (offset + len, rounding) -1 ;
@@ -1085,10 +1083,6 @@ xfs_free_file_space(if (len <=0 )/* if nothing being freed */ return 0 ; - error =xfs_flush_unmap_range (ip, offset, len); -if (error) -return error; - startoffset_fsb =XFS_B_TO_FSB (mp, offset); endoffset_fsb =XFS_B_TO_FSBT (mp, offset + len);
fs/xfs/xfs_file.c +23 -0
@@ -817,6 +817,29 @@ xfs_file_fallocate(if (error) goto out_unlock; +/* +* Must wait for all AIO to complete before we continue as AIO can +* change the file size on completion without holding any locks we +* currently hold. We must do this first because AIO can update both +* the on disk and in memory inode sizes, and the operations that follow +* require the in-memory size to be fully up-to-date. +*/ +inode_dio_wait (inode); + +/* +* Now that AIO and DIO has drained we can flush and (if necessary) +* invalidate the cached range over the first operation we are about to +* run. We include zero and collapse here because they both start with a +* hole punch over the target range. Insert and collapse both invalidate +* the broader range affected by the shift in xfs_prepare_shift(). +*/ +if (mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_ZERO_RANGE | +FALLOC_FL_COLLAPSE_RANGE )) { + error =xfs_flush_unmap_range (ip, offset, len); +if (error) + goto out_unlock; + } +if (mode &FALLOC_FL_PUNCH_HOLE ) { error =xfs_free_file_space (ip, offset, len);if (error)
fs/xfs/xfs_ioctl.c +1 -0
@@ -623,6 +623,7 @@ xfs_ioc_space( error =xfs_break_layouts (inode, &iolock,BREAK_UNMAP );if (error) goto out_unlock; +inode_dio_wait (inode);switch (bf->l_whence) {case 0 :/*SEEK_SET*/
Signature · 1 line
2.24.0.rc0
Quoted 52 lines · Dave Chinner
From: Dave Chinner <[email protected]>
AIO+DIO can extend the file size on IO completion, and it holds no inode locks while the IO is in flight. Therefore, a race condition exists in file size updates if we do something like this:
aio-thread fallocate-thread
lock inode
submit IO beyondinode->i_size
unlock inode
.....lock inode break layouts if (off + len > inode->i_size) new_size = off + len ..... inode_dio_wait() <blocks>.....
completesinode->i_sizeupdatedinode_dio_done()
....<wakes> <does stuff no long beyond EOF> if (new_size) xfs_vn_setattr(inode, new_size)Yup, that attempt to extend the file size in the fallocate code turns into a truncate - it removes the whatever the aio write allocated and put to disk, and reduced the inode size back down to where the fallocate operation ends.
Fundamentally,
xfs_file_fallocate()not compatible with racing AIO+DIO completions, so we need to move theinode_dio_wait()call up to where the lock the inode and break the layouts.Secondly, storing the inode size and then using it unchecked without holding the ILOCK is not safe; we can only do such a thing if we've locked out and drained all IO and other modification operations, which we don't do initially in
xfs_file_fallocate.It should be noted that some of the fallocate operations are compound operations - they are made up of multiple manipulations that may zero data, and so we may need to flush and invalidate the file multiple times during an operation. However, we only need to lock out IO and other space manipulation operations once, as that lockout is maintained until the entire fallocate operation has been completed.
Quoted 1 line
Signed-off-by Dave Chinner <[email protected]>
Looks reasonable to me; what do you think of my regression test?
--D
Quoted 88 lines
--- fs/xfs/xfs_bmap_util.c | 8 +------- fs/xfs/xfs_file.c | 23 +++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 1 + 3 files changed, 25 insertions(+), 7 deletions(-)fs/xfs/xfs_bmap_util.c +1
-7@@ -1040,6 +1040,7 @@ xfs_unmap_extent( goto out_unlock; } +/* Caller must first wait for the completion of any pending DIOs if required. */ int xfs_flush_unmap_range( struct xfs_inode *ip,@@ -1051,9 +1052,6 @@ xfs_flush_unmap_range(xfs_off_t rounding, start, end;int error; -/* wait for the completion of any pending DIOs */ -inode_dio_wait (inode); - rounding =max_t (xfs_off_t,1 << mp->m_sb.sb_blocklog,PAGE_SIZE ); start =round_down (offset, rounding); end =round_up (offset + len, rounding) -1 ;@@ -1085,10 +1083,6 @@ xfs_free_file_space(if (len <=0 )/* if nothing being freed */ return 0 ; - error =xfs_flush_unmap_range (ip, offset, len); -if (error) -return error; - startoffset_fsb =XFS_B_TO_FSB (mp, offset); endoffset_fsb =XFS_B_TO_FSBT (mp, offset + len);fs/xfs/xfs_file.c +23
-0@@ -817,6 +817,29 @@ xfs_file_fallocate(if (error) goto out_unlock; +/* +* Must wait for all AIO to complete before we continue as AIO can +* change the file size on completion without holding any locks we +* currently hold. We must do this first because AIO can update both +* the on disk and in memory inode sizes, and the operations that follow +* require the in-memory size to be fully up-to-date. +*/ +inode_dio_wait (inode); + +/* +* Now that AIO and DIO has drained we can flush and (if necessary) +* invalidate the cached range over the first operation we are about to +* run. We include zero and collapse here because they both start with a +* hole punch over the target range. Insert and collapse both invalidate +* the broader range affected by the shift in xfs_prepare_shift(). +*/ +if (mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_ZERO_RANGE | +FALLOC_FL_COLLAPSE_RANGE )) { + error =xfs_flush_unmap_range (ip, offset, len); +if (error) + goto out_unlock; + } +if (mode &FALLOC_FL_PUNCH_HOLE ) { error =xfs_free_file_space (ip, offset, len);if (error)fs/xfs/xfs_ioctl.c +1
-0@@ -623,6 +623,7 @@ xfs_ioc_space( error =xfs_break_layouts (inode, &iolock,BREAK_UNMAP );if (error) goto out_unlock; +inode_dio_wait (inode);switch (bf->l_whence) {case 0 :/*SEEK_SET*/ Signature · 1 line
2.24.0.rc0
Quoted 53 lines · Darrick J. Wong
Quoted 52 lines · Dave Chinner
From: Dave Chinner <[email protected]>
AIO+DIO can extend the file size on IO completion, and it holds no inode locks while the IO is in flight. Therefore, a race condition exists in file size updates if we do something like this:
aio-thread fallocate-thread
lock inode
submit IO beyondinode->i_size
unlock inode
.....lock inode break layouts if (off + len > inode->i_size) new_size = off + len ..... inode_dio_wait() <blocks>.....
completesinode->i_sizeupdatedinode_dio_done()
....<wakes> <does stuff no long beyond EOF> if (new_size) xfs_vn_setattr(inode, new_size)Yup, that attempt to extend the file size in the fallocate code turns into a truncate - it removes the whatever the aio write allocated and put to disk, and reduced the inode size back down to where the fallocate operation ends.
Fundamentally,
xfs_file_fallocate()not compatible with racing AIO+DIO completions, so we need to move theinode_dio_wait()call up to where the lock the inode and break the layouts.Secondly, storing the inode size and then using it unchecked without holding the ILOCK is not safe; we can only do such a thing if we've locked out and drained all IO and other modification operations, which we don't do initially in
xfs_file_fallocate.It should be noted that some of the fallocate operations are compound operations - they are made up of multiple manipulations that may zero data, and so we may need to flush and invalidate the file multiple times during an operation. However, we only need to lock out IO and other space manipulation operations once, as that lockout is maintained until the entire fallocate operation has been completed.
Quoted 3 lines
Quoted 1 line
Signed-off-by Dave Chinner <[email protected]>Looks reasonable to me; what do you think of my regression test?
Looks reasonable at a first glance. Not much different what I was using to test this patch. I haven't looked in more detail than that yet...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 52 lines · Dave Chinner
From: Dave Chinner <[email protected]>
AIO+DIO can extend the file size on IO completion, and it holds no inode locks while the IO is in flight. Therefore, a race condition exists in file size updates if we do something like this:
aio-thread fallocate-thread
lock inode
submit IO beyondinode->i_size
unlock inode
.....lock inode break layouts if (off + len > inode->i_size) new_size = off + len ..... inode_dio_wait() <blocks>.....
completesinode->i_sizeupdatedinode_dio_done()
....<wakes> <does stuff no long beyond EOF> if (new_size) xfs_vn_setattr(inode, new_size)Yup, that attempt to extend the file size in the fallocate code turns into a truncate - it removes the whatever the aio write allocated and put to disk, and reduced the inode size back down to where the fallocate operation ends.
Fundamentally,
xfs_file_fallocate()not compatible with racing AIO+DIO completions, so we need to move theinode_dio_wait()call up to where the lock the inode and break the layouts.Secondly, storing the inode size and then using it unchecked without holding the ILOCK is not safe; we can only do such a thing if we've locked out and drained all IO and other modification operations, which we don't do initially in
xfs_file_fallocate.It should be noted that some of the fallocate operations are compound operations - they are made up of multiple manipulations that may zero data, and so we may need to flush and invalidate the file multiple times during an operation. However, we only need to lock out IO and other space manipulation operations once, as that lockout is maintained until the entire fallocate operation has been completed.
Quoted 2 lines
Signed-off-by Dave Chinner <[email protected]>---
Quoted 87 lines
fs/xfs/xfs_bmap_util.c | 8 +------- fs/xfs/xfs_file.c | 23 +++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 1 + 3 files changed, 25 insertions(+), 7 deletions(-)fs/xfs/xfs_bmap_util.c +1
-7@@ -1040,6 +1040,7 @@ xfs_unmap_extent( goto out_unlock; } +/* Caller must first wait for the completion of any pending DIOs if required. */ int xfs_flush_unmap_range( struct xfs_inode *ip,@@ -1051,9 +1052,6 @@ xfs_flush_unmap_range(xfs_off_t rounding, start, end;int error; -/* wait for the completion of any pending DIOs */ -inode_dio_wait (inode); - rounding =max_t (xfs_off_t,1 << mp->m_sb.sb_blocklog,PAGE_SIZE ); start =round_down (offset, rounding); end =round_up (offset + len, rounding) -1 ;@@ -1085,10 +1083,6 @@ xfs_free_file_space(if (len <=0 )/* if nothing being freed */ return 0 ; - error =xfs_flush_unmap_range (ip, offset, len); -if (error) -return error; - startoffset_fsb =XFS_B_TO_FSB (mp, offset); endoffset_fsb =XFS_B_TO_FSBT (mp, offset + len);fs/xfs/xfs_file.c +23
-0@@ -817,6 +817,29 @@ xfs_file_fallocate(if (error) goto out_unlock; +/* +* Must wait for all AIO to complete before we continue as AIO can +* change the file size on completion without holding any locks we +* currently hold. We must do this first because AIO can update both +* the on disk and in memory inode sizes, and the operations that follow +* require the in-memory size to be fully up-to-date. +*/ +inode_dio_wait (inode); + +/* +* Now that AIO and DIO has drained we can flush and (if necessary) +* invalidate the cached range over the first operation we are about to +* run. We include zero and collapse here because they both start with a +* hole punch over the target range. Insert and collapse both invalidate +* the broader range affected by the shift in xfs_prepare_shift(). +*/ +if (mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_ZERO_RANGE | +FALLOC_FL_COLLAPSE_RANGE )) { + error =xfs_flush_unmap_range (ip, offset, len); +if (error) + goto out_unlock; + } +if (mode &FALLOC_FL_PUNCH_HOLE ) { error =xfs_free_file_space (ip, offset, len);if (error)fs/xfs/xfs_ioctl.c +1
-0@@ -623,6 +623,7 @@ xfs_ioc_space( error =xfs_break_layouts (inode, &iolock,BREAK_UNMAP );if (error) goto out_unlock; +inode_dio_wait (inode);switch (bf->l_whence) {case 0 :/*SEEK_SET*/ Signature · 1 line
2.24.0.rc0
Looks good,
Btw, I've been pondering multiple times if we can kill off i_dio_count again, at least for iomap users. I've added in the request of Thomas how want to kill non-owner rw_semaphore unlocks. But it turns out those were needed in other placeѕ and have been added back at least partially. I'll try to just use those again when I find some time, which should simplify a lot of the mess we around waiting for direct I/O.
Quoted 10 lines · Christoph Hellwig
Looks good,
Reviewed-by Christoph Hellwig <[email protected]>Btw, I've been pondering multiple times if we can kill off
i_dio_countagain, at least for iomap users. I've added in the request of Thomas how want to kill non-ownerrw_semaphoreunlocks. But it turns out those were needed in other placeѕ and have been added back at least partially. I'll try to just use those again when I find some time, which should simplify a lot of the mess we around waiting for direct I/O.
I kinda planned to kill off inode_dio_wait() for XFS via range locks. i.e. hold the range lock all the way to DIO completion, then release it there. This allows things like fallocate, truncate, EOF extension, etc to run concurrently with all IO and not require serialisation of the IO in progress to perform extent and size modification operations.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hmm, do you think this is a reasonable backport of this patch for pre-5.5?
--D
---------------------
From: Dave Chinner <[email protected]>
Subject: [PATCH] xfs: properly serialise fallocate against AIO+DIO
AIO+DIO can extend the file size on IO completion, and it holds no inode locks while the IO is in flight. Therefore, a race condition exists in file size updates if we do something like this:
aio-thread fallocate-thread
lock inode
submit IO beyond inode->i_size
unlock inode
.....
lock inode break layouts if (off + len > inode->i_size) new_size = off + len ..... inode_dio_wait() <blocks>
.....
completesinode->i_size updatedinode_dio_done()
....
<wakes> <does stuff no long beyond EOF> if (new_size) xfs_vn_setattr(inode, new_size)
Yup, that attempt to extend the file size in the fallocate code turns into a truncate - it removes the whatever the aio write allocated and put to disk, and reduced the inode size back down to where the fallocate operation ends.
Fundamentally, xfs_file_fallocate() not compatible with racing AIO+DIO completions, so we need to move the inode_dio_wait() call up to where the lock the inode and break the layouts.
Secondly, storing the inode size and then using it unchecked without holding the ILOCK is not safe; we can only do such a thing if we've locked out and drained all IO and other modification operations, which we don't do initially in xfs_file_fallocate.
It should be noted that some of the fallocate operations are compound operations - they are made up of multiple manipulations that may zero data, and so we may need to flush and invalidate the file multiple times during an operation. However, we only need to lock out IO and other space manipulation operations once, as that lockout is maintained until the entire fallocate operation has been completed.
--- fs/xfs/xfs_bmap_util.c | 8 +------- fs/xfs/xfs_file.c | 23 +++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 25 +++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 7 deletions(-)
fs/xfs/xfs_bmap_util.c +1 -7
@@ -929,6 +929,7 @@ xfs_unmap_extent( goto out_unlock; } +/* Caller must first wait for the completion of any pending DIOs if required. */ int xfs_flush_unmap_range( struct xfs_inode *ip,
@@ -940,9 +941,6 @@ xfs_flush_unmap_range(xfs_off_t rounding, start, end;int error; -/* wait for the completion of any pending DIOs */ -inode_dio_wait (inode); - rounding =max_t (xfs_off_t,1 << mp->m_sb.sb_blocklog,PAGE_SIZE ); start =round_down (offset, rounding); end =round_up (offset + len, rounding) -1 ;
@@ -974,10 +972,6 @@ xfs_free_file_space(if (len <=0 )/* if nothing being freed */ return 0 ; - error =xfs_flush_unmap_range (ip, offset, len); -if (error) -return error; - startoffset_fsb =XFS_B_TO_FSB (mp, offset); endoffset_fsb =XFS_B_TO_FSBT (mp, offset + len);
fs/xfs/xfs_file.c +23 -0
@@ -818,6 +818,29 @@ xfs_file_fallocate(if (error) goto out_unlock; +/* +* Must wait for all AIO to complete before we continue as AIO can +* change the file size on completion without holding any locks we +* currently hold. We must do this first because AIO can update both +* the on disk and in memory inode sizes, and the operations that follow +* require the in-memory size to be fully up-to-date. +*/ +inode_dio_wait (inode); + +/* +* Now that AIO and DIO has drained we can flush and (if necessary) +* invalidate the cached range over the first operation we are about to +* run. We include zero and collapse here because they both start with a +* hole punch over the target range. Insert and collapse both invalidate +* the broader range affected by the shift in xfs_prepare_shift(). +*/ +if (mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_ZERO_RANGE | +FALLOC_FL_COLLAPSE_RANGE )) { + error =xfs_flush_unmap_range (ip, offset, len); +if (error) + goto out_unlock; + } +if (mode &FALLOC_FL_PUNCH_HOLE ) { error =xfs_free_file_space (ip, offset, len);if (error)
fs/xfs/xfs_ioctl.c +25 -0
@@ -665,6 +665,31 @@ xfs_ioc_space( goto out_unlock; } +/* +* Must wait for all AIO to complete before we continue as AIO can +* change the file size on completion without holding any locks we +* currently hold. We must do this first because AIO can update both +* the on disk and in memory inode sizes, and the operations that follow +* require the in-memory size to be fully up-to-date. +*/ +inode_dio_wait (inode); + +/* +* Now that AIO and DIO has drained we can flush and (if necessary) +* invalidate the cached range over the first operation we are about to +* run. We include zero range here because it starts with a hole punch +* over the target range. +*/ +switch (cmd) { +case XFS_IOC_ZERO_RANGE : +case XFS_IOC_UNRESVSP : +case XFS_IOC_UNRESVSP64 : + error =xfs_flush_unmap_range (ip, bf->l_start, bf->l_len); +if (error) + goto out_unlock; +break ; + } +switch (cmd) {case XFS_IOC_ZERO_RANGE : flags |=XFS_PREALLOC_SET ;
Quoted 2 lines · Darrick J. Wong
Hmm, do you think this is a reasonable backport of this patch for pre-5.5?
Seems reasonable to me.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]