xfs: don't flush the entire filesystem when a buffered write runs out of space
PATCH 1/1 xfs: don't flush the entire filesystem when a buffered write runs out of space
From: Darrick J. Wong <[email protected]>
A customer reported rcu stalls and softlockup warnings on a computer with many CPU cores and many many more IO threads trying to write to a filesystem that is totally out of space. Subsequent analysis pointed to the many many IO threads calling xfs_flush_inodes -> sync_inodes_sb, which causes a lot of wb_writeback_work to be queued. The writeback worker spends so much time trying to wake the many many threads waiting for writeback completion that it trips the softlockup detector, and (in this case) the system automatically reboots.
In addition, they complain that the lengthy xfs_flush_inodes scan traps all of those threads in uninterruptible sleep, which hampers their ability to kill the program or do anything else to escape the situation.
Fix this by replacing the full filesystem flush (which is offloaded to a workqueue which we then have to wait for) with directly flushing the file that we're trying to write.
--- fs/xfs/xfs_file.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-)
fs/xfs/xfs_file.c +11 -18
@@ -651,14 +651,18 @@ xfs_file_buffered_aio_write(/* * If we hit a space limit, try to free up some lingering preallocated -* space before returning an error. In the case of ENOSPC, first try to -* write back all dirty inodes to free up some of the excess reserved -* metadata space. This reduces the chances that the eofblocks scan -* waits on dirty mappings. Since xfs_flush_inodes() is serialized, this -* also behaves as a filter to prevent too many eofblocks scans from -* running at the same time. +* space and delalloc reservations before returning an error. */ -if (ret == -EDQUOT && !enospc) { +if ((ret == -EDQUOT || ret == -ENOSPC ) && !enospc) { +/* +* Flush the current file's dirty data to free up any delalloc +* reservation blocks that might have been reserved for bmbt +* expansion. Ignore the return code because we don't want to +* return EIO for a different write that failed. +*/ +filemap_fdatawrite (mapping); +filemap_fdatawait_keep_errors (mapping); +xfs_iunlock (ip, iolock); enospc =xfs_inode_free_quota_eofblocks (ip);if (enospc)
@@ -667,17 +671,6 @@ xfs_file_buffered_aio_write(if (enospc) goto write_retry; iolock =0 ; - }else if (ret == -ENOSPC && !enospc) { -struct xfs_eofblocks eofb = {0 }; - - enospc =1 ; -xfs_flush_inodes (ip->i_mount); - -xfs_iunlock (ip, iolock); - eofb.eof_flags =XFS_EOF_FLAGS_SYNC ; -xfs_icache_free_eofblocks (ip->i_mount, &eofb); -xfs_icache_free_cowblocks (ip->i_mount, &eofb); - goto write_retry; } current->backing_dev_info =NULL ;
Quoted 10 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
A customer reported rcu stalls and softlockup warnings on a computer with many CPU cores and many many more IO threads trying to write to a filesystem that is totally out of space. Subsequent analysis pointed to the many many IO threads calling
xfs_flush_inodes->sync_inodes_sb, which causes a lot ofwb_writeback_workto be queued. The writeback worker spends so much time trying to wake the many many threads waiting for writeback completion that it trips the softlockup detector, and (in this case) the system automatically reboots.
That doesn't sound right. Each writeback work that is queued via sync_inodes_sb should only have a single process waiting on it's completion. And how many threads do you actually have to need to wake up for it to trigger a 10s soft-lockup timeout?
More detail, please?
Quoted 7 lines
In addition, they complain that the lengthy
xfs_flush_inodesscan traps all of those threads in uninterruptible sleep, which hampers their ability to kill the program or do anything else to escape the situation.Fix this by replacing the full filesystem flush (which is offloaded to a workqueue which we then have to wait for) with directly flushing the file that we're trying to write.
Which does nothing to flush -other- outstanding delalloc reservations and allow the eofblocks/cowblock scan to reclaim unused post-EOF speculative preallocations.
That's the purpose of the xfs_flush_inodes() - without it we can get very premature ENOSPC, especially on small filesystems when writing largish files in the background. So I'm not sure that dropping the sync is a viable solution. It is actually needed.
Perhaps we need to go back to the ancient code thatonly allowed XFS to run a single xfs_flush_inodes() at a time - everything else waited on the single flush to complete, then all returned at the same time...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 12 lines · Dave Chinner
Quoted 10 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
A customer reported rcu stalls and softlockup warnings on a computer with many CPU cores and many many more IO threads trying to write to a filesystem that is totally out of space. Subsequent analysis pointed to the many many IO threads calling
xfs_flush_inodes->sync_inodes_sb, which causes a lot ofwb_writeback_workto be queued. The writeback worker spends so much time trying to wake the many many threads waiting for writeback completion that it trips the softlockup detector, and (in this case) the system automatically reboots.
Quoted 6 lines
That doesn't sound right. Each writeback work that is queued via
sync_inodes_sbshould only have a single process waiting on it's completion. And how many threads do you actually have to need to wake up for it to trigger a 10s soft-lockup timeout?More detail, please?
It's a two socket 64-core system with some sort of rdma/infiniband magic and somewhere between 600-900 processes doing who knows what with the magic. Each of those threads *also* is writing trace data to its own separate trace file (three private log files per process). Hilariously they never check the return code from write() so they keep pounding the system forever.
(I don't know what the rdma/infiniband magic is, they won't tell me.)
When the filesystem fills up all the way (it's a 10G fs with 8,207 blocks free) they keep banging away on it until something finally dies.
I tried writing a dumb fstest to simulate the log writer part, but that never succeeds in triggering the rcu stalls.
If you want the gory dmesg details I can send you some dmesg log.
Quoted 8 lines
Quoted 7 lines
In addition, they complain that the lengthy
xfs_flush_inodesscan traps all of those threads in uninterruptible sleep, which hampers their ability to kill the program or do anything else to escape the situation.Fix this by replacing the full filesystem flush (which is offloaded to a workqueue which we then have to wait for) with directly flushing the file that we're trying to write.
Quoted 8 lines
Which does nothing to flush -other- outstanding delalloc reservations and allow the eofblocks/cowblock scan to reclaim unused post-EOF speculative preallocations.
That's the purpose of the
xfs_flush_inodes()- without it we can get very premature ENOSPC, especially on small filesystems when writing largish files in the background. So I'm not sure that dropping the sync is a viable solution. It is actually needed.
Yeah, I did kinda wonder about that...
Quoted 4 lines
Perhaps we need to go back to the ancient code thatonly allowed XFS to run a single
xfs_flush_inodes()at a time - everything else waited on the single flush to complete, then all returned at the same time...
That might work too. Admittedly it's pretty silly to be running this scan over and over and over considering that there's never going to be any more free space.
--D
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 18 lines · Darrick J. Wong
Quoted 17 lines · Dave Chinner
Quoted 10 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
A customer reported rcu stalls and softlockup warnings on a computer with many CPU cores and many many more IO threads trying to write to a filesystem that is totally out of space. Subsequent analysis pointed to the many many IO threads calling
xfs_flush_inodes->sync_inodes_sb, which causes a lot ofwb_writeback_workto be queued. The writeback worker spends so much time trying to wake the many many threads waiting for writeback completion that it trips the softlockup detector, and (in this case) the system automatically reboots.That doesn't sound right. Each writeback work that is queued via
sync_inodes_sbshould only have a single process waiting on it's completion. And how many threads do you actually have to need to wake up for it to trigger a 10s soft-lockup timeout?
Quoted 8 lines
Quoted 1 line
More detail, please?
It's a two socket 64-core system with some sort of rdma/infiniband magic and somewhere between 600-900 processes doing who knows what with the magic. Each of those threads *also* is writing trace data to its own separate trace file (three private log files per process). Hilariously they never check the return code from
write()so they keep pounding the system forever.
<facepalm>
Ah, another game of ye olde "blame the filesystem because it's the first to complain"...
Quoted 7 lines
(I don't know what the rdma/infiniband magic is, they won't tell me.)
When the filesystem fills up all the way (it's a 10G fs with 8,207 blocks free) they keep banging away on it until something finally dies.
I tried writing a dumb fstest to simulate the log writer part, but that never succeeds in triggering the rcu stalls.
Which means it probably requires a bunch of other RCU magic to be done by other part of the system to trigger it...
Quoted 1 line
If you want the gory dmesg details I can send you some dmesg log.
No need, won't be able to read it anyway because facepalm...
Quoted 19 lines
Quoted 8 lines
Quoted 7 lines
In addition, they complain that the lengthy
xfs_flush_inodesscan traps all of those threads in uninterruptible sleep, which hampers their ability to kill the program or do anything else to escape the situation.Fix this by replacing the full filesystem flush (which is offloaded to a workqueue which we then have to wait for) with directly flushing the file that we're trying to write.
Quoted 8 lines
Which does nothing to flush -other- outstanding delalloc reservations and allow the eofblocks/cowblock scan to reclaim unused post-EOF speculative preallocations.
That's the purpose of the
xfs_flush_inodes()- without it we can get very premature ENOSPC, especially on small filesystems when writing largish files in the background. So I'm not sure that dropping the sync is a viable solution. It is actually needed.Yeah, I did kinda wonder about that...
Quoted 8 lines
Quoted 4 lines
Perhaps we need to go back to the ancient code thatonly allowed XFS to run a single
xfs_flush_inodes()at a time - everything else waited on the single flush to complete, then all returned at the same time...That might work too. Admittedly it's pretty silly to be running this scan over and over and over considering that there's never going to be any more free space.
Actually, what if we just rate limit the calls? Once a second would probably do the trick just fine - after the first few seconds there'll be no space left to reclaim, anyway...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
FYI, independ of the discussion what to flush, using filemap_fdatawait_keep_errors is a very obvious and important bug fix. Can you split that out and apply it with a Fixes tag so that it gets backported ASAP?
Quoted 4 lines · Christoph Hellwig
FYI, independ of the discussion what to flush, using
filemap_fdatawait_keep_errorsis a very obvious and important bug fix. Can you split that out and apply it with a Fixes tag so that it gets backported ASAP?
Sorry, -ENOCOFFEE. We didn't have a previous call that comsumed the errors, so the above doesn't make any sense. Please ignore it.