Fix a couple of random hangs.
cover Fix a couple of random hangs.
The first patch fixes a low memory hang where I/O completion processing triggers memory reclaim which hangs waiting for I/O completion. The trigger is unwritten extent conversion blocking completion of normal writes.
The second fixes a log hang which results from excessive load; we update the push target after we go to sleep so if we are unlucky enough not to have a current push target far enough advanced we'll hang.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/2 XFS: Prevent unwritten extent conversion from blocking I/O completion
Unwritten extent conversion can recurse back into the filesystem due to memory allocation. Memory reclaim requires I/O completions to be processed to allow the callers to make progress. If the I/O completion workqueue thread is doing the recursion, then we have a deadlock situation.
Move unwritten extent completion into it's own workqueue so it doesn't block I/O completions for normal delayed allocation or overwrite data.
--- fs/xfs/linux-2.6/xfs_aops.c | 38 +++++++++++++++++++++----------------- fs/xfs/linux-2.6/xfs_aops.h | 1 + fs/xfs/linux-2.6/xfs_buf.c | 9 +++++++++ 3 files changed, 31 insertions(+), 17 deletions(-)
fs/xfs/linux-2.6/xfs_aops.c +21 -17
@@ -153,23 +153,6 @@ xfs_find_bdev_for_inode( }/* -* Schedule IO completion handling on a xfsdatad if this was -* the final hold on this ioend. If we are asked to wait, -* flush the workqueue. -*/ -STATIC void -xfs_finish_ioend ( -xfs_ioend_t *ioend, -int wait) -{ -if (atomic_dec_and_test (&ioend->io_remaining)) { -queue_work (xfsdatad_workqueue, &ioend->io_work); -if (wait) -flush_workqueue (xfsdatad_workqueue); - } -} - -/* * We're now finished for good with this ioend structure. * Update the page state via the associated buffer_heads, * release holds on the inode and bio, and finally free
@@ -310,6 +293,27 @@ xfs_end_bio_read( }/* +* Schedule IO completion handling on a xfsdatad if this was +* the final hold on this ioend. If we are asked to wait, +* flush the workqueue. +*/ +STATIC void +xfs_finish_ioend ( +xfs_ioend_t *ioend, +int wait) +{ +if (atomic_dec_and_test (&ioend->io_remaining)) { +struct workqueue_struct *wq = xfsdatad_workqueue; +if (ioend->io_work.func == xfs_end_bio_unwritten) + wq = xfsconvertd_workqueue; + +queue_work (wq, &ioend->io_work); +if (wait) +flush_workqueue (wq); + } +} + +/* * Allocate and initialise an IO completion structure. * We need to track unwritten extent write completion here initially. * We'll need to extend this for updating the ondisk inode size later
fs/xfs/linux-2.6/xfs_aops.h +1 -0
@@ -19,6 +19,7 @@#define __XFS_AOPS_H__extern struct workqueue_struct *xfsdatad_workqueue; +extern struct workqueue_struct *xfsconvertd_workqueue;extern mempool_t *xfs_ioend_pool;/*
fs/xfs/linux-2.6/xfs_buf.c +9 -0
@@ -51,6 +51,7 @@ static struct shrinker xfs_buf_shake = {static struct workqueue_struct *xfslogd_workqueue;struct workqueue_struct *xfsdatad_workqueue; +struct workqueue_struct *xfsconvertd_workqueue;#ifdef XFS_BUF_TRACE void
@@ -1775,6 +1776,7 @@ xfs_flush_buftarg(xfs_buf_t *bp, *n;int pincount =0 ; +xfs_buf_runall_queues (xfsconvertd_workqueue);xfs_buf_runall_queues (xfsdatad_workqueue);xfs_buf_runall_queues (xfslogd_workqueue);
@@ -1831,9 +1833,15 @@ xfs_buf_init(void)if (!xfsdatad_workqueue) goto out_destroy_xfslogd_workqueue; + xfsconvertd_workqueue =create_workqueue ("xfsconvertd" ); +if (!xfsconvertd_workqueue) + goto out_destroy_xfsdatad_workqueue; +register_shrinker (&xfs_buf_shake);return 0 ; + out_destroy_xfsdatad_workqueue: +destroy_workqueue (xfsdatad_workqueue); out_destroy_xfslogd_workqueue:destroy_workqueue (xfslogd_workqueue); out_free_buf_zone:
@@ -1849,6 +1857,7 @@ voidxfs_buf_terminate (void) {unregister_shrinker (&xfs_buf_shake); +destroy_workqueue (xfsconvertd_workqueue);destroy_workqueue (xfsdatad_workqueue);destroy_workqueue (xfslogd_workqueue);kmem_zone_destroy (xfs_buf_zone);
Signature · 6 lines
1.6.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/2] XFS: Prevent unwritten extent conversion from blocking I/O completion
Quoted 9 lines · Dave Chinner
Unwritten extent conversion can recurse back into the filesystem due to memory allocation. Memory reclaim requires I/O completions to be processed to allow the callers to make progress. If the I/O completion workqueue thread is doing the recursion, then we have a deadlock situation.
Move unwritten extent completion into it's own workqueue so it doesn't block I/O completions for normal delayed allocation or overwrite data.
Hmm. That was the original reason behind splitting the data from
xfsbufd queue. So maybe the split should be just unwritten vs the rest and three queues?
Btw, do you have a testcase that can reproduce this?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/2] XFS: Prevent unwritten extent conversion from blocking I/O completion
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Dave Chinner
Unwritten extent conversion can recurse back into the filesystem due to memory allocation. Memory reclaim requires I/O completions to be processed to allow the callers to make progress. If the I/O completion workqueue thread is doing the recursion, then we have a deadlock situation.
Move unwritten extent completion into it's own workqueue so it doesn't block I/O completions for normal delayed allocation or overwrite data.
Quoted 5 lines
Hmm. That was the original reason behind splitting the data from
xfsbufd queue. So maybe the split should be just unwritten vs the rest and three queues?Btw, do you have a testcase that can reproduce this?
No, I hit it a couple of times running xfsqa on a low memory UML image - 256MB of RAM, IIRC - during one of the fstress tests. I got enough information to determine this was the problem and it hasn't showed up since. I think someone also posted a lockdep trace on LKML a couple of months back as well...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/2 XFS: Inform the xfsaild of the push target before sleeping
In reply to Dave ChinnerWhen trying to reserve log space, we find the amount of space we need, then go to sleep waiting for space. When we are woken, we try to push the tail of the log forward to make sure we have space available.
Unfortunately, this means that if there is not space available, and everyone who needs space goes to sleep there is no-one left to push the tail of the log to make space available. Once we have a thread waiting for space to become available, the others queue up behind it in a FIFO, and none of them push the tail of the log.
This can result in everyone going to sleep in xlog_grant_log_space() if the first sleeper races with the last I/O that moves the tail of the log forward. With no further I/O tomove the tail of the log, there is nothing to wake the sleepers and hence all transactions just stop.
Fix this by making sure the xfsaild will create enough space for the transaction that is about to sleep by moving the push target far enough forwards to ensure that that the curent proceeees will have enough space available when it is woken. That is, we push the AIL before we go to sleep.
Because we've inserted the log ticket into the queue before we've pushed and gone to sleep, subsequent transactions will wait behind this one. Hence we are guaranteed to have space available when we are woken.
--- fs/xfs/xfs_log.c | 37 +++++++++++++++++++------------------ 1 files changed, 19 insertions(+), 18 deletions(-)
fs/xfs/xfs_log.c +19 -18
@@ -2573,18 +2573,19 @@ redo:xlog_ins_ticketq (&log->l_reserve_headq, tic);xlog_trace_loggrant (log, tic,"xlog_grant_log_space: sleep 2" ); +spin_unlock (&log->l_grant_lock); +xlog_grant_push_ail (log->l_mp, need_bytes); +spin_lock (&log->l_grant_lock); +XFS_STATS_INC (xs_sleep_logspace);sv_wait (&tic->t_wait,PINOD |PLTWAIT , &log->l_grant_lock, s); -if (XLOG_FORCED_SHUTDOWN (log)) { -spin_lock (&log->l_grant_lock); +spin_lock (&log->l_grant_lock); +if (XLOG_FORCED_SHUTDOWN (log)) goto error_return; - }xlog_trace_loggrant (log, tic,"xlog_grant_log_space: wake 2" ); -xlog_grant_push_ail (log->l_mp, need_bytes); -spin_lock (&log->l_grant_lock); goto redo; }else if (tic->t_flags &XLOG_TIC_IN_Q )xlog_del_ticketq (&log->l_reserve_headq, tic);
@@ -2663,7 +2664,7 @@ xlog_regrant_write_log_space(xlog_t *log,* for more free space, otherwise try to get some space for * this transaction. */ - + need_bytes = tic->t_unit_res;if ((ntic = log->l_write_headq)) { free_bytes =xlog_space_left (log, log->l_grant_write_cycle, log->l_grant_write_bytes);
@@ -2683,26 +2684,25 @@ xlog_regrant_write_log_space(xlog_t *log,xlog_trace_loggrant (log, tic,"xlog_regrant_write_log_space: sleep 1" ); +spin_unlock (&log->l_grant_lock); +xlog_grant_push_ail (log->l_mp, need_bytes); +spin_lock (&log->l_grant_lock); +XFS_STATS_INC (xs_sleep_logspace);sv_wait (&tic->t_wait,PINOD |PLTWAIT , &log->l_grant_lock, s);/* If we're shutting down, this tic is already * off the queue */ -if (XLOG_FORCED_SHUTDOWN (log)) { -spin_lock (&log->l_grant_lock); +spin_lock (&log->l_grant_lock); +if (XLOG_FORCED_SHUTDOWN (log)) goto error_return; - }xlog_trace_loggrant (log, tic,"xlog_regrant_write_log_space: wake 1" ); -xlog_grant_push_ail (log->l_mp, tic->t_unit_res); -spin_lock (&log->l_grant_lock); } } - need_bytes = tic->t_unit_res; - redo:if (XLOG_FORCED_SHUTDOWN (log)) goto error_return;
@@ -2712,19 +2712,20 @@ redo:if (free_bytes < need_bytes) {if ((tic->t_flags &XLOG_TIC_IN_Q ) ==0 )xlog_ins_ticketq (&log->l_write_headq, tic); +spin_unlock (&log->l_grant_lock); +xlog_grant_push_ail (log->l_mp, need_bytes); +spin_lock (&log->l_grant_lock); +XFS_STATS_INC (xs_sleep_logspace);sv_wait (&tic->t_wait,PINOD |PLTWAIT , &log->l_grant_lock, s);/* If we're shutting down, this tic is already off the queue */ -if (XLOG_FORCED_SHUTDOWN (log)) { -spin_lock (&log->l_grant_lock); +spin_lock (&log->l_grant_lock); +if (XLOG_FORCED_SHUTDOWN (log)) goto error_return; - }xlog_trace_loggrant (log, tic,"xlog_regrant_write_log_space: wake 2" ); -xlog_grant_push_ail (log->l_mp, need_bytes); -spin_lock (&log->l_grant_lock); goto redo; }else if (tic->t_flags &XLOG_TIC_IN_Q )xlog_del_ticketq (&log->l_write_headq, tic);
Signature · 6 lines
1.6.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/2] XFS: Inform the xfsaild of the push target before sleeping
This looks good to me.
But looking at that area I must say I'm not happy with it in general. There are tons of useless roundtrips l_grant_lock because we drop it just before calling xlog_grant_push_ail which needs it for most of it's operation, and because we acquire it just before sv_wait just to drop it.
And the duplication between xlog_grant_log_spacexlog_regrant_write_log_space is quite ugly, too.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/2] XFS: Inform the xfsaild of the push target before sleeping
Quoted 5 lines · Christoph Hellwig
This looks good to me.
Reviewed-by Christoph Hellwig <[email protected]>
Quoted 8 lines
But looking at that area I must say I'm not happy with it in general. There are tons of useless roundtrips
l_grant_lockbecause we drop it just before callingxlog_grant_push_ailwhich needs it for most of it's operation, and because we acquire it just beforesv_waitjust to drop it.And the duplication between
xlog_grant_log_spacexlog_regrant_write_log_spaceis quite ugly, too.
I totally agree, but I don't feel like trying to rewrite this code right now. Maybe some rainy day....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs