xfs: use dedicated log worker wq to avoid deadlock with cil wq
PATCH 1/1 xfs: use dedicated log worker wq to avoid deadlock with cil wq
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit 5889608df ("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.
Since xfs_log_worker() can invoke a log flush, a deadlock is possible between the xfs-log and xfs-cil workqueues. Consider the following codepath from xfs_log_worker():
xfs_log_worker()
xfs_log_force()
_xfs_log_force()
xlog_cil_force()
xlog_cil_force_lsn()
xlog_cil_push_now()
flush_work()The above is in xfs-log wq context and blocked waiting on the completion of an xfs-cil work item. Concurrently, the cil push in progress can end up blocked here:
xlog_cil_push_work()
xlog_cil_push()
xlog_write()
xlog_state_get_iclog_space()
xlog_wait(&log->l_flush_wait, ...)The above is in xfs-cil context waiting on log buffer I/O completion, which executes in xfs-log wq context. In this scenario both workqueues are deadlocked waiting on eachother.
Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit 5889608df.
---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Given that, one alternative approach I considered here is to use the rescuer thread check brought up in the other thread[1] regarding WQ_MEM_RECLAIM. I opted for this approach because I think the former is brittle in that it assumes context within the xfs_log_worker() code, adds non-determinism to the overall job of the background worker, and also doesn't address the fact that it's probably unnecessary to run this code at high priority.
Brian
[1] http://www.spinics.net/lists/linux-xfs/msg04673.html
fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-)
fs/xfs/xfs_log.c +1 -1
@@ -1293,7 +1293,7 @@ voidxfs_log_work_queue ( struct xfs_mount *mp) { -queue_delayed_work (mp->m_log_workqueue, &mp->m_log->l_work, +queue_delayed_work (mp->m_sync_workqueue, &mp->m_log->l_work,msecs_to_jiffies (xfs_syncd_centisecs *10 )); }
fs/xfs/xfs_mount.h +1 -0
@@ -183,6 +183,7 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; +struct workqueue_struct *m_sync_workqueue;/* * Generation of the filesysyem layout. This is incremented by each
fs/xfs/xfs_super.c +8 -0
@@ -877,8 +877,15 @@ xfs_init_mount_workqueues(if (!mp->m_eofblocks_workqueue) goto out_destroy_log; + mp->m_sync_workqueue =alloc_workqueue ("xfs-sync/%s" ,WQ_FREEZABLE ,0 , + mp->m_fsname); +if (!mp->m_sync_workqueue) + goto out_destroy_eofb; +return 0 ; +out_destroy_eofb: +destroy_workqueue (mp->m_eofblocks_workqueue); out_destroy_log:destroy_workqueue (mp->m_log_workqueue); out_destroy_reclaim:
@@ -899,6 +906,7 @@ STATIC voidxfs_destroy_mount_workqueues ( struct xfs_mount *mp) { +destroy_workqueue (mp->m_sync_workqueue);destroy_workqueue (mp->m_eofblocks_workqueue);destroy_workqueue (mp->m_log_workqueue);destroy_workqueue (mp->m_reclaim_workqueue);
Signature · 1 line
2.7.4
Quoted 64 lines · Brian Foster
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit
5889608df("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.Since
xfs_log_worker()can invoke a log flush, a deadlock is possible between the xfs-log and xfs-cil workqueues. Consider the following codepath fromxfs_log_worker():
xfs_log_worker()
xfs_log_force()_xfs_log_force() xlog_cil_force() xlog_cil_force_lsn() xlog_cil_push_now() flush_work()The above is in xfs-log wq context and blocked waiting on the completion of an xfs-cil work item. Concurrently, the cil push in progress can end up blocked here:
xlog_cil_push_work()
xlog_cil_push()xlog_write() xlog_state_get_iclog_space() xlog_wait(&log->l_flush_wait, ...)The above is in xfs-cil context waiting on log buffer I/O completion, which executes in xfs-log wq context. In this scenario both workqueues are deadlocked waiting on eachother.
Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit
5889608df.Diagnosed-by David Jeffery <[email protected]>Signed-off-by Brian Foster <[email protected]>---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Given that, one alternative approach I considered here is to use the rescuer thread check brought up in the other thread[1] regarding
WQ_MEM_RECLAIM. I opted for this approach because I think the former is brittle in that it assumes context within thexfs_log_worker()code, adds non-determinism to the overall job of the background worker, and also doesn't address the fact that it's probably unnecessary to run this code at high priority.Brian
[1] http://www.spinics.net/lists/linux-xfs/msg04673.html
fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-)fs/xfs/xfs_log.c +0
-0
fs/xfs/xfs_log.c void · line 1297
Quoted 6 lines
fs/xfs/xfs_log.c +1
-1@@ -1293,7 +1293,7 @@ voidxfs_log_work_queue ( struct xfs_mount *mp) { -queue_delayed_work (mp->m_log_workqueue, &mp->m_log->l_work, +queue_delayed_work (mp->m_sync_workqueue, &mp->m_log->l_work,msecs_to_jiffies (xfs_syncd_centisecs *10 ));
I /think/ this looks ok? But let me run it through xfstests before I commit to anything. :)
--D
Quoted 49 lines
fs/xfs/xfs_log.c +0
-0}fs/xfs/xfs_mount.h +1
-0@@ -183,6 +183,7 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; +struct workqueue_struct *m_sync_workqueue;/* * Generation of the filesysyem layout. This is incremented by each fs/xfs/xfs_super.c +8
-0@@ -877,8 +877,15 @@ xfs_init_mount_workqueues(if (!mp->m_eofblocks_workqueue) goto out_destroy_log; + mp->m_sync_workqueue =alloc_workqueue ("xfs-sync/%s" ,WQ_FREEZABLE ,0 , + mp->m_fsname); +if (!mp->m_sync_workqueue) + goto out_destroy_eofb; +return 0 ; +out_destroy_eofb: +destroy_workqueue (mp->m_eofblocks_workqueue); out_destroy_log:destroy_workqueue (mp->m_log_workqueue); out_destroy_reclaim:@@ -899,6 +906,7 @@ STATIC voidxfs_destroy_mount_workqueues ( struct xfs_mount *mp) { +destroy_workqueue (mp->m_sync_workqueue);destroy_workqueue (mp->m_eofblocks_workqueue);destroy_workqueue (mp->m_log_workqueue);destroy_workqueue (mp->m_reclaim_workqueue);Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Darrick J. Wong
Quoted 4 lines · Brian Foster
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit
5889608df("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.
That went in on v3.8.
<-- snip -->
Quoted 6 lines
Quoted 6 lines
In this scenario
both workqueues are deadlocked waiting on eachother.Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit
5889608df.
Quoted 8 lines
Quoted 8 lines
Diagnosed-by David Jeffery <[email protected]>Signed-off-by Brian Foster <[email protected]>---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Still, a dead lock is no bueno. No happy user can exist with a deadlock.
Quoted 2 lines
I /think/ this looks ok? But let me run it through xfstests before I commit to anything. :)
Provided that goes smooth -- this seems like a stable candidate to me. Otherwise we're going to have to carry it anyway on distributions.
Luis
Quoted 60 lines · Darrick J. Wong
Quoted 59 lines · Brian Foster
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit
5889608df("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.Since
xfs_log_worker()can invoke a log flush, a deadlock is possible between the xfs-log and xfs-cil workqueues. Consider the following codepath fromxfs_log_worker():
xfs_log_worker()
xfs_log_force()_xfs_log_force() xlog_cil_force() xlog_cil_force_lsn() xlog_cil_push_now() flush_work()The above is in xfs-log wq context and blocked waiting on the completion of an xfs-cil work item. Concurrently, the cil push in progress can end up blocked here:
xlog_cil_push_work()
xlog_cil_push()xlog_write() xlog_state_get_iclog_space() xlog_wait(&log->l_flush_wait, ...)The above is in xfs-cil context waiting on log buffer I/O completion, which executes in xfs-log wq context. In this scenario both workqueues are deadlocked waiting on eachother.
Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit
5889608df.Diagnosed-by David Jeffery <[email protected]>Signed-off-by Brian Foster <[email protected]>---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Given that, one alternative approach I considered here is to use the rescuer thread check brought up in the other thread[1] regarding
WQ_MEM_RECLAIM. I opted for this approach because I think the former is brittle in that it assumes context within thexfs_log_worker()code, adds non-determinism to the overall job of the background worker, and also doesn't address the fact that it's probably unnecessary to run this code at high priority.Brian
[1] http://www.spinics.net/lists/linux-xfs/msg04673.html
fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-)
Quoted 15 lines
fs/xfs/xfs_log.c void · line 1297
Quoted 11 lines
fs/xfs/xfs_log.c +1
-1@@ -1293,7 +1293,7 @@ voidxfs_log_work_queue ( struct xfs_mount *mp) { -queue_delayed_work (mp->m_log_workqueue, &mp->m_log->l_work, +queue_delayed_work (mp->m_sync_workqueue, &mp->m_log->l_work,msecs_to_jiffies (xfs_syncd_centisecs *10 ));I /think/ this looks ok? But let me run it through xfstests before I commit to anything. :)
Ping... just checking in on this one. Is this in the pipe..? Thanks.
Brian
Quoted 55 lines
--D
Quoted 49 lines
fs/xfs/xfs_log.c +0
-0}fs/xfs/xfs_mount.h +1
-0@@ -183,6 +183,7 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; +struct workqueue_struct *m_sync_workqueue;/* * Generation of the filesysyem layout. This is incremented by each fs/xfs/xfs_super.c +8
-0@@ -877,8 +877,15 @@ xfs_init_mount_workqueues(if (!mp->m_eofblocks_workqueue) goto out_destroy_log; + mp->m_sync_workqueue =alloc_workqueue ("xfs-sync/%s" ,WQ_FREEZABLE ,0 , + mp->m_fsname); +if (!mp->m_sync_workqueue) + goto out_destroy_eofb; +return 0 ; +out_destroy_eofb: +destroy_workqueue (mp->m_eofblocks_workqueue); out_destroy_log:destroy_workqueue (mp->m_log_workqueue); out_destroy_reclaim:@@ -899,6 +906,7 @@ STATIC voidxfs_destroy_mount_workqueues ( struct xfs_mount *mp) { +destroy_workqueue (mp->m_sync_workqueue);destroy_workqueue (mp->m_eofblocks_workqueue);destroy_workqueue (mp->m_log_workqueue);destroy_workqueue (mp->m_reclaim_workqueue);Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 77 lines · Brian Foster
Quoted 60 lines · Darrick J. Wong
Quoted 59 lines · Brian Foster
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit
5889608df("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.Since
xfs_log_worker()can invoke a log flush, a deadlock is possible between the xfs-log and xfs-cil workqueues. Consider the following codepath fromxfs_log_worker():
xfs_log_worker()
xfs_log_force()_xfs_log_force() xlog_cil_force() xlog_cil_force_lsn() xlog_cil_push_now() flush_work()The above is in xfs-log wq context and blocked waiting on the completion of an xfs-cil work item. Concurrently, the cil push in progress can end up blocked here:
xlog_cil_push_work()
xlog_cil_push()xlog_write() xlog_state_get_iclog_space() xlog_wait(&log->l_flush_wait, ...)The above is in xfs-cil context waiting on log buffer I/O completion, which executes in xfs-log wq context. In this scenario both workqueues are deadlocked waiting on eachother.
Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit
5889608df.Diagnosed-by David Jeffery <[email protected]>Signed-off-by Brian Foster <[email protected]>---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Given that, one alternative approach I considered here is to use the rescuer thread check brought up in the other thread[1] regarding
WQ_MEM_RECLAIM. I opted for this approach because I think the former is brittle in that it assumes context within thexfs_log_worker()code, adds non-determinism to the overall job of the background worker, and also doesn't address the fact that it's probably unnecessary to run this code at high priority.Brian
[1] http://www.spinics.net/lists/linux-xfs/msg04673.html
fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-)Quoted 15 lines
fs/xfs/xfs_log.c void · line 1297
Quoted 11 lines
fs/xfs/xfs_log.c +1
-1@@ -1293,7 +1293,7 @@ voidxfs_log_work_queue ( struct xfs_mount *mp) { -queue_delayed_work (mp->m_log_workqueue, &mp->m_log->l_work, +queue_delayed_work (mp->m_sync_workqueue, &mp->m_log->l_work,msecs_to_jiffies (xfs_syncd_centisecs *10 ));I /think/ this looks ok? But let me run it through xfstests before I commit to anything. :)
Quoted 1 line
Ping... just checking in on this one. Is this in the pipe..? Thanks.
Tested ok; do you want me to put this in for 4.11 or save it for 4.12?
--D
Quoted 58 lines
Brian
Quoted 55 lines
--D
Quoted 49 lines
fs/xfs/xfs_log.c +0
-0}fs/xfs/xfs_mount.h +1
-0@@ -183,6 +183,7 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; +struct workqueue_struct *m_sync_workqueue;/* * Generation of the filesysyem layout. This is incremented by each fs/xfs/xfs_super.c +8
-0@@ -877,8 +877,15 @@ xfs_init_mount_workqueues(if (!mp->m_eofblocks_workqueue) goto out_destroy_log; + mp->m_sync_workqueue =alloc_workqueue ("xfs-sync/%s" ,WQ_FREEZABLE ,0 , + mp->m_fsname); +if (!mp->m_sync_workqueue) + goto out_destroy_eofb; +return 0 ; +out_destroy_eofb: +destroy_workqueue (mp->m_eofblocks_workqueue); out_destroy_log:destroy_workqueue (mp->m_log_workqueue); out_destroy_reclaim:@@ -899,6 +906,7 @@ STATIC voidxfs_destroy_mount_workqueues ( struct xfs_mount *mp) { +destroy_workqueue (mp->m_sync_workqueue);destroy_workqueue (mp->m_eofblocks_workqueue);destroy_workqueue (mp->m_log_workqueue);destroy_workqueue (mp->m_reclaim_workqueue);Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 78 lines · Darrick J. Wong
Quoted 77 lines · Brian Foster
Quoted 75 lines · Darrick J. Wong
Quoted 64 lines · Brian Foster
The log covering background task used to be part of the xfssyncd workqueue. That workqueue was removed as of commit
5889608df("xfs:
syncd workqueue is no more") and the associated work item scheduled to the xfs-log wq. The latter is used for log buffer I/O completion.Since
xfs_log_worker()can invoke a log flush, a deadlock is possible between the xfs-log and xfs-cil workqueues. Consider the following codepath fromxfs_log_worker():
xfs_log_worker()
xfs_log_force()_xfs_log_force() xlog_cil_force() xlog_cil_force_lsn() xlog_cil_push_now() flush_work()The above is in xfs-log wq context and blocked waiting on the completion of an xfs-cil work item. Concurrently, the cil push in progress can end up blocked here:
xlog_cil_push_work()
xlog_cil_push()xlog_write() xlog_state_get_iclog_space() xlog_wait(&log->l_flush_wait, ...)The above is in xfs-cil context waiting on log buffer I/O completion, which executes in xfs-log wq context. In this scenario both workqueues are deadlocked waiting on eachother.
Add a new workqueue specifically for the high level log covering and ail pushing worker, as was the case prior to commit
5889608df.Diagnosed-by David Jeffery <[email protected]>Signed-off-by Brian Foster <[email protected]>---
Note that this seems difficult to reproduce in practice because I believe it also relies on memory pressure. Otherwise, the xfs-log wq rescuer thread may be available to unwind the deadlock.
Given that, one alternative approach I considered here is to use the rescuer thread check brought up in the other thread[1] regarding
WQ_MEM_RECLAIM. I opted for this approach because I think the former is brittle in that it assumes context within thexfs_log_worker()code, adds non-determinism to the overall job of the background worker, and also doesn't address the fact that it's probably unnecessary to run this code at high priority.Brian
[1] http://www.spinics.net/lists/linux-xfs/msg04673.html
fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-)fs/xfs/xfs_log.c +0
-0fs/xfs/xfs_log.c void · line 1297
Quoted 6 lines
fs/xfs/xfs_log.c +1
-1@@ -1293,7 +1293,7 @@ voidxfs_log_work_queue ( struct xfs_mount *mp) { -queue_delayed_work (mp->m_log_workqueue, &mp->m_log->l_work, +queue_delayed_work (mp->m_sync_workqueue, &mp->m_log->l_work,msecs_to_jiffies (xfs_syncd_centisecs *10 ));I /think/ this looks ok? But let me run it through xfstests before I commit to anything. :)
Quoted 4 lines
Quoted 1 line
Ping... just checking in on this one. Is this in the pipe..? Thanks.
Tested ok; do you want me to put this in for 4.11 or save it for 4.12?
Thanks. No preference, though I guess that means 4.12 is probably appropriate. Does that mean this would fall into for-next soon-ish? I'm basically just looking to land it somewhere upstream so we have a commit id and we can backport it downstream for customers that hit the problem.
Brian
Quoted 64 lines
--D
Quoted 58 lines
Brian
Quoted 55 lines
--D
Quoted 49 lines
fs/xfs/xfs_log.c +0
-0}fs/xfs/xfs_mount.h +1
-0@@ -183,6 +183,7 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; +struct workqueue_struct *m_sync_workqueue;/* * Generation of the filesysyem layout. This is incremented by each fs/xfs/xfs_super.c +8
-0@@ -877,8 +877,15 @@ xfs_init_mount_workqueues(if (!mp->m_eofblocks_workqueue) goto out_destroy_log; + mp->m_sync_workqueue =alloc_workqueue ("xfs-sync/%s" ,WQ_FREEZABLE ,0 , + mp->m_fsname); +if (!mp->m_sync_workqueue) + goto out_destroy_eofb; +return 0 ; +out_destroy_eofb: +destroy_workqueue (mp->m_eofblocks_workqueue); out_destroy_log:destroy_workqueue (mp->m_log_workqueue); out_destroy_reclaim:@@ -899,6 +906,7 @@ STATIC voidxfs_destroy_mount_workqueues ( struct xfs_mount *mp) { +destroy_workqueue (mp->m_sync_workqueue);destroy_workqueue (mp->m_eofblocks_workqueue);destroy_workqueue (mp->m_log_workqueue);destroy_workqueue (mp->m_reclaim_workqueue);Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html