xfs: split up xfslogd global workqueue

PATCH In review Brian Foster 2 participants 5 messages Nov 13, 2014 – Nov 28, 2014 XFS filesystem

Share & tools

Brian Foster

author

2014-11-13 19:24 UTC #0 lore raw

v3 cover xfs: split up xfslogd global workqueue

I've broken this up into two patches because the bugfix is easy (patch
1) and I don't want to rework it for the purpose of refactoring usage of the queue. Patch 1 is a straightforward, tested and backportable fix for a specific problem and it's no more difficult to refactor queue usage from a per-mount queue than from a single global queue.

The metadata/log buffer I/O separation is not as straightforward as defining separate queues. We need to also define a mechanism to split the completions across multiple queues within the buffer code. The ioend code does this for the data/unwritten queues via the io type. There are a few different ways we can go from here with regard to buffers. For example:

- Define per-buftarg workqueues and configure them appropriately for data and log devices.
- Split buf completion processing based on a buffer flag (or some other identification).
- Continue using a single queue (per-mount) for metadata/log completions.

The first approach introduces variance in that the separation only occurs with separate data and log devices. This is not ideal and doesn't seem worthwhile to me. It doesn't change anything in the common case and introduces a new regression vector for the uncommon case.

The second approach is implemented by the 2/2 RFC of this series using the XBF_SYNCIO flag simply because it only appears to be used by log buffers at the moment. If we take this approach, should we define something like an XBF_LOGIO or XBF_HIGHPRI flag?

The third approach simply involves dropping patch 2/2 and using the per-mount queue as we use the global queue today.

Brian

v3:
- Split bug fix and buf/log I/O split into separate patches.
- Rename xfslogd workqueue to xfs-buf.
v2: http://oss.sgi.com/archives/xfs/2014-11/msg00164.html
- Rename xfslogd workqueue to xfs-iodone.
v1: http://oss.sgi.com/archives/xfs/2014-10/msg00539.html

Brian Foster (2):
xfs: replace global xfslogd wq with per-mount wq
xfs: split metadata and log buffer completion to separate workqueues

 fs/xfs/xfs_buf.c   | 19 ++++++++-----------
 fs/xfs/xfs_mount.h |  1 +
 fs/xfs/xfs_super.c | 12 ++++++++++--
 3 files changed, 19 insertions(+), 13 deletions(-)

Signature · 6 lines

1.8.3.1

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Brian Foster

author

2014-11-13 19:24 UTC #1 lore raw

PATCH v3 1/2 xfs: replace global xfslogd wq with per-mount wq

The xfslogd workqueue is a global, single-job workqueue for buffer ioend processing. This means we allow for a single work item at a time for all possible XFS mounts on a system. fsstress testing in loopback XFS over XFS configurations has reproduced xfslogd deadlocks due to the single threaded nature of the queue and dependencies introduced between the separate XFS instances by online discard (-o discard).

Discard over a loopback device converts the discard request to a hole punch (fallocate) on the underlying file. Online discard requests are issued synchronously and from xfslogd context in XFS, hence the xfslogd workqueue is blocked in the upper fs waiting on a hole punch request to be servied in the lower fs. If the lower fs issues I/O that depends on xfslogd to complete, both filesystems end up hung indefinitely. This is reproduced reliabily by generic/013 on XFS->loop->XFS test devices with the '-o discard' mount option.

Further, docker implementations appear to use this kind of configuration for container instance filesystems by default (container fs->dm-> loop->base fs) and therefore are subject to this deadlock when running on XFS.

Replace the global xfslogd workqueue with a per-mount variant. This guarantees each mount access to a single worker and prevents deadlocks due to inter-fs dependencies introduced by discard. Since the queue is only responsible for buffer iodone processing at this point in time, rename xfslogd to xfs-buf.

Signed-off-by Brian Foster <[email protected]>

---
 fs/xfs/xfs_buf.c   | 12 +-----------
 fs/xfs/xfs_mount.h |  1 +
 fs/xfs/xfs_super.c | 11 ++++++++++-
 3 files changed, 12 insertions(+), 12 deletions(-)

fs/xfs/xfs_buf.c +1 ~~-11~~

index 24b4ebe..c06d790 100644
@@ -44,8 +44,6 @@
 
 static kmem_zone_t *xfs_buf_zone;
 
-static struct workqueue_struct *xfslogd_workqueue;
-
 #ifdef XFS_BUF_LOCK_TRACKING
 # define XB_SET_OWNER(bp)	((bp)->b_last_holder = current->pid)
 # define XB_CLEAR_OWNER(bp)	((bp)->b_last_holder = -1)

@@ -1053,7 +1051,7 @@ xfs_buf_ioend_async(
 	struct xfs_buf	*bp)
 {
 	INIT_WORK(&bp->b_iodone_work, xfs_buf_ioend_work);
-	queue_work(xfslogd_workqueue, &bp->b_iodone_work);
+	queue_work(bp->b_target->bt_mount->m_buf_workqueue, &bp->b_iodone_work);
 }
 
 void

@@ -1882,15 +1880,8 @@ xfs_buf_init(void)
 	if (!xfs_buf_zone)
 		goto out;
 
-	xfslogd_workqueue = alloc_workqueue("xfslogd",
-				WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_FREEZABLE, 1);
-	if (!xfslogd_workqueue)
-		goto out_free_buf_zone;
-
 	return 0;
 
- out_free_buf_zone:
-	kmem_zone_destroy(xfs_buf_zone);
  out:
 	return -ENOMEM;
 }

@@ -1898,6 +1889,5 @@ xfs_buf_init(void)
 void
 xfs_buf_terminate(void)
 {
-	destroy_workqueue(xfslogd_workqueue);
 	kmem_zone_destroy(xfs_buf_zone);
 }

fs/xfs/xfs_mount.h +1 -0

index b0447c8..394bc71 100644
@@ -168,6 +168,7 @@ typedef struct xfs_mount {
 						/* low free space thresholds */
 	struct xfs_kobj		m_kobj;
 
+	struct workqueue_struct *m_buf_workqueue;
 	struct workqueue_struct	*m_data_workqueue;
 	struct workqueue_struct	*m_unwritten_workqueue;
 	struct workqueue_struct	*m_cil_workqueue;

fs/xfs/xfs_super.c +10 -1

index 9f622fe..03e3cc2 100644
@@ -842,10 +842,16 @@ STATIC int
 xfs_init_mount_workqueues(
 	struct xfs_mount	*mp)
 {
+	mp->m_buf_workqueue = alloc_workqueue("xfs-buf/%s",
+			WQ_MEM_RECLAIM|WQ_HIGHPRI|WQ_FREEZABLE, 1,
+			mp->m_fsname);
+	if (!mp->m_buf_workqueue)
+		goto out;
+
 	mp->m_data_workqueue = alloc_workqueue("xfs-data/%s",
 			WQ_MEM_RECLAIM|WQ_FREEZABLE, 0, mp->m_fsname);
 	if (!mp->m_data_workqueue)
-		goto out;
+		goto out_destroy_buf;
 
 	mp->m_unwritten_workqueue = alloc_workqueue("xfs-conv/%s",
 			WQ_MEM_RECLAIM|WQ_FREEZABLE, 0, mp->m_fsname);

@@ -884,6 +890,8 @@ out_destroy_unwritten:
 	destroy_workqueue(mp->m_unwritten_workqueue);
 out_destroy_data_iodone_queue:
 	destroy_workqueue(mp->m_data_workqueue);
+out_destroy_buf:
+	destroy_workqueue(mp->m_buf_workqueue);
 out:
 	return -ENOMEM;
 }

@@ -898,6 +906,7 @@ xfs_destroy_mount_workqueues(
 	destroy_workqueue(mp->m_cil_workqueue);
 	destroy_workqueue(mp->m_data_workqueue);
 	destroy_workqueue(mp->m_unwritten_workqueue);
+	destroy_workqueue(mp->m_buf_workqueue);
 }
 
 /*

Signature · 6 lines

1.8.3.1

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2014-11-28 02:49 UTC #2 lore raw

Re: [PATCH v3 1/2] xfs: replace global xfslogd wq with per-mount wq

Quoted 21 lines · Brian Foster

The xfslogd workqueue is a global, single-job workqueue for buffer ioend processing. This means we allow for a single work item at a time for all possible XFS mounts on a system. fsstress testing in loopback XFS over XFS configurations has reproduced xfslogd deadlocks due to the single threaded nature of the queue and dependencies introduced between the separate XFS instances by online discard (-o discard).
Discard over a loopback device converts the discard request to a hole punch (fallocate) on the underlying file. Online discard requests are issued synchronously and from xfslogd context in XFS, hence the xfslogd workqueue is blocked in the upper fs waiting on a hole punch request to be servied in the lower fs. If the lower fs issues I/O that depends on xfslogd to complete, both filesystems end up hung indefinitely. This is reproduced reliabily by generic/013 on XFS->loop->XFS test devices with the '-o discard' mount option.
Further, docker implementations appear to use this kind of configuration for container instance filesystems by default (container fs->dm-> loop->base fs) and therefore are subject to this deadlock when running on XFS.

Quoted 7 lines

Replace the global xfslogd workqueue with a per-mount variant. This guarantees each mount access to a single worker and prevents deadlocks due to inter-fs dependencies introduced by discard. Since the queue is only responsible for buffer iodone processing at this point in time, rename xfslogd to xfs-buf.
Signed-off-by Brian Foster <[email protected]>

Looks good. I'll take this as is and we can refine the way we point to the workqueue in the patches that separate the log buffer completions...

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Brian Foster

author

2014-11-13 19:24 UTC #3 lore raw

RFC PATCH v3 2/2 xfs: split metadata and log buffer completion to separate workqueues

In reply to Brian Foster

XFS traditionally sends all buffer I/O completion work to a single queue. This includes metadata buffer completion and log buffer completion. The log buffer completion requires a high priority queue to prevent stalls due to log forces getting stuck behind other queued work.

Rather than continue to prioritize all buffer I/O completion due to the needs of log completion, split log buffer completion off to m_log_workqueue and move the high priority flag from m_buf_workqueue to m_log_workqueue.

[XXX: Use of XBF_SYNCIO is purely for demonstration. Define a new flag.]

Signed-off-by Brian Foster <[email protected]>

---
 fs/xfs/xfs_buf.c   | 9 ++++++++-
 fs/xfs/xfs_super.c | 5 ++---
 2 files changed, 10 insertions(+), 4 deletions(-)

fs/xfs/xfs_buf.c +8 -1

index c06d790..58d729c 100644
@@ -1050,8 +1050,15 @@ void
 xfs_buf_ioend_async(
 	struct xfs_buf	*bp)
 {
+	struct workqueue_struct *wq;
+
+	if (bp->b_flags & XBF_SYNCIO)
+		wq = bp->b_target->bt_mount->m_log_workqueue;
+	else
+		wq = bp->b_target->bt_mount->m_buf_workqueue;
+
 	INIT_WORK(&bp->b_iodone_work, xfs_buf_ioend_work);
-	queue_work(bp->b_target->bt_mount->m_buf_workqueue, &bp->b_iodone_work);
+	queue_work(wq, &bp->b_iodone_work);
 }
 
 void

fs/xfs/xfs_super.c +2 -3

index 03e3cc2..4b8cd37 100644
@@ -843,8 +843,7 @@ xfs_init_mount_workqueues(
 	struct xfs_mount	*mp)
 {
 	mp->m_buf_workqueue = alloc_workqueue("xfs-buf/%s",
-			WQ_MEM_RECLAIM|WQ_HIGHPRI|WQ_FREEZABLE, 1,
-			mp->m_fsname);
+			WQ_MEM_RECLAIM|WQ_FREEZABLE, 1, mp->m_fsname);
 	if (!mp->m_buf_workqueue)
 		goto out;

@@ -869,7 +868,7 @@ xfs_init_mount_workqueues(
 		goto out_destroy_cil;
 
 	mp->m_log_workqueue = alloc_workqueue("xfs-log/%s",
-			WQ_FREEZABLE, 0, mp->m_fsname);
+			WQ_FREEZABLE|WQ_HIGHPRI, 0, mp->m_fsname);
 	if (!mp->m_log_workqueue)
 		goto out_destroy_reclaim;

Signature · 6 lines

1.8.3.1

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2014-11-28 02:48 UTC #4 lore raw

Re: [PATCH RFC v3 2/2] xfs: split metadata and log buffer completion to separate workqueues

Quoted 26 lines · Brian Foster

XFS traditionally sends all buffer I/O completion work to a single queue. This includes metadata buffer completion and log buffer completion. The log buffer completion requires a high priority queue to prevent stalls due to log forces getting stuck behind other queued work.
Rather than continue to prioritize all buffer I/O completion due to the needs of log completion, split log buffer completion off to m_log_workqueue and move the high priority flag from m_buf_workqueue to m_log_workqueue.
[XXX: Use of XBF_SYNCIO is purely for demonstration. Define a new flag.]
Signed-off-by Brian Foster <[email protected]>
---
 fs/xfs/xfs_buf.c   | 9 ++++++++-
 fs/xfs/xfs_super.c | 5 ++---
 2 files changed, 10 insertions(+), 4 deletions(-)
fs/xfs/xfs_buf.c +0 -0
index c06d790..58d729c 100644
@@ -1050,8 +1050,15 @@ void
 xfs_buf_ioend_async(
 	struct xfs_buf	*bp)
 {

fs/xfs/xfs_buf.c void · line 1061

Quoted 10 lines

fs/xfs/xfs_buf.c +8 -1

+	struct workqueue_struct *wq;
+
+	if (bp->b_flags & XBF_SYNCIO)
+		wq = bp->b_target->bt_mount->m_log_workqueue;
+	else
+		wq = bp->b_target->bt_mount->m_buf_workqueue;
+
 	INIT_WORK(&bp->b_iodone_work, xfs_buf_ioend_work);
-	queue_work(bp->b_target->bt_mount->m_buf_workqueue, &bp->b_iodone_work);
+	queue_work(wq, &bp->b_iodone_work);

I can see what you are doing here, but I still think it would be better to set this up at IO submission rather than taking all those cacheline misses chasing pointers on IO completion. Adding an extra pointer to the struct xfs_buf is not a big deal....

Otherwise this looks fine....

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs