FAILED: patch "[PATCH] dm integrity: fix a deadlock due to offloading to an" failed to apply to 4.19-stable tree

PATCH In review [email protected] 2 participants 3 messages Mar 9, 2020 – Mar 10, 2020

Subsystems (2)

Stable branch Device mapper

lore

Share & tools

lore

[email protected]

author maintainer

2020-03-09 20:05 UTC #0 lore raw

The patch below does not apply to the 4.19-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 53770f0ec5fd417429775ba006bc4abe14002335 Mon Sep 17 00:00:00 2001
From: Mikulas Patocka <[email protected]>
Date: Mon, 17 Feb 2020 07:43:03 -0500
Subject: [PATCH] dm integrity: fix a deadlock due to offloading to an
incorrect workqueue

If we need to perform synchronous I/O in dm_integrity_map_continue(), we must make sure that we are not in the map function - in order to avoid the deadlock due to bio queuing in generic_make_request. To avoid the deadlock, we offload the request to metadata_wq.

However, metadata_wq also processes metadata updates for write requests. If there are too many requests that get offloaded to metadata_wq at the beginning of dm_integrity_map_continue, the workqueue metadata_wq becomes clogged and the system is incapable of processing any metadata updates.

This causes a deadlock because all the requests that need to do metadata updates wait for metadata_wq to proceed and metadata_wq waits inside wait_and_add_new_range until some existing request releases its range lock (which doesn't happen because the range lock is released after metadata update).

In order to fix the deadlock, we create a new workqueue offload_wq and offload requests to it - so that processing of offload_wq is independent from processing of metadata_wq.

Fixes 7eada909bfd7 ("dm: add integrity target")

Cc [email protected] # v4.12+

Reported-by Heinz Mauelshagen <[email protected]>

Tested-by Heinz Mauelshagen <[email protected]>

Signed-off-by Heinz Mauelshagen <[email protected]>

Signed-off-by Mikulas Patocka <[email protected]>

Signed-off-by Mike Snitzer <[email protected]>

drivers/md/dm-integrity.c +15 -4

index 166727a47cef..6b6c3e1deaa8 100644
@@ -212,6 +212,7 @@ struct dm_integrity_c {
 	struct list_head wait_list;
 	wait_queue_head_t endio_wait;
 	struct workqueue_struct *wait_wq;
+	struct workqueue_struct *offload_wq;
 
 	unsigned char commit_seq;
 	commit_id_t commit_ids[N_COMMIT_IDS];

@@ -1439,7 +1440,7 @@ static void dec_in_flight(struct dm_integrity_io *dio)
 			dio->range.logical_sector += dio->range.n_sectors;
 			bio_advance(bio, dio->range.n_sectors << SECTOR_SHIFT);
 			INIT_WORK(&dio->work, integrity_bio_wait);
-			queue_work(ic->wait_wq, &dio->work);
+			queue_work(ic->offload_wq, &dio->work);
 			return;
 		}
 		do_endio_flush(ic, dio);

@@ -1865,7 +1866,7 @@ static void dm_integrity_map_continue(struct dm_integrity_io *dio, bool from_map
 
 	if (need_sync_io && from_map) {
 		INIT_WORK(&dio->work, integrity_bio_wait);
-		queue_work(ic->metadata_wq, &dio->work);
+		queue_work(ic->offload_wq, &dio->work);
 		return;
 	}

@@ -2501,7 +2502,7 @@ static void bitmap_block_work(struct work_struct *w)
 				    dio->range.n_sectors, BITMAP_OP_TEST_ALL_SET)) {
 			remove_range(ic, &dio->range);
 			INIT_WORK(&dio->work, integrity_bio_wait);
-			queue_work(ic->wait_wq, &dio->work);
+			queue_work(ic->offload_wq, &dio->work);
 		} else {
 			block_bitmap_op(ic, ic->journal, dio->range.logical_sector,
 					dio->range.n_sectors, BITMAP_OP_SET);

@@ -2524,7 +2525,7 @@ static void bitmap_block_work(struct work_struct *w)
 
 		remove_range(ic, &dio->range);
 		INIT_WORK(&dio->work, integrity_bio_wait);
-		queue_work(ic->wait_wq, &dio->work);
+		queue_work(ic->offload_wq, &dio->work);
 	}
 
 	queue_delayed_work(ic->commit_wq, &ic->bitmap_flush_work, ic->bitmap_flush_interval);

@@ -3843,6 +3844,14 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned argc, char **argv)
 		goto bad;
 	}
 
+	ic->offload_wq = alloc_workqueue("dm-integrity-offload", WQ_MEM_RECLAIM,
+					  METADATA_WORKQUEUE_MAX_ACTIVE);
+	if (!ic->offload_wq) {
+		ti->error = "Cannot allocate workqueue";
+		r = -ENOMEM;
+		goto bad;
+	}
+
 	ic->commit_wq = alloc_workqueue("dm-integrity-commit", WQ_MEM_RECLAIM, 1);
 	if (!ic->commit_wq) {
 		ti->error = "Cannot allocate workqueue";

@@ -4147,6 +4156,8 @@ static void dm_integrity_dtr(struct dm_target *ti)
 		destroy_workqueue(ic->metadata_wq);
 	if (ic->wait_wq)
 		destroy_workqueue(ic->wait_wq);
+	if (ic->offload_wq)
+		destroy_workqueue(ic->offload_wq);
 	if (ic->commit_wq)
 		destroy_workqueue(ic->commit_wq);
 	if (ic->writer_wq)

Mikulas Patocka

maintainer

2020-03-10 09:00 UTC #1 lore raw

PATCH 1/1 dm integrity: fix a deadlock due to offloading to an incorrect workqueue

Quoted 9 lines · On Mon, 9 Mar 2020, [email protected] wrote:

The patch below does not apply to the 4.19-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
thanks,
greg k-h

Here I'm sending the patch for the stable branches 4.14 and 4.19.

Mikulas

------------------ original commit in Linus's tree ------------------

Quoted 1 line

From 53770f0ec5fd417429775ba006bc4abe14002335 Mon Sep 17 00:00:00 2001

From: Mikulas Patocka <[email protected]>
Date: Mon, 17 Feb 2020 07:43:03 -0500
Subject: [PATCH] dm integrity: fix a deadlock due to offloading to an
incorrect workqueue

In order to fix the deadlock, we create a new workqueue offload_wq and offload requests to it - so that processing of offload_wq is independent from processing of metadata_wq.

Fixes 7eada909bfd7 ("dm: add integrity target")

Cc [email protected] # v4.12+

Reported-by Heinz Mauelshagen <[email protected]>

Tested-by Heinz Mauelshagen <[email protected]>

Signed-off-by Heinz Mauelshagen <[email protected]>

Signed-off-by Mikulas Patocka <[email protected]>

Signed-off-by Mike Snitzer <[email protected]>

---
 drivers/md/dm-integrity.c |   15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

drivers/md/dm-integrity.c +13 -2

Index: linux-stable/drivers/md/dm-integrity.c
===================================================================
@@ -197,6 +197,7 @@ struct dm_integrity_c {
 	struct list_head wait_list;
 	wait_queue_head_t endio_wait;
 	struct workqueue_struct *wait_wq;
+	struct workqueue_struct *offload_wq;
 
 	unsigned char commit_seq;
 	commit_id_t commit_ids[N_COMMIT_IDS];

@@ -1236,7 +1237,7 @@ static void dec_in_flight(struct dm_inte
 			dio->range.logical_sector += dio->range.n_sectors;
 			bio_advance(bio, dio->range.n_sectors << SECTOR_SHIFT);
 			INIT_WORK(&dio->work, integrity_bio_wait);
-			queue_work(ic->wait_wq, &dio->work);
+			queue_work(ic->offload_wq, &dio->work);
 			return;
 		}
 		do_endio_flush(ic, dio);

@@ -1656,7 +1657,7 @@ static void dm_integrity_map_continue(st
 
 	if (need_sync_io && from_map) {
 		INIT_WORK(&dio->work, integrity_bio_wait);
-		queue_work(ic->metadata_wq, &dio->work);
+		queue_work(ic->offload_wq, &dio->work);
 		return;
 	}

@@ -3310,6 +3311,14 @@ static int dm_integrity_ctr(struct dm_ta
 		goto bad;
 	}
 
+	ic->offload_wq = alloc_workqueue("dm-integrity-offload", WQ_MEM_RECLAIM,
+					  METADATA_WORKQUEUE_MAX_ACTIVE);
+	if (!ic->offload_wq) {
+		ti->error = "Cannot allocate workqueue";
+		r = -ENOMEM;
+		goto bad;
+	}
+
 	ic->commit_wq = alloc_workqueue("dm-integrity-commit", WQ_MEM_RECLAIM, 1);
 	if (!ic->commit_wq) {
 		ti->error = "Cannot allocate workqueue";

@@ -3546,6 +3555,8 @@ static void dm_integrity_dtr(struct dm_t
 		destroy_workqueue(ic->metadata_wq);
 	if (ic->wait_wq)
 		destroy_workqueue(ic->wait_wq);
+	if (ic->offload_wq)
+		destroy_workqueue(ic->offload_wq);
 	if (ic->commit_wq)
 		destroy_workqueue(ic->commit_wq);
 	if (ic->writer_wq)

Greg KH

author maintainer

2020-03-10 11:40 UTC #2 lore raw

Re: [PATCH v4.14 v4.19] dm integrity: fix a deadlock due to offloading to an incorrect workqueue

Quoted 10 lines · Mikulas Patocka

Quoted 6 lines · On Mon, 9 Mar 2020, [email protected] wrote:
The patch below does not apply to the 4.19-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Quoted 7 lines

Quoted 3 lines
thanks,
greg k-h
Hi
Here I'm sending the patch for the stable branches 4.14 and 4.19.

Now applied, thanks.

greg k-h