Skip to contentSkip to conversation

Various block related fixes

PATCH Applied Jens Axboe 13 participants 203 messages Oct 28 – Nov 4 Page 1 of 2

Subsystems (15)
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Various block related fixes

Hi,

Got a giant pile of stuff dumped on me, and went through and did some vetting and, particularly, commit message rewriting to get it to a somewhat comprehensible shape. If you're CC'ed on a patch, please review it. Feel free to queue it up, or reply back why the particular patch is garbage, if it is garbage.

Thanks!

Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/98 blk-cgroup: keep the old blkg while reassociating a bio

From: Josef Bacik <[email protected]>

bio_associate_blkg_from_css() drops the bio's old blkg before it looks up the new one through the css. The css is pinned by the blkg, not by the bio, so if the cgroup is gone the bio may hold the last reference and the blkcg can be freed under blkg_tryget_closest(). The rcu_read_lock() around the whole thing used to cover that, until commit 457d3c4f0fdd dropped it.

Put the old blkg after the new one has been looked up.

Fixes 457d3c4f0fdd ("blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()")
Cc Tejun Heo <[email protected]>
---
 block/blk-cgroup.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
block/blk-cgroup.c +5 -2
index 82b66147d778..e79177ac2dc2 100644
@@ -2139,8 +2139,7 @@ static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
 void bio_associate_blkg_from_css(struct bio *bio,
 				 struct cgroup_subsys_state *css)
 {
-	if (bio->bi_blkg)
-		blkg_put(bio->bi_blkg);
+	struct blkcg_gq *old_blkg = bio->bi_blkg;
 
 	if (css && css->parent) {
 		bio->bi_blkg = blkg_tryget_closest(bio, css);
@@ -2148,6 +2147,10 @@ void bio_associate_blkg_from_css(struct bio *bio,
 		blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg);
 		bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg;
 	}
+
+	/* @old_blkg may be the only thing which pins @css */
+	if (old_blkg)
+		blkg_put(old_blkg);
 }
 EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);
 
Signature · 1 line
2.56.0
maintainer Acked-by on v1 1/98
#2 lore raw

Re: [PATCH 01/98] blk-cgroup: keep the old blkg while reassociating a bio

Quoted 7 lines · Jens Axboe

bio_associate_blkg_from_css() drops the bio's old blkg before it looks up the new one through the css. The css is pinned by the blkg, not by the bio, so if the cgroup is gone the bio may hold the last reference and the blkcg can be freed under blkg_tryget_closest(). The rcu_read_lock() around the whole thing used to cover that, until commit 457d3c4f0fdd dropped it.

Quoted 6 lines

Put the old blkg after the new one has been looked up.

Fixes 457d3c4f0fdd ("blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#3 lore raw

PATCH 2/98 blk-iocost: free struct ioc after a grace period

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

ioc_qos_prfill() and ioc_cost_model_prfill() take ioc->lock from blkcg_print_blkgs(), which since commit 56cc24f59c14 runs under RCU only. That covers the policy data, which is freed after a grace period, but not the ioc it points to: ioc_rqos_exit() frees that right away, and a concurrent read of io.cost.qos locks freed memory.

Free the ioc with kfree_rcu_mightsleep().

Fixes 56cc24f59c14 ("blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()")
Cc Tejun Heo <[email protected]>
---
 block/blk-iocost.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-iocost.c +2 -1
index a81488c755ca..7b5d865d8c14 100644
@@ -2920,7 +2920,8 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
 
 	timer_shutdown_sync(&ioc->timer);
 	free_percpu(ioc->pcpu_stat);
-	kfree(ioc);
+	/* blkcg_print_blkgs() gets here from a pd with only RCU held */
+	kfree_rcu_mightsleep(ioc);
 }
 
 static const struct rq_qos_ops ioc_rqos_ops = {
Signature · 1 line
2.56.0
maintainer Acked-by on v1 2/98
#4 lore raw

Re: [PATCH 02/98] blk-iocost: free struct ioc after a grace period

Quoted 6 lines · Jens Axboe

ioc_qos_prfill() and ioc_cost_model_prfill() take ioc->lock from blkcg_print_blkgs(), which since commit 56cc24f59c14 runs under RCU only. That covers the policy data, which is freed after a grace period, but not the ioc it points to: ioc_rqos_exit() frees that right away, and a concurrent read of io.cost.qos locks freed memory.

Quoted 6 lines

Free the ioc with kfree_rcu_mightsleep().

Fixes 56cc24f59c14 ("blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#5 lore raw

PATCH 3/98 blk-throttle: don't truncate the cgroup v1 iops limits

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

tg_set_conf() parses the limit into a u64 and stores the iops ones through an unsigned int, so anything above UINT_MAX is truncated and a multiple of 2^32 becomes a limit of 0. The next IO of the group then divides by it in tg_within_iops_limit().

Clamp to UINT_MAX, which means no limit, like tg_set_limit() does on the default hierarchy.

Fixes af133ceb2610 ("blkcg: move blkio_group_conf->iops and ->bps to blk-throttle")
Cc Tejun Heo <[email protected]>
---
 block/blk-throttle.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-throttle.c +2 -1
index ffc3b70065d4..4f10fe05744b 100644
@@ -1386,7 +1386,8 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of,
 	if (is_u64)
 		*(u64 *)((void *)tg + of_cft(of)->private) = v;
 	else
-		*(unsigned int *)((void *)tg + of_cft(of)->private) = v;
+		*(unsigned int *)((void *)tg + of_cft(of)->private) =
+			min_t(u64, v, UINT_MAX);
 
 	tg_conf_updated(tg, false);
 	ret = 0;
Signature · 1 line
2.56.0
maintainer Acked-by on v1 3/98
#6 lore raw

Re: [PATCH 03/98] blk-throttle: don't truncate the cgroup v1 iops limits

Quoted 5 lines · Jens Axboe

tg_set_conf() parses the limit into a u64 and stores the iops ones through an unsigned int, so anything above UINT_MAX is truncated and a multiple of 2^32 becomes a limit of 0. The next IO of the group then divides by it in tg_within_iops_limit().

Quoted 7 lines

Clamp to UINT_MAX, which means no limit, like tg_set_limit() does on the default hierarchy.

Fixes af133ceb2610 ("blkcg: move blkio_group_conf->iops and ->bps to blk-throttle")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#7 lore raw

PATCH 4/98 block: fix use-after-free of the disk in sync_bdevs()

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

sync_bdevs() holds a reference to the bdev inode while it takes bd_disk->open_mutex. For a partition the inode doesn't keep the gendisk alive, the device reference does: part_release() drops the disk as soon as the last reference to the partition device is gone. A sync(2) racing with the last close of a partition of an already deleted disk locks a freed open_mutex.

Take a device reference before touching the disk, like blkdev_get_no_open() does, and skip the bdev if it's on its way out.

Fixes a8698707a183 ("block: move bd_mutex to struct gendisk")
---
 block/bdev.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/bdev.c +10 -0
index fac74319e9fb..cc35126e483d 100644
@@ -1366,6 +1366,15 @@ void sync_bdevs(bool wait)
 		old_inode = inode;
 		bdev = I_BDEV(inode);
 
+		/*
+		 * The inode does not keep the gendisk of a partition alive, the
+		 * device does.
+		 */
+		if (!kobject_get_unless_zero(&bdev->bd_device.kobj)) {
+			spin_lock(&blockdev_superblock->s_inode_list_lock);
+			continue;
+		}
+
 		mutex_lock(&bdev->bd_disk->open_mutex);
 		if (!atomic_read(&bdev->bd_openers)) {
 			; /* skip */
@@ -1381,6 +1390,7 @@ void sync_bdevs(bool wait)
 			filemap_fdatawrite(inode->i_mapping);
 		}
 		mutex_unlock(&bdev->bd_disk->open_mutex);
+		put_device(&bdev->bd_device);
 
 		spin_lock(&blockdev_superblock->s_inode_list_lock);
 	}
Signature · 1 line
2.56.0
Reviewed-by on v1 4/98
#8 lore raw

Re: [PATCH 04/98] block: fix use-after-free of the disk in sync_bdevs()

Quoted 11 lines · Jens Axboe

From: Josef Bacik <[email protected]>

sync_bdevs() holds a reference to the bdev inode while it takes bd_disk->open_mutex. For a partition the inode doesn't keep the gendisk alive, the device reference does: part_release() drops the disk as soon as the last reference to the partition device is gone. A sync(2) racing with the last close of a partition of an already deleted disk locks a freed open_mutex.

Take a device reference before touching the disk, like blkdev_get_no_open() does, and skip the bdev if it's on its way out.

Looks good:

In the long run I wonder if we should find a way to add the bdevfs special cases to sync_inodes_sb and reuse the common syncing logic instead of this. But some of the special cases might be a bit to much.

author maintainer
#9 lore raw

PATCH 5/98 block: don't lose the status of a failed flush request

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

flush_end_io() stores the status in fq->rq_status for whoever drops the last reference to finish the flush sequence. blk_mq_put_rq_ref() calls it with BLK_STS_OK from tag iterators and the timeout handler and overwrites an error the real completion stored. With two extra references, reading /sys/block/<disk>/inflight is enough for that, a failed flush completes everything waiting on it with success and fsync() lies to the application.

Only store a status that is an error.

Fixes 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter")
---
 block/blk-flush.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
block/blk-flush.c +7 -1
index 403a46c86411..38bd4ae5d9e7 100644
@@ -212,7 +212,13 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
 	spin_lock_irqsave(&fq->mq_flush_lock, flags);
 
 	if (!req_ref_put_and_test(flush_rq)) {
-		fq->rq_status = error;
+		/*
+		 * Remember the status of the flush for whoever drops the last
+		 * reference. blk_mq_put_rq_ref() gets here with BLK_STS_OK and
+		 * must not wipe it out.
+		 */
+		if (error != BLK_STS_OK)
+			fq->rq_status = error;
 		spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
 		return RQ_END_IO_NONE;
 	}
Signature · 1 line
2.56.0
author maintainer
#11 lore raw

PATCH 6/98 blk-mq: don't batch allocate tags from an inactive hctx

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_get_tag() checks BLK_MQ_S_INACTIVE after getting a tag and gives it back if the flag is set, so the caller retries on the hctx of the CPU it runs on now. The batched allocation for plugged submitters doesn't, so a task migrated off a dying CPU can fill plug->cached_rqs from a drained hctx, and those requests never complete.

Check the flag in blk_mq_get_tags() too and release the tags if it's set. The caller falls back to blk_mq_get_tag(), which fails the same way and starts over.

Fixes 47c122e35d7e ("block: pre-allocate requests if plug is started and is a batch")
---
 block/blk-mq-tag.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
block/blk-mq-tag.c +12 -0
index 2b751bfa50e1..aaea2e2c2f23 100644
@@ -139,6 +139,18 @@ unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags,
 		return 0;
 	ret = __sbitmap_queue_get_batch(bt, nr_tags, offset);
 	*offset += tags->nr_reserved_tags;
+
+	/*
+	 * Give up this allocation if the hctx is inactive, like blk_mq_get_tag()
+	 * does. The caller falls back to it and will retry on an active hctx.
+	 */
+	if (ret && unlikely(test_bit(BLK_MQ_S_INACTIVE, &data->hctx->state))) {
+		unsigned int i;
+
+		for_each_set_bit(i, &ret, BITS_PER_LONG)
+			blk_mq_put_tag(tags, data->ctx, *offset + i);
+		return 0;
+	}
 	return ret;
 }
 
Signature · 1 line
2.56.0
#12 lore raw

Re: [PATCH 06/98] blk-mq: don't batch allocate tags from an inactive hctx

block/blk-mq-tag.c unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags, · line 145

Quoted 1 line · Jens Axboe
block/blk-mq-tag.c +1 -0
+	 * does. The caller falls back to it and will retry on an active hctx.

it I guess is blk_mq_get_tag? Maybe spell that out.

Or just use the comment from blk_mq_tag which would apply here as well:

	/*
         * Give up this allocation if the hctx is inactive.  The caller will
         * retry on an active hctx.
         */

Otherwise this looks fine.

author maintainer
#13 lore raw

PATCH 7/98 block: keep zone write plugs with plugged BIOs alive

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The issuer of a plugged zone write hears about the completion in bio_endio(), before blk_zone_write_plug_finish_request() unplugs the zone, and may reset the zone and write it again in between. Since commit 836efd35c472 that revives the plug the reset marked dead, with the new BIO on the list and wp_offset still 0. The unplug of the first write then sees an empty zone and marks the plug dead again, BIO and all. Once that BIO is done the plug is freed and the write pointer offset is lost.

Only remove the plug of an empty zone when no BIO is left on it.

Fixes 836efd35c472 ("block: fix handling of dead zone write plugs")
Cc Damien Le Moal <[email protected]>
---
 block/blk-zoned.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
block/blk-zoned.c +8 -1
index 4c5889296191..18c05e2a1746 100644
@@ -1812,7 +1812,14 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,
 	else if (!bio_list_empty(&zwplug->bio_list))
 		disk_zone_wplug_schedule_work(disk, zwplug);
 
-	if (!zwplug->wp_offset || disk_zone_wplug_is_full(disk, zwplug))
+	/*
+	 * The issuer of the write was told about its completion before we got
+	 * here. It may already have reset the zone and started to write it
+	 * again, in which case the zone is empty but has BIOs plugged. The plug
+	 * must stay alive to track them.
+	 */
+	if ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
+	    disk_zone_wplug_is_full(disk, zwplug))
 		disk_mark_zone_wplug_dead(zwplug);
 
 	spin_unlock_irqrestore(&zwplug->lock, flags);
Signature · 1 line
2.56.0
#14 lore raw

Re: [PATCH 07/98] block: keep zone write plugs with plugged BIOs alive

Quoted 5 lines · Jens Axboe

From: Josef Bacik <[email protected]>

The issuer of a plugged zone write hears about the completion in bio_endio(), before blk_zone_write_plug_finish_request() unplugs the zone, and may reset the zone and write it again in between.

How are supposed to a zone reset and a write in the short race window?

Quoted 7 lines

Since commit
836efd35c472 that revives the plug the reset marked dead, with the new BIO on the list and wp_offset still 0. The unplug of the first write then sees an empty zone and marks the plug dead again, BIO and all. Once that BIO is done the plug is freed and the write pointer offset is lost.

Only remove the plug of an empty zone when no BIO is left on it.

I'm not sure this is the correct thing to do. It would seem more sensible to make the blk-mq path match the bio-based one and call into the write plug machinery before finishing the last bio in a request.

author maintainer
#15 lore raw

PATCH 8/98 block: keep the zone write plug of a failed write

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

When a write fails, blk_zone_write_plug_bio_endio() sets BLK_ZONE_WPLUG_NEED_WP_UPDATE as the write pointer offset is now wrong. disk_zone_wplug_unplug_bio() doesn't look at that flag, so a failed write that would have filled the zone makes it look full, the plug is freed and "full" is written to zones_cond[] for a partially written zone. Reports then show it as full and zone append emulation fails with unaligned writes, even after the zone report an error requires.

Leave the plug alone while BLK_ZONE_WPLUG_NEED_WP_UPDATE is set.

Fixes fe0418eb9bd6 ("block: Prevent potential deadlocks in zone write plug error recovery")
Cc Damien Le Moal <[email protected]>
---
 block/blk-zoned.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
block/blk-zoned.c +6 -2
index 18c05e2a1746..882d07a7c80a 100644
@@ -1817,9 +1817,13 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,
 	 * here. It may already have reset the zone and started to write it
 	 * again, in which case the zone is empty but has BIOs plugged. The plug
 	 * must stay alive to track them.
+	 *
+	 * If the write failed, the write pointer offset says nothing about the
+	 * zone. Keep the plug until the offset was updated.
 	 */
-	if ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
-	    disk_zone_wplug_is_full(disk, zwplug))
+	if (!(zwplug->flags & BLK_ZONE_WPLUG_NEED_WP_UPDATE) &&
+	    ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
+	     disk_zone_wplug_is_full(disk, zwplug)))
 		disk_mark_zone_wplug_dead(zwplug);
 
 	spin_unlock_irqrestore(&zwplug->lock, flags);
Signature · 1 line
2.56.0
#16 lore raw

Re: [PATCH 08/98] block: keep the zone write plug of a failed write

Quoted 9 lines · Jens Axboe

From: Josef Bacik <[email protected]>

When a write fails, blk_zone_write_plug_bio_endio() sets BLK_ZONE_WPLUG_NEED_WP_UPDATE as the write pointer offset is now wrong. disk_zone_wplug_unplug_bio() doesn't look at that flag, so a failed write that would have filled the zone makes it look full, the plug is freed and "full" is written to zones_cond[] for a partially written zone. Reports then show it as full and zone append emulation fails with unaligned writes, even after the zone report an error requires.

I can't parse the second half of the last sentence. Can you explain what is is supposed to mean?

Otherwise the change looks sensible.

author maintainer
#17 lore raw

PATCH 9/98 blk-mq: end the request if getting a keyslot fails

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio by hand and frees the request. By then blk_mq_bio_to_request() has done the IO accounting, which blk_mq_free_request() doesn't undo, so the in_flight counter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.

Use blk_mq_end_request() like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.

Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")
---
 block/blk-mq.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/blk-mq.c +3 -2
index 7400f678258e..c91306f8156f 100644
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
 
 	ret = blk_crypto_rq_get_keyslot(rq);
 	if (ret != BLK_STS_OK) {
-		bio_endio_status(bio, ret);
-		blk_mq_free_request(rq);
+		if (bio_zone_write_plugging(bio))
+			blk_zone_write_plug_init_request(rq);
+		blk_mq_end_request(rq, ret);
 		return;
 	}
 
Signature · 1 line
2.56.0
#18 lore raw

Re: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails

Adding Eric for the blk-crypto code.

Quoted 24 lines · Jens Axboe

From: Josef Bacik <[email protected]>

When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio by hand and frees the request. By then blk_mq_bio_to_request() has done the IO accounting, which blk_mq_free_request() doesn't undo, so the in_flight counter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.

Use blk_mq_end_request() like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.

Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")
---
 block/blk-mq.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/blk-mq.c +0 -0
index 7400f678258e..c91306f8156f 100644

block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200

Quoted 8 lines
block/blk-mq.c +3 -2
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
 
 	ret = blk_crypto_rq_get_keyslot(rq);
 	if (ret != BLK_STS_OK) {
-		bio_endio_status(bio, ret);
-		blk_mq_free_request(rq);
+		if (bio_zone_write_plugging(bio))
+			blk_zone_write_plug_init_request(rq);
+		blk_mq_end_request(rq, ret);

If blk_mq_end_request does the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder the blk_crypto_rq_get_keyslot and blk_zone_write_plug_init_request calls so that the extra copy of the blk_zone_write_plug_init_request call is not needed.

#19 lore raw

Re: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails

Quoted 28 lines · Christoph Hellwig

Adding Eric for the blk-crypto code.

Quoted 25 lines · Jens Axboe

From: Josef Bacik <[email protected]>

When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio by hand and frees the request. By then blk_mq_bio_to_request() has done the IO accounting, which blk_mq_free_request() doesn't undo, so the in_flight counter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.

Use blk_mq_end_request() like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.

Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")
---
 block/blk-mq.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/blk-mq.c +0 -0
index 7400f678258e..c91306f8156f 100644
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
 
Quoted 13 lines

block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200

Quoted 7 lines
block/blk-mq.c +3 -2
 	ret = blk_crypto_rq_get_keyslot(rq);
 	if (ret != BLK_STS_OK) {
-		bio_endio_status(bio, ret);
-		blk_mq_free_request(rq);
+		if (bio_zone_write_plugging(bio))
+			blk_zone_write_plug_init_request(rq);
+		blk_mq_end_request(rq, ret);

If blk_mq_end_request does the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder the blk_crypto_rq_get_keyslot and blk_zone_write_plug_init_request calls so that the extra copy of the blk_zone_write_plug_init_request call is not needed.

I don't know the blk-mq code super well. But yes, blk_zone_write_plug_init_request() has no dependency on the keyslot; in particular the back-merging relies on crypt_ctx, not crypt_keyslot. So calling it before blk_crypto_rq_get_keyslot() would be simpler than duplicating it in the success and failure cases.

And yes, blk_mq_end_request() doesn't assume blk_crypto_rq_get_keyslot() has succeeded. On failure crypt_keyslot just remains NULL.

Josef and Jens, what are your expectations about this patch? Sending it as part of a 98-patch series (that didn't get picked up by Sashiko) is kind of awkward. If you need help I'd be glad to send a revised version as a standalone patch. Or you can handle it: up to you.

- Eric

author maintainer
#20 lore raw

Re: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails

Quoted 49 lines · Eric Biggers
Quoted 28 lines · Christoph Hellwig

Adding Eric for the blk-crypto code.

Quoted 25 lines · Jens Axboe

From: Josef Bacik <[email protected]>

When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio by hand and frees the request. By then blk_mq_bio_to_request() has done the IO accounting, which blk_mq_free_request() doesn't undo, so the in_flight counter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.

Use blk_mq_end_request() like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.

Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")
---
 block/blk-mq.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/blk-mq.c +0 -0
index 7400f678258e..c91306f8156f 100644
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
 
Quoted 13 lines

block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200

Quoted 7 lines
block/blk-mq.c +3 -2
 	ret = blk_crypto_rq_get_keyslot(rq);
 	if (ret != BLK_STS_OK) {
-		bio_endio_status(bio, ret);
-		blk_mq_free_request(rq);
+		if (bio_zone_write_plugging(bio))
+			blk_zone_write_plug_init_request(rq);
+		blk_mq_end_request(rq, ret);

If blk_mq_end_request does the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder the blk_crypto_rq_get_keyslot and blk_zone_write_plug_init_request calls so that the extra copy of the blk_zone_write_plug_init_request call is not needed.

I don't know the blk-mq code super well. But yes, blk_zone_write_plug_init_request() has no dependency on the keyslot; in particular the back-merging relies on crypt_ctx, not crypt_keyslot. So calling it before blk_crypto_rq_get_keyslot() would be simpler than duplicating it in the success and failure cases.

Quoted 8 lines

And yes, blk_mq_end_request() doesn't assume blk_crypto_rq_get_keyslot() has succeeded. On failure crypt_keyslot just remains NULL.

Josef and Jens, what are your expectations about this patch? Sending it as part of a 98-patch series (that didn't get picked up by Sashiko) is kind of awkward. If you need help I'd be glad to send a revised version as a standalone patch. Or you can handle it: up to you.

I should have put this in the cover letter, that's my mistake. But I've replied to other patches in this thread, and the idea is "congrats, as the maintainer, you now own this bug" and you are responsible for fixing it. If the patch is good, feel free to ack it and run with it. If it needs work, do the work and send it out as any other patch you would. If you do that, feel free to rewrite it and claim authorship as well, nobody cares about that part.

So yes, would be great if you could send a revised patch in this case. Thanks!

Signature · 1 line
Jens Axboe
author maintainer
#21 lore raw

PATCH 10/98 block: don't unplug a zone under a failed write

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Since commit 552c1149af7a disk_zone_wplug_abort() clears BLK_ZONE_WPLUG_PLUGGED. For a failed write that happens in bio_endio(), before the request reaches blk_zone_write_plug_finish_request(). The issuer can reset the zone and submit a new write in between, which is issued unplugged, and when the old request is finished it clears the flag under that one, so the next write goes out alongside it.

Preserve the flag across the abort, the unplug of the failed write clears it. The exception is QUEUE_FLAG_ZONED_QD1_WRITES, where the abort takes the plug off the worker's list and nobody is left to clear it.

Fixes 552c1149af7a ("block: Clear BLK_ZONE_WPLUG_PLUGGED when aborting plugged BIOs")
Cc Damien Le Moal <[email protected]>
---
 block/blk-zoned.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
block/blk-zoned.c +12 -1
index 882d07a7c80a..7f405dce2fcb 100644
@@ -1871,9 +1871,20 @@ void blk_zone_write_plug_bio_endio(struct bio *bio)
 	 * needing a write pointer update.
 	 */
 	if (bio->bi_status != BLK_STS_OK) {
+		unsigned int plugged = 0;
+
 		spin_lock_irqsave(&zwplug->lock, flags);
+		/*
+		 * The abort unplugs the zone. But the flag still belongs to a
+		 * write in flight (this one, until its request is finished) or
+		 * to the BIO work, and they clear it when they are done. The
+		 * zone write plugs worker does not rely on the flag to issue
+		 * one write at a time and forgets the plug with the abort.
+		 */
+		if (!blk_queue_zoned_qd1_writes(disk->queue))
+			plugged = zwplug->flags & BLK_ZONE_WPLUG_PLUGGED;
 		disk_zone_wplug_abort(zwplug);
-		zwplug->flags |= BLK_ZONE_WPLUG_NEED_WP_UPDATE;
+		zwplug->flags |= BLK_ZONE_WPLUG_NEED_WP_UPDATE | plugged;
 		spin_unlock_irqrestore(&zwplug->lock, flags);
 	}
 
Signature · 1 line
2.56.0
author maintainer
#22 lore raw

PATCH 11/98 block: check dma alignment of bvec iters in async dio

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

__blkdev_direct_IO_async() uses bio_iov_iter_set() for an ITER_BVEC iterator and never gets to the alignment check in bio_iov_iter_get_pages(), which is the one that exists for io_uring registered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misaligned READ_FIXED from brd returns the wrong data, and dm-verity with panic_on_error panics.

Move bio_iov_bvec_aligned() to blk.h and check in the async path too, failing with -EINVAL like the other direct IO paths.

Fixes 5ff3f74e145a ("block: simplify direct io validity check")
---
 block/bio.c  | 33 ---------------------------------
 block/blk.h  | 33 +++++++++++++++++++++++++++++++++
 block/fops.c | 10 ++++++++--
 3 files changed, 41 insertions(+), 35 deletions(-)
block/bio.c +0 -33
index 06f805d3866a..57d84ca7170f 100644
@@ -1272,39 +1272,6 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
 	return 0;
 }
 
-#ifdef CONFIG_DEBUG_KERNEL
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	struct bvec_iter iter;
-	struct bio_vec bv;
-
-	/*
-	 * Correct callers never break the alignment requirements, so this
-	 * exhaustive check is only paid for in debug builds.
-	 */
-	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
-		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
-			return false;
-	return true;
-}
-#else
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	/*
-	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
-	 * segments already aligned to the device's DMA alignment. The only
-	 * unchecked user-controllable offset that reaches here is an io_uring
-	 * registered buffer where just the first segment can be unaligned
-	 * (the rest is virtually contiguous), so checking only that one is
-	 * sufficient to know if the entire vector is valid.
-	 */
-	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
-							mem_align_mask);
-}
-#endif
-
 /**
  * bio_iov_iter_get_pages - add user or kernel pages to a bio
  * @bio: bio to add pages to
block/blk.h +33 -0
index 2cc03aa54c53..a48665e10994 100644
@@ -202,6 +202,39 @@ static inline bool rq_mergeable(struct request *rq)
 	return true;
 }
 
+#ifdef CONFIG_DEBUG_KERNEL
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	struct bvec_iter iter;
+	struct bio_vec bv;
+
+	/*
+	 * Correct callers never break the alignment requirements, so this
+	 * exhaustive check is only paid for in debug builds.
+	 */
+	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
+		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
+			return false;
+	return true;
+}
+#else
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	/*
+	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
+	 * segments already aligned to the device's DMA alignment. The only
+	 * unchecked user-controllable offset that reaches here is an io_uring
+	 * registered buffer where just the first segment can be unaligned
+	 * (the rest is virtually contiguous), so checking only that one is
+	 * sufficient to know if the entire vector is valid.
+	 */
+	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
+							mem_align_mask);
+}
+#endif
+
 /*
  * There are two different ways to handle DISCARD merges:
  *  1) If max_discard_segments > 1, the driver treats every bio as a range and
block/fops.c +8 -2
index 375ec2c15926..dfeac910f126 100644
@@ -365,9 +365,15 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	 * Users don't rely on the iterator being in any particular
 	 * state for async I/O returning -EIOCBQUEUED, hence we can
 	 * avoid expensive iov_iter_advance(). Bypass
-	 * bio_iov_iter_get_pages() and set the bvec directly.
+	 * bio_iov_iter_get_pages() and set the bvec directly, which leaves
+	 * its alignment check to us.
 	 */
-	if (!bio_iov_iter_set(bio, iter)) {
+	if (bio_iov_iter_set(bio, iter)) {
+		if (!bio_iov_bvec_aligned(bio, bdev_dma_alignment(bdev))) {
+			ret = -EINVAL;
+			goto out_bio_put;
+		}
+	} else {
 		ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
 		if (unlikely(ret))
 			goto out_bio_put;
Signature · 1 line
2.56.0
#23 lore raw

Re: [PATCH 11/98] block: check dma alignment of bvec iters in async dio

Quoted 11 lines · Jens Axboe

From: Josef Bacik <[email protected]>

__blkdev_direct_IO_async() uses bio_iov_iter_set() for an ITER_BVEC iterator and never gets to the alignment check in bio_iov_iter_get_pages(), which is the one that exists for io_uring registered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misaligned READ_FIXED from brd returns the wrong data, and dm-verity with panic_on_error panics.

Move bio_iov_bvec_aligned() to blk.h and check in the async path too, failing with -EINVAL like the other direct IO paths.

I think the theory was that bvec callers have to validate the offsets themselves. It looks like that doesn't always work? Should we fix that in the callers or do we decide we're unlikely to get all of them right and do the checks unconditionally?

Quoted 123 lines
Fixes 5ff3f74e145a ("block: simplify direct io validity check")
---
 block/bio.c  | 33 ---------------------------------
 block/blk.h  | 33 +++++++++++++++++++++++++++++++++
 block/fops.c | 10 ++++++++--
 3 files changed, 41 insertions(+), 35 deletions(-)
block/bio.c +0 -33
index 06f805d3866a..57d84ca7170f 100644
@@ -1272,39 +1272,6 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
 	return 0;
 }
 
-#ifdef CONFIG_DEBUG_KERNEL
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	struct bvec_iter iter;
-	struct bio_vec bv;
-
-	/*
-	 * Correct callers never break the alignment requirements, so this
-	 * exhaustive check is only paid for in debug builds.
-	 */
-	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
-		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
-			return false;
-	return true;
-}
-#else
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	/*
-	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
-	 * segments already aligned to the device's DMA alignment. The only
-	 * unchecked user-controllable offset that reaches here is an io_uring
-	 * registered buffer where just the first segment can be unaligned
-	 * (the rest is virtually contiguous), so checking only that one is
-	 * sufficient to know if the entire vector is valid.
-	 */
-	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
-							mem_align_mask);
-}
-#endif
-
 /**
  * bio_iov_iter_get_pages - add user or kernel pages to a bio
  * @bio: bio to add pages to
block/blk.h +33 -0
index 2cc03aa54c53..a48665e10994 100644
@@ -202,6 +202,39 @@ static inline bool rq_mergeable(struct request *rq)
 	return true;
 }
 
+#ifdef CONFIG_DEBUG_KERNEL
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	struct bvec_iter iter;
+	struct bio_vec bv;
+
+	/*
+	 * Correct callers never break the alignment requirements, so this
+	 * exhaustive check is only paid for in debug builds.
+	 */
+	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
+		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
+			return false;
+	return true;
+}
+#else
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	/*
+	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
+	 * segments already aligned to the device's DMA alignment. The only
+	 * unchecked user-controllable offset that reaches here is an io_uring
+	 * registered buffer where just the first segment can be unaligned
+	 * (the rest is virtually contiguous), so checking only that one is
+	 * sufficient to know if the entire vector is valid.
+	 */
+	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
+							mem_align_mask);
+}
+#endif
+
 /*
  * There are two different ways to handle DISCARD merges:
  *  1) If max_discard_segments > 1, the driver treats every bio as a range and
block/fops.c +8 -2
index 375ec2c15926..dfeac910f126 100644
@@ -365,9 +365,15 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	 * Users don't rely on the iterator being in any particular
 	 * state for async I/O returning -EIOCBQUEUED, hence we can
 	 * avoid expensive iov_iter_advance(). Bypass
-	 * bio_iov_iter_get_pages() and set the bvec directly.
+	 * bio_iov_iter_get_pages() and set the bvec directly, which leaves
+	 * its alignment check to us.
 	 */
-	if (!bio_iov_iter_set(bio, iter)) {
+	if (bio_iov_iter_set(bio, iter)) {
+		if (!bio_iov_bvec_aligned(bio, bdev_dma_alignment(bdev))) {
+			ret = -EINVAL;
+			goto out_bio_put;
+		}
+	} else {
 		ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
 		if (unlikely(ret))
 			goto out_bio_put;
Signature · 1 line
2.56.0

---end quoted text---

maintainer
#24 lore raw

Re: [PATCH 11/98] block: check dma alignment of bvec iters in async dio

Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Jens Axboe

From: Josef Bacik <[email protected]>

__blkdev_direct_IO_async() uses bio_iov_iter_set() for an ITER_BVEC iterator and never gets to the alignment check in bio_iov_iter_get_pages(), which is the one that exists for io_uring registered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misaligned READ_FIXED from brd returns the wrong data, and dm-verity with panic_on_error panics.

Quoted 7 lines
Quoted 2 lines

Move bio_iov_bvec_aligned() to blk.h and check in the async path too, failing with -EINVAL like the other direct IO paths.

I think the theory was that bvec callers have to validate the offsets themselves. It looks like that doesn't always work? Should we fix that in the callers or do we decide we're unlikely to get all of them right and do the checks unconditionally?

Yes that was the idea. Not that that's happening here with vectored registered buffer usage, so the patch is correct as far as that goes. It's a bit of a bummer as the point was to avoid repeated vector walks, and I don't think we can efficiently validate while importing the vector.

The block layer will still catch the invalid alignment, but further down the stack, and I've learned the hard way that some mappers just are not handling these kinds of errors that late in the submission path.

author maintainer
#25 lore raw

PATCH 12/98 drbd: stop the start_resync timer of a dying device

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

start_resync_timer gets armed for up to a second and nothing ever deletes it, so drbd_destroy_device() can free the device with the timer pending. Shut it down where request_timer is shut down.

Fixes 370a43e7982d ("drbd: Work on the Ahead -> SyncSource transition")
Cc Philipp Reisner <[email protected]>
Cc Lars Ellenberg <[email protected]>
Cc "Christoph Böhmwalder" <[email protected]>
---
 drivers/block/drbd/drbd_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/block/drbd/drbd_main.c +1 -0
index a2a841c89201..099dff1ea256 100644
@@ -2163,6 +2163,7 @@ void drbd_destroy_device(struct kref *kref)
 	struct drbd_peer_device *peer_device, *tmp_peer_device;
 
 	timer_shutdown_sync(&device->request_timer);
+	timer_shutdown_sync(&device->start_resync_timer);
 
 	/* paranoia asserts */
 	D_ASSERT(device, device->open_cnt == 0);
Signature · 1 line
2.56.0
author maintainer
#26 lore raw

PATCH 13/98 blk-cgroup: fix root io.stat byte counts on 32-bit

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blkcg_fill_root_iostats() shifts the unsigned long sector counts by 9 before adding them to the u64 byte counts, so on 32-bit the contribution of a CPU wraps every 4GB. Widen before the shift.

Fixes ef45fe470e1e ("blk-cgroup: show global disk stats in root cgroup io.stat")
Cc Tejun Heo <[email protected]>
---
 block/blk-cgroup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
block/blk-cgroup.c +3 -3
index e79177ac2dc2..c9187d95a54b 100644
@@ -1118,11 +1118,11 @@ static void blkcg_fill_root_iostats(void)
 				cpu_dkstats->ios[STAT_DISCARD];
 			// convert sectors to bytes
 			tmp.bytes[BLKG_IOSTAT_READ] +=
-				cpu_dkstats->sectors[STAT_READ] << 9;
+				(u64)cpu_dkstats->sectors[STAT_READ] << 9;
 			tmp.bytes[BLKG_IOSTAT_WRITE] +=
-				cpu_dkstats->sectors[STAT_WRITE] << 9;
+				(u64)cpu_dkstats->sectors[STAT_WRITE] << 9;
 			tmp.bytes[BLKG_IOSTAT_DISCARD] +=
-				cpu_dkstats->sectors[STAT_DISCARD] << 9;
+				(u64)cpu_dkstats->sectors[STAT_DISCARD] << 9;
 		}
 
 		flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
Signature · 1 line
2.56.0
maintainer Acked-by on v1 13/98
#27 lore raw

Re: [PATCH 13/98] blk-cgroup: fix root io.stat byte counts on 32-bit

Quoted 8 lines · Jens Axboe

blkcg_fill_root_iostats() shifts the unsigned long sector counts by 9 before adding them to the u64 byte counts, so on 32-bit the contribution of a CPU wraps every 4GB. Widen before the shift.

Fixes ef45fe470e1e ("blk-cgroup: show global disk stats in root cgroup io.stat")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#28 lore raw

PATCH 14/98 drbd: take req_lock to skip the initial sync as peer

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

_drbd_set_state() must be called with req_lock held. drbd_adm_new_c_uuid() does, receive_uuids() on the peer doesn't, so a concurrent state change gets lost: the device is Secondary again, or its failed disk UpToDate, while the work for the lost change is already queued.

Fixes b411b3637fa7 ("The DRBD driver")
Cc Philipp Reisner <[email protected]>
Cc Lars Ellenberg <[email protected]>
Cc "Christoph Böhmwalder" <[email protected]>
---
 drivers/block/drbd/drbd_receiver.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/block/drbd/drbd_receiver.c +2 -0
index 2135c14354a8..a405ccdf919e 100644
@@ -4096,8 +4096,10 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 					BM_LOCKED_TEST_ALLOWED, NULL);
 			_drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
 			_drbd_uuid_set(device, UI_BITMAP, 0);
+			spin_lock_irq(&device->resource->req_lock);
 			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
 					CS_VERBOSE, NULL);
+			spin_unlock_irq(&device->resource->req_lock);
 			drbd_md_sync(device);
 			updated_uuids = 1;
 		}
Signature · 1 line
2.56.0
author maintainer
#29 lore raw

PATCH 15/98 zloop: refuse zone files without read/write_iter

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

zloop_do_rw() calls ->read_iter() and ->write_iter() on the zone files without checking that they exist. A base_dir on hugetlbfs, or zone files that are links into procfs, create a device just fine and the first IO jumps to NULL. Check for both when a zone file is opened, like loop does for its backing file.

Fixes eb0570c7df23 ("block: new zoned loop block device driver")
Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
---
 drivers/block/zloop.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/block/zloop.c +10 -0
index 9145e0f98d56..b4076940ae7f 100644
@@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,
 		return ERR_PTR(-ENOMEM);
 	file = filp_open(p, oflags, mode);
 	kfree(p);
+	if (IS_ERR(file))
+		return file;
+
+	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
+	if ((oflags & O_ACCMODE) == O_RDWR &&
+	    (!file->f_op->read_iter || !file->f_op->write_iter)) {
+		fput(file);
+		return ERR_PTR(-EINVAL);
+	}
+
 	return file;
 }
 
Signature · 1 line
2.56.0
Reviewed-by on v1 15/98
#30 lore raw

Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter

Quoted 7 lines · Jens Axboe

From: Josef Bacik <[email protected]>

zloop_do_rw() calls ->read_iter() and ->write_iter() on the zone files without checking that they exist. A base_dir on hugetlbfs, or zone files that are links into procfs, create a device just fine and the first IO jumps to NULL. Check for both when a zone file is opened, like loop does for its backing file.

Not particularly against the patch, but I don't think you can have a zloop

zone file be a link into procfs. Well you can if you create the zloop, tear it

down again, then rm a zone file and re-create it as a link into proc. So I guess:

Quoted 14 lines
Fixes eb0570c7df23 ("block: new zoned loop block device driver")
Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
---
  drivers/block/zloop.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/block/zloop.c +0 -0
index 9145e0f98d56..b4076940ae7f 100644

drivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063

Quoted 8 lines
drivers/block/zloop.c +5 -0
@@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,
  		return ERR_PTR(-ENOMEM);
  	file = filp_open(p, oflags, mode);
  	kfree(p);
+	if (IS_ERR(file))
+		return file;
+
+	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
+	if ((oflags & O_ACCMODE) == O_RDWR &&

I think you can drop the O_RDWR check, the zone files are always opened O_RDWR,

only zloop_dev_exists() passes O_RDONLY to zloop_filp_open_fmt() but that is just to

check if the files exist and they're fput() right after it.

Quoted 8 lines
drivers/block/zloop.c +5 -0
+	    (!file->f_op->read_iter || !file->f_op->write_iter)) {
+		fput(file);
+		return ERR_PTR(-EINVAL);
+	}
+
  	return file;
  }
 
#31 lore raw

Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter

In reply to Jens Axboe

drivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063

Quoted 4 lines · Jens Axboe
drivers/block/zloop.c +4 -0
+		return file;
+
+	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
+	if ((oflags & O_ACCMODE) == O_RDWR &&

I don't think this checks makes any sense.

The proper fix here is to copy and paste loop_check_backing_file from loop. I an prepare a patch for that.

author maintainer
#32 lore raw

Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter

Quoted 10 lines · Christoph Hellwig

drivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063

Quoted 4 lines · Jens Axboe
drivers/block/zloop.c +4 -0
+		return file;
+
+	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
+	if ((oflags & O_ACCMODE) == O_RDWR &&

I don't think this checks makes any sense.

The proper fix here is to copy and paste loop_check_backing_file from loop. I an prepare a patch for that.

Perfect, I'll drop this one.

Signature · 1 line
Jens Axboe
author maintainer
#33 lore raw

PATCH 16/98 block/rnbd-srv: don't free the device on a sysfs error

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

rnbd_srv_create_dev_sysfs() drops the only kobject reference on its error paths, and the release function frees the rnbd_srv_dev. process_msg_open() keeps using it and ends in destroy_device() with a list_del() and a second kfree(). This doesn't need memory pressure: a client opening a device while its last mapping goes away gets -EEXIST from kobject_add() against the old directory.

Only undo kobject_add() on failure and leave the reference to destroy_device(). Initialise the kobject once, the next open retries.

Fixes 8cee532f469b ("block/rnbd: server: sysfs interface functions")
Fixes d3a95ccaaf4d ("block/rnbd: call kobject_put in the failure path")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>
---
 drivers/block/rnbd/rnbd-srv-sysfs.c | 16 ++++++++++------
 drivers/block/rnbd/rnbd-srv.c       |  3 +++
 2 files changed, 13 insertions(+), 6 deletions(-)
drivers/block/rnbd/rnbd-srv-sysfs.c +10 -6
index 64780094442c..33d5d1738be5 100644
@@ -44,12 +44,17 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,
 	struct kobject *bdev_kobj;
 	int ret;
 
-	ret = kobject_init_and_add(&dev->dev_kobj, &dev_ktype,
-				   rnbd_devs_kobj, "%pg", bdev);
-	if (ret) {
-		kobject_put(&dev->dev_kobj);
+	/*
+	 * The release function frees the device, which the caller goes on
+	 * using and which others may share: never drop the reference of the
+	 * kobject here. That is for destroy_device().
+	 */
+	if (!dev->dev_kobj.state_initialized)
+		kobject_init(&dev->dev_kobj, &dev_ktype);
+
+	ret = kobject_add(&dev->dev_kobj, rnbd_devs_kobj, "%pg", bdev);
+	if (ret)
 		return ret;
-	}
 
 	dev->dev_sessions_kobj = kobject_create_and_add("sessions",
 							&dev->dev_kobj);
@@ -69,7 +74,6 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,
 	kobject_put(dev->dev_sessions_kobj);
 free_dev_kobj:
 	kobject_del(&dev->dev_kobj);
-	kobject_put(&dev->dev_kobj);
 	return ret;
 }
 
drivers/block/rnbd/rnbd-srv.c +3 -0
index 79c9a5fb418f..2b0619389fc9 100644
@@ -208,6 +208,9 @@ static void destroy_device(struct kref *kref)
 		 * Destroy kobj only if it was really created.
 		 */
 		rnbd_srv_destroy_dev_sysfs(dev);
+	else if (dev->dev_kobj.state_initialized)
+		/* The files could not be created. Frees the name, too. */
+		kobject_put(&dev->dev_kobj);
 	else
 		kfree(dev);
 }
Signature · 1 line
2.56.0
author maintainer
#34 lore raw

PATCH 17/98 block/rnbd-srv: drop the sess_dev kobject unlocked

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

rnbd_srv_create_dev_session_sysfs() drops the kobject reference when it fails. The release function is rnbd_destroy_sess_dev(), which takes srv_dev->lock, and the caller holds that lock: the worker deadlocks on itself. A client opening one device as "ram0" and "./ram0" gets there through -EEXIST from kobject_add().

Have process_msg_open() drop the reference after releasing the lock, and leave the rest of the cleanup to the release function.

Fixes 8cee532f469b ("block/rnbd: server: sysfs interface functions")
Fixes d3a95ccaaf4d ("block/rnbd: call kobject_put in the failure path")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>
---
 drivers/block/rnbd/rnbd-srv-sysfs.c | 12 ++++++------
 drivers/block/rnbd/rnbd-srv.c       |  8 +++++++-
 2 files changed, 13 insertions(+), 7 deletions(-)
drivers/block/rnbd/rnbd-srv-sysfs.c +6 -6
index 33d5d1738be5..9d00c42f2624 100644
@@ -193,6 +193,10 @@ static const struct kobj_type rnbd_srv_sess_dev_ktype = {
 	.release	= rnbd_srv_sess_dev_release,
 };
 
+/*
+ * On failure the caller has to drop the reference of the kobject. The release
+ * function takes the lock of the device, which the caller holds here.
+ */
 int rnbd_srv_create_dev_session_sysfs(struct rnbd_srv_sess_dev *sess_dev)
 {
 	int ret;
@@ -200,17 +204,13 @@ int rnbd_srv_create_dev_session_sysfs(struct rnbd_srv_sess_dev *sess_dev)
 	ret = kobject_init_and_add(&sess_dev->kobj, &rnbd_srv_sess_dev_ktype,
 				   sess_dev->dev->dev_sessions_kobj, "%s",
 				   sess_dev->sess->sessname);
-	if (ret) {
-		kobject_put(&sess_dev->kobj);
+	if (ret)
 		return ret;
-	}
 
 	ret = sysfs_create_group(&sess_dev->kobj,
 				 &rnbd_srv_default_dev_session_attr_group);
-	if (ret) {
+	if (ret)
 		kobject_del(&sess_dev->kobj);
-		kobject_put(&sess_dev->kobj);
-	}
 
 	return ret;
 }
drivers/block/rnbd/rnbd-srv.c +7 -1
index 2b0619389fc9..1e3de82fd70b 100644
@@ -590,6 +590,7 @@ rnbd_srv_create_set_sess_dev(struct rnbd_srv_session *srv_sess,
 		return sdev;
 
 	kref_init(&sdev->kref);
+	INIT_LIST_HEAD(&sdev->dev_list);
 
 	strscpy(sdev->pathname, open_msg->dev_name, sizeof(sdev->pathname));
 
@@ -789,7 +790,12 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 		rnbd_srv_err(srv_sess_dev,
 			      "Opening device failed, failed to create dev client sysfs files, err: %d\n",
 			      ret);
-		goto free_srv_sess_dev;
+		/*
+		 * rnbd_destroy_sess_dev() undoes everything down to the open
+		 * of the block device.
+		 */
+		kobject_put(&srv_sess_dev->kobj);
+		goto free_path;
 	}
 
 	list_add(&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
Signature · 1 line
2.56.0
author maintainer
#35 lore raw

PATCH 18/98 aoe: don't free a device with its work queued

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

flush() picks the devices without pending work and frees them, but since aoedev_downdev() may sleep, the check and the DEVFL_TKILL marking are no longer one step under the lock. A reply to an ATA IDENTIFY in between queues d->work, and the device is freed with it queued.

Set DEVFL_TKILL under d->lock, don't queue the work for a device that has it, and cancel the work in freedev().

Fixes 430380b4637a ("block: aoe: Fix kernel crash due to atomic sleep when exiting")
Cc Justin Sanders <[email protected]>
---
 drivers/block/aoe/aoecmd.c | 2 +-
 drivers/block/aoe/aoedev.c | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)
drivers/block/aoe/aoecmd.c +1 -1
index a4744a30a8af..c5b102947998 100644
@@ -974,7 +974,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id)
 			d->fw_ver, (long long)ssize);
 	d->ssize = ssize;
 	d->geo.start = 0;
-	if (d->flags & (DEVFL_GDALLOC|DEVFL_NEWSIZE))
+	if (d->flags & (DEVFL_GDALLOC | DEVFL_NEWSIZE | DEVFL_TKILL))
 		return;
 	if (d->gd != NULL)
 		d->flags |= DEVFL_NEWSIZE;
drivers/block/aoe/aoedev.c +4 -0
index 6e8b3807c1c8..5b443b3b224e 100644
@@ -286,6 +286,8 @@ freedev(struct aoedev *d)
 		return;
 
 	timer_delete_sync(&d->timer);
+	/* With DEVFL_TKILL set only the work itself still queues it */
+	cancel_work_sync(&d->work);
 	if (d->gd) {
 		aoedisk_rm_debugfs(d);
 		del_gendisk(d->gd);
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
 		aoedev_downdev(d);
+		spin_lock_irqsave(&d->lock, flags);
 		d->flags |= DEVFL_TKILL;
+		spin_unlock_irqrestore(&d->lock, flags);
 		goto restart1;
 cont:
 		spin_unlock(&d->lock);
Signature · 1 line
2.56.0
#36 lore raw

Re: [PATCH 18/98] aoe: don't free a device with its work queued

drivers/block/aoe/aoedev.c flush(const char __user *str, size_t cnt, int exiting) · line 364

Quoted 8 lines · Jens Axboe
drivers/block/aoe/aoedev.c +2 -0
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
 		aoedev_downdev(d);
+		spin_lock_irqsave(&d->lock, flags);
 		d->flags |= DEVFL_TKILL;
+		spin_unlock_irqrestore(&d->lock, flags);
 		goto restart1;

Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race window open while aoedev_downdev(d) drops d->lock and sleeps in blk_mq_freeze_queue(d->blkq):

1. An incoming ATA IDENTIFY reply processed by ataid_complete() during that window still sees !(d->flags & DEVFL_TKILL), sets DEVFL_NEWSIZE (or DEVFL_GDALLOC), and queues d->work.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs before freedev() reaches cancel_work_sync(&d->work), it calls aoeblk_gdalloc() or sets d->flags |= DEVFL_UP again after aoedev_downdev() cleared DEVFL_UP, allowing new requests to be queued onto d and hanging del_gendisk() in freedev().

Should d->flags |= DEVFL_TKILL be set under d->lock before unlocking and calling aoedev_downdev(d) (which also prevents two concurrent aoedev_flush() callers from running aoedev_downdev() on the same device), and should aoecmd_sleepwork() bail out if DEVFL_TKILL is set in case d->work was queued between flush_workqueue(aoe_wq) and acquiring d->lock?

#37 lore raw

Re: [PATCH 18/98] aoe: don't free a device with its work queued

In reply to Jens Axboe

Review flush(const char __user *str, size_t cnt, int exiting) · line 364

Quoted 8 lines · Jens Axboe
(quoted diff) +2 -0
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
 		aoedev_downdev(d);
+		spin_lock_irqsave(&d->lock, flags);
 		d->flags |= DEVFL_TKILL;
+		spin_unlock_irqrestore(&d->lock, flags);
 		goto restart1;

Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race window open while aoedev_downdev(d) drops d->lock and sleeps in blk_mq_freeze_queue(d->blkq):

1. ataid_complete(), rexmit_timer(), or aoeblk_gdalloc()'s error path can still see !(d->flags & DEVFL_TKILL) and queue d->work during that window.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs before freedev() reaches cancel_work_sync(&d->work), it can call aoeblk_gdalloc() (hitting WARN_ON(d->flags & DEVFL_TKILL) if DEVFL_TKILL was set in the meantime) or set d->flags |= DEVFL_UP again after aoedev_downdev() cleared DEVFL_UP, letting new I/O enter and hanging del_gendisk() in freedev().

Instead of setting DEVFL_TKILL after aoedev_downdev(d), how about:

- Serializing flush() with a static flush_mutex so a second aoedev_flush() caller cannot enter pass 2 (freedev()) and pass 3 (kfree()) on a device while the first caller is still in pass 1.
- Setting d->flags |= DEVFL_TKILL under d->lock before unlocking, then running cancel_work_sync(&d->work) before aoedev_downdev(d) (and keeping cancel_work_sync(&d->work) after timer_delete_sync(&d->timer) in freedev()).
- Checking DEVFL_TKILL at all three queue_work(aoe_wq, &d->work) sites (ataid_complete(), rexmit_timer(), and aoeblk_gdalloc()), as well as in aoecmd_sleepwork() and aoeblk_gdalloc().

Here is a replacement patch against the same baseline (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

drivers/block/aoe/aoeblk.c +6 -2
index 34ead75e7e02..69586dc6d379 100644
@@ -385,7 +385,10 @@ aoeblk_gdalloc(void *vp)
 	spin_lock_irqsave(&d->lock, flags);
 	WARN_ON(!(d->flags & DEVFL_GD_NOW));
 	WARN_ON(!(d->flags & DEVFL_GDALLOC));
-	WARN_ON(d->flags & DEVFL_TKILL);
+	if (d->flags & DEVFL_TKILL) {
+		spin_unlock_irqrestore(&d->lock, flags);
+		goto out_disk_cleanup;
+	}
 	WARN_ON(d->gd);
 	WARN_ON(d->flags & DEVFL_UP);
 	d->bufpool = mp;
@@ -427,7 +430,8 @@ err_mempool:
 err:
 	spin_lock_irqsave(&d->lock, flags);
 	d->flags &= ~DEVFL_GD_NOW;
-	queue_work(aoe_wq, &d->work);
+	if (!(d->flags & DEVFL_TKILL))
+		queue_work(aoe_wq, &d->work);
 	spin_unlock_irqrestore(&d->lock, flags);
 }
 
drivers/block/aoe/aoecmd.c +12 -3
index a4744a30a8af..c7f762180b9b 100644
@@ -787,7 +787,8 @@ rexmit_timer(struct timer_list *timer)
 			 */
 			list_splice(&flist, &d->factive[0]);
 			d->flags |= DEVFL_DEAD;
-			queue_work(aoe_wq, &d->work);
+			if (!(d->flags & DEVFL_TKILL))
+				queue_work(aoe_wq, &d->work);
 			goto out;
 		}
 
@@ -899,6 +900,13 @@ aoecmd_sleepwork(struct work_struct *work)
 {
 	struct aoedev *d = container_of(work, struct aoedev, work);
 
+	spin_lock_irq(&d->lock);
+	if (d->flags & DEVFL_TKILL) {
+		spin_unlock_irq(&d->lock);
+		return;
+	}
+	spin_unlock_irq(&d->lock);
+
 	if (d->flags & DEVFL_DEAD)
 		aoedev_downdev(d);
 
@@ -909,7 +917,8 @@ aoecmd_sleepwork(struct work_struct *work)
 		set_capacity_and_notify(d->gd, d->ssize);
 
 		spin_lock_irq(&d->lock);
-		d->flags |= DEVFL_UP;
+		if (!(d->flags & DEVFL_TKILL))
+			d->flags |= DEVFL_UP;
 		d->flags &= ~DEVFL_NEWSIZE;
 		spin_unlock_irq(&d->lock);
 	}
@@ -974,7 +983,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id)
 			d->fw_ver, (long long)ssize);
 	d->ssize = ssize;
 	d->geo.start = 0;
-	if (d->flags & (DEVFL_GDALLOC|DEVFL_NEWSIZE))
+	if (d->flags & (DEVFL_GDALLOC | DEVFL_NEWSIZE | DEVFL_TKILL))
 		return;
 	if (d->gd != NULL)
 		d->flags |= DEVFL_NEWSIZE;
drivers/block/aoe/aoedev.c +7 -1
index 6e8b3807c1c8..4eb2e68439db 100644
@@ -286,6 +286,7 @@ freedev(struct aoedev *d)
 		return;
 
 	timer_delete_sync(&d->timer);
+	cancel_work_sync(&d->work);
 	if (d->gd) {
 		aoedisk_rm_debugfs(d);
 		del_gendisk(d->gd);
@@ -311,6 +312,8 @@ enum flush_parms {
 	EXITING = 1,
 };
 
+static DEFINE_MUTEX(flush_mutex);
+
 static int
 flush(const char __user *str, size_t cnt, int exiting)
 {
@@ -333,6 +336,7 @@ flush(const char __user *str, size_t cnt, int exiting)
 			specified = 1;
 	}
 
+	mutex_lock(&flush_mutex);
 	flush_workqueue(aoe_wq);
 	/* pass one: do aoedev_downdev, which might sleep */
 restart1:
@@ -353,10 +357,11 @@ restart1:
 		|| d->ref)
 			goto cont;
 
+		d->flags |= DEVFL_TKILL;
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
+		cancel_work_sync(&d->work);
 		aoedev_downdev(d);
-		d->flags |= DEVFL_TKILL;
 		goto restart1;
 cont:
 		spin_unlock(&d->lock);
@@ -397,6 +402,7 @@ restart2:
 		kfree(doomed);
 	}
 	spin_unlock_irqrestore(&devlist_lock, flags);
+	mutex_unlock(&flush_mutex);
 
 	return 0;
 }
author maintainer
#38 lore raw

Re: [PATCH 18/98] aoe: don't free a device with its work queued

Quoted 37 lines · Bart Van Assche

drivers/block/aoe/aoedev.c flush(const char __user *str, size_t cnt, int exiting) · line 364

Quoted 8 lines · Jens Axboe
drivers/block/aoe/aoedev.c +2 -0
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
 		aoedev_downdev(d);
+		spin_lock_irqsave(&d->lock, flags);
 		d->flags |= DEVFL_TKILL;
+		spin_unlock_irqrestore(&d->lock, flags);
 		goto restart1;

Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race window open while aoedev_downdev(d) drops d->lock and sleeps in blk_mq_freeze_queue(d->blkq):

1. ataid_complete(), rexmit_timer(), or aoeblk_gdalloc()'s error path can still see !(d->flags & DEVFL_TKILL) and queue d->work during that window.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs before freedev() reaches cancel_work_sync(&d->work), it can call aoeblk_gdalloc() (hitting WARN_ON(d->flags & DEVFL_TKILL) if DEVFL_TKILL was set in the meantime) or set d->flags |= DEVFL_UP again after aoedev_downdev() cleared DEVFL_UP, letting new I/O enter and hanging del_gendisk() in freedev().

Instead of setting DEVFL_TKILL after aoedev_downdev(d), how about:

- Serializing flush() with a static flush_mutex so a second aoedev_flush() caller cannot enter pass 2 (freedev()) and pass 3 (kfree()) on a device while the first caller is still in pass 1.
- Setting d->flags |= DEVFL_TKILL under d->lock before unlocking, then running cancel_work_sync(&d->work) before aoedev_downdev(d) (and keeping cancel_work_sync(&d->work) after timer_delete_sync(&d->timer) in freedev()).
- Checking DEVFL_TKILL at all three queue_work(aoe_wq, &d->work) sites (ataid_complete(), rexmit_timer(), and aoeblk_gdalloc()), as well as in aoecmd_sleepwork() and aoeblk_gdalloc().

Quoted 2 lines

Here is a replacement patch against the same baseline (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.

Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.

Signature · 1 line
Jens Axboe
#39 lore raw

Re: [PATCH 18/98] aoe: don't free a device with its work queued

Quoted 5 lines · Jens Axboe

Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.

Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.

Done - see also https://lore.kernel.org/linux-block/[email protected]/.

As mentioned in the description of that patch, it has only been compile-tested.

Bart.

author maintainer
#40 lore raw

Re: [PATCH 18/98] aoe: don't free a device with its work queued

Quoted 11 lines · Bart Van Assche
Quoted 5 lines · Jens Axboe

Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.

Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.

Done - see also https://lore.kernel.org/linux-block/[email protected]/.

As mentioned in the description of that patch, it has only been compile-tested.

I would strongly believe that was also the extent of testing that happened on the previous patch it replaces, so not a big deal. It's not like any of us use aoe...

Signature · 1 line
Jens Axboe
author maintainer
#41 lore raw

PATCH 19/98 blk-cgroup: don't free a blkg with its punt work queued

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blkg_async_bio_workfn() relies on the bios on the list pinning the blkg. But blkcg_punt_bio_submit() can call queue_work() after a running instance already took and completed the bio, and queue_work() succeeds on an empty list for the same reason. Nothing holds the blkg then, and free_work shares its memory with async_bio_work on top of that.

Hold a reference across blkcg_punt_bio_submit(), give the two works memory of their own, and cancel async_bio_work in blkg_free_workfn().

Fixes d3f77dfdc718 ("blkcg: implement REQ_CGROUP_PUNT")
Cc Tejun Heo <[email protected]>
---
 block/blk-cgroup.c | 13 ++++++++++++-
 block/blk-cgroup.h |  6 ++----
 2 files changed, 14 insertions(+), 5 deletions(-)
block/blk-cgroup.c +12 -1
index c9187d95a54b..00da55c6ec43 100644
@@ -121,6 +121,11 @@ static void blkg_free_workfn(struct work_struct *work)
 	struct request_queue *q = blkg->q;
 	int i;
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	/* It may have been queued for bios which an earlier run has taken */
+	cancel_work_sync(&blkg->async_bio_work);
+#endif
+
 	/*
 	 * pd_free_fn() can also be called from blkcg_deactivate_policy(),
 	 * in order to make sure pd_free_fn() is called in order, the deletion
@@ -218,7 +223,7 @@ static void blkg_async_bio_workfn(struct work_struct *work)
 	struct blk_plug plug;
 	bool need_plug = false;
 
-	/* as long as there are pending bios, @blkg can't go away */
+	/* blkg_free_workfn() waits for us */
 	spin_lock(&blkg->async_bio_lock);
 	bio_list_merge_init(&bios, &blkg->async_bios);
 	spin_unlock(&blkg->async_bio_lock);
@@ -245,10 +250,16 @@ void blkcg_punt_bio_submit(struct bio *bio)
 	struct blkcg_gq *blkg = bio->bi_blkg;
 
 	if (blkg->parent) {
+		/*
+		 * Once it is on the list, a run which is under way can submit
+		 * @bio, and its reference goes when it completes.
+		 */
+		blkg_get(blkg);
 		spin_lock(&blkg->async_bio_lock);
 		bio_list_add(&blkg->async_bios, bio);
 		spin_unlock(&blkg->async_bio_lock);
 		queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work);
+		blkg_put(blkg);
 	} else {
 		/* never bounce for the root cgroup */
 		submit_bio(bio);
block/blk-cgroup.h +2 -4
index e67c69839129..1fd60cb4157c 100644
@@ -76,11 +76,9 @@ struct blkcg_gq {
 #ifdef CONFIG_BLK_CGROUP_PUNT_BIO
 	spinlock_t			async_bio_lock;
 	struct bio_list			async_bios;
+	struct work_struct		async_bio_work;
 #endif
-	union {
-		struct work_struct	async_bio_work;
-		struct work_struct	free_work;
-	};
+	struct work_struct		free_work;
 
 	atomic_t			use_delay;
 	atomic64_t			delay_nsec;
Signature · 1 line
2.56.0
maintainer Acked-by on v1 19/98
#42 lore raw

Re: [PATCH 19/98] blk-cgroup: don't free a blkg with its punt work queued

Quoted 6 lines · Jens Axboe

blkg_async_bio_workfn() relies on the bios on the list pinning the blkg. But blkcg_punt_bio_submit() can call queue_work() after a running instance already took and completed the bio, and queue_work() succeeds on an empty list for the same reason. Nothing holds the blkg then, and free_work shares its memory with async_bio_work on top of that.

Quoted 7 lines

Hold a reference across blkcg_punt_bio_submit(), give the two works memory of their own, and cancel async_bio_work in blkg_free_workfn().

Fixes d3f77dfdc718 ("blkcg: implement REQ_CGROUP_PUNT")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#43 lore raw

PATCH 20/98 mtip32xx: don't mark the disk dead in the irq handler

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

mtip_check_surprise_removal() calls blk_mark_disk_dead() from the interrupt handler, and that takes bd_holder_lock and calls into the holder. Only note the removal there and let the service thread tell the block layer.

Fixes f55e017c6420 ("block: add a mark_dead holder operation")
---
 drivers/block/mtip32xx/mtip32xx.c | 35 +++++++++++++++++++++++++------
 drivers/block/mtip32xx/mtip32xx.h |  3 +++
 2 files changed, 32 insertions(+), 6 deletions(-)
drivers/block/mtip32xx/mtip32xx.c +29 -6
index 113bdb868c46..58dc0d63df32 100644
@@ -130,7 +130,7 @@ struct mtip_compat_ide_task_request_s {
  * return value
  *	 true if device removed, else false
  */
-static bool mtip_check_surprise_removal(struct driver_data *dd)
+static bool mtip_device_removed(struct driver_data *dd)
 {
 	u16 vendor_id = 0;
 
@@ -141,14 +141,34 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)
 	pci_read_config_word(dd->pdev, 0x00, &vendor_id);
 	if (vendor_id == 0xFFFF) {
 		dd->sr = true;
-		if (dd->disk)
-			blk_mark_disk_dead(dd->disk);
 		return true; /* device removed */
 	}
 
 	return false; /* device present */
 }
 
+/* As above, and tell the block layer. May sleep. */
+static bool mtip_check_surprise_removal(struct driver_data *dd)
+{
+	if (!mtip_device_removed(dd))
+		return false;
+
+	if (dd->disk && !test_and_set_bit(MTIP_DDF_DISK_DEAD_BIT, &dd->dd_flag))
+		blk_mark_disk_dead(dd->disk);
+	return true;
+}
+
+/* For the interrupt handler: the service thread tells the block layer. */
+static bool mtip_check_surprise_removal_irq(struct driver_data *dd)
+{
+	if (!mtip_device_removed(dd))
+		return false;
+
+	set_bit(MTIP_PF_SR_BIT, &dd->port->flags);
+	wake_up_interruptible(&dd->port->svc_wait);
+	return true;
+}
+
 static struct mtip_cmd *mtip_cmd_from_tag(struct driver_data *dd,
 					  unsigned int tag)
 {
@@ -715,7 +735,7 @@ static inline void mtip_process_errors(struct driver_data *dd, u32 port_stat)
 		dev_warn(&dd->pdev->dev,
 			"Port stat errors %x unhandled\n",
 			(port_stat & ~PORT_IRQ_HANDLED));
-		if (mtip_check_surprise_removal(dd))
+		if (mtip_check_surprise_removal_irq(dd))
 			return;
 	}
 	if (likely(port_stat & (PORT_IRQ_TF_ERR | PORT_IRQ_IF_ERR))) {
@@ -740,7 +760,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data)
 		/* Acknowledge the interrupt status on the port.*/
 		port_stat = readl(port->mmio + PORT_IRQ_STAT);
 		if (unlikely(port_stat == 0xFFFFFFFF)) {
-			mtip_check_surprise_removal(dd);
+			mtip_check_surprise_removal_irq(dd);
 			return IRQ_HANDLED;
 		}
 		writel(port_stat, port->mmio + PORT_IRQ_STAT);
@@ -784,7 +804,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data)
 		}
 
 		if (unlikely(port_stat & PORT_IRQ_ERR)) {
-			if (unlikely(mtip_check_surprise_removal(dd))) {
+			if (unlikely(mtip_check_surprise_removal_irq(dd))) {
 				/* don't proceed further */
 				return IRQ_HANDLED;
 			}
@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
+			mtip_check_surprise_removal(dd);
+
 restart_eh:
 		/* Demux bits: start with error handling */
 		if (test_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags)) {
drivers/block/mtip32xx/mtip32xx.h +3 -0
index 0963c07b5845..1af2c75414a5 100644
@@ -134,11 +134,13 @@ enum {
 	MTIP_PF_ISSUE_CMDS_BIT      = 5,
 	MTIP_PF_REBUILD_BIT         = 6,
 	MTIP_PF_SVC_THD_STOP_BIT    = 8,
+	MTIP_PF_SR_BIT              = 11, /* surprise removal seen in irq */
 
 	MTIP_PF_SVC_THD_WORK	= ((1 << MTIP_PF_EH_ACTIVE_BIT) |
 				  (1 << MTIP_PF_ISSUE_CMDS_BIT) |
 				  (1 << MTIP_PF_REBUILD_BIT) |
 				  (1 << MTIP_PF_SVC_THD_STOP_BIT) |
+				  (1 << MTIP_PF_SR_BIT) |
 				  (1 << MTIP_PF_TO_ACTIVE_BIT)),
 
 	/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,6 +152,7 @@ enum {
 	MTIP_DDF_RESUME_BIT         = 6,
 	MTIP_DDF_INIT_DONE_BIT      = 7,
 	MTIP_DDF_REBUILD_FAILED_BIT = 8,
+	MTIP_DDF_DISK_DEAD_BIT      = 9,
 
 	MTIP_DDF_STOP_IO      = ((1 << MTIP_DDF_REMOVE_PENDING_BIT) |
 				(1 << MTIP_DDF_SEC_LOCK_BIT) |
Signature · 1 line
2.56.0
#44 lore raw

Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

drivers/block/mtip32xx/mtip32xx.c static int mtip_service_thread(void *data) · line 2518

Quoted 8 lines · Jens Axboe
drivers/block/mtip32xx/mtip32xx.c +3 -0
@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
+			mtip_check_surprise_removal(dd);
+
 restart_eh:

Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but calling mtip_check_surprise_removal(dd) -> blk_mark_disk_dead(dd->disk) in mtip_service_thread() before aborting in-flight NCQ commands can deadlock when a filesystem is mounted on the device:

1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which locks sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If another task already holds s_umount while waiting for in-flight I/O on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() then calls mtip_stop_service_thread(dd) (kthread_stop()) before calling blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd), so mtip_pci_remove() blocks waiting for mtip_service_thread() before it can abort the in-flight commands holding up s_umount.

Quiescing dd->queue and aborting in-flight commands via blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd) before calling blk_mark_disk_dead(dd->disk) on surprise removal would avoid that deadlock.

#45 lore raw

Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

In reply to Jens Axboe

Review static int mtip_service_thread(void *data) · line 2518

Quoted 8 lines · Jens Axboe
(quoted diff) +3 -0
@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
+			mtip_check_surprise_removal(dd);
+
 restart_eh:

Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but calling blk_mark_disk_dead(dd->disk) in mtip_service_thread() before aborting in-flight NCQ commands can deadlock when a filesystem is mounted on the device:

1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which takes sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If another task already holds s_umount while waiting for in-flight I/O on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() calls mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands, so mtip_pci_remove() blocks waiting for mtip_service_thread() before it can abort the commands holding up s_umount.

Also, mtip_abort_cmd() currently calls mtip_softirq_done_fn(req) directly without transitioning the request through blk_mq_complete_request().

How about keeping mtip_check_surprise_removal() non-sleeping everywhere (setting dd->sr, a new MTIP_DDF_DEAD_BIT to fail new normal and reserved requests immediately with BLK_STS_IOERR, and waking the service thread with MTIP_PF_SR_BIT), and having a helper mtip_handle_surprise_removal() quiesce dd->queue and abort in-flight requests via blk_mq_tagset_busy_iter() + blk_mq_complete_request() before calling blk_mark_disk_dead(dd->disk)?

Patch 21/98 still applies cleanly on top of this replacement, and my reply to patch 22/98 has the matching follow-up change. Checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested on mtip32xx hardware:

---

drivers/block/mtip32xx/mtip32xx.c +35 -3
index 113bdb868c46..35568d0e74cf 100644
@@ -141,8 +141,11 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)
 	pci_read_config_word(dd->pdev, 0x00, &vendor_id);
 	if (vendor_id == 0xFFFF) {
 		dd->sr = true;
-		if (dd->disk)
-			blk_mark_disk_dead(dd->disk);
+		set_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag);
+		if (dd->port) {
+			set_bit(MTIP_PF_SR_BIT, &dd->port->flags);
+			wake_up_interruptible(&dd->port->svc_wait);
+		}
 		return true; /* device removed */
 	}
 
@@ -2438,9 +2441,12 @@ static bool mtip_abort_cmd(struct request *req, void *data)
 
 	dbg_printk(MTIP_DRV_NAME " Aborting request, tag = %d\n", req->tag);
 
+	if (blk_mq_request_completed(req))
+		return true;
+
 	clear_bit(req->tag, dd->port->cmds_to_issue);
 	cmd->status = BLK_STS_IOERR;
-	mtip_softirq_done_fn(req);
+	blk_mq_complete_request(req);
 	return true;
 }
 
@@ -2462,6 +2468,18 @@ static bool mtip_queue_cmd(struct request *req, void *data)
  *	0
  */
 
+static void mtip_handle_surprise_removal(struct driver_data *dd)
+{
+	if (dd->queue) {
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag) &&
+	    !test_and_set_bit(MTIP_DDF_DISK_DEAD_BIT, &dd->dd_flag))
+		blk_mark_disk_dead(dd->disk);
+}
+
 static int mtip_service_thread(void *data)
 {
 	struct driver_data *dd = (struct driver_data *)data;
@@ -2492,6 +2510,15 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags)) {
+			mtip_handle_surprise_removal(dd);
+			clear_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags);
+			clear_bit(MTIP_PF_TO_ACTIVE_BIT, &port->flags);
+			clear_bit(MTIP_PF_ISSUE_CMDS_BIT, &port->flags);
+			clear_bit(MTIP_PF_REBUILD_BIT, &port->flags);
+			continue;
+		}
+
 restart_eh:
 		/* Demux bits: start with error handling */
 		if (test_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags)) {
@@ -3241,6 +3268,8 @@ static inline bool is_stopped(struct driver_data *dd, struct request *rq)
 		return true;
 	if (test_bit(MTIP_DDF_REBUILD_FAILED_BIT, &dd->dd_flag))
 		return true;
+	if (test_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag))
+		return true;
 
 	return false;
 }
@@ -3279,6 +3308,9 @@ static blk_status_t mtip_issue_reserved_cmd(struct blk_mq_hw_ctx *hctx,
 		dd->port->command_list + sizeof(struct mtip_cmd_hdr) * rq->tag;
 	struct mtip_cmd_sg *command_sg;
 
+	if (test_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag))
+		return BLK_STS_IOERR;
+
 	if (mtip_commands_active(dd->port))
 		return BLK_STS_DEV_RESOURCE;
 
drivers/block/mtip32xx/mtip32xx.h +6 -1
index 0963c07b5845..af0249e3c550 100644
@@ -134,11 +134,13 @@ enum {
 	MTIP_PF_ISSUE_CMDS_BIT      = 5,
 	MTIP_PF_REBUILD_BIT         = 6,
 	MTIP_PF_SVC_THD_STOP_BIT    = 8,
+	MTIP_PF_SR_BIT              = 11, /* surprise removal seen */
 
 	MTIP_PF_SVC_THD_WORK	= ((1 << MTIP_PF_EH_ACTIVE_BIT) |
 				  (1 << MTIP_PF_ISSUE_CMDS_BIT) |
 				  (1 << MTIP_PF_REBUILD_BIT) |
 				  (1 << MTIP_PF_SVC_THD_STOP_BIT) |
+				  (1 << MTIP_PF_SR_BIT) |
 				  (1 << MTIP_PF_TO_ACTIVE_BIT)),
 
 	/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,12 +152,15 @@ enum {
 	MTIP_DDF_RESUME_BIT         = 6,
 	MTIP_DDF_INIT_DONE_BIT      = 7,
 	MTIP_DDF_REBUILD_FAILED_BIT = 8,
+	MTIP_DDF_DEAD_BIT           = 9,
+	MTIP_DDF_DISK_DEAD_BIT      = 10,
 
 	MTIP_DDF_STOP_IO      = ((1 << MTIP_DDF_REMOVE_PENDING_BIT) |
 				(1 << MTIP_DDF_SEC_LOCK_BIT) |
 				(1 << MTIP_DDF_OVER_TEMP_BIT) |
 				(1 << MTIP_DDF_WRITE_PROTECT_BIT) |
-				(1 << MTIP_DDF_REBUILD_FAILED_BIT)),
+				(1 << MTIP_DDF_REBUILD_FAILED_BIT) |
+				(1 << MTIP_DDF_DEAD_BIT)),
 
 };
 
#46 lore raw

Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

In reply to Jens Axboe

Not objections against the quick fixes, but maybe we should use as a pretext to remove the driver? The hardware is also supported by ahci (with reduced performance) and the driver is unmaintained and pretty gross.

author maintainer
#47 lore raw

PATCH 21/98 mtip32xx: abort timed out commands of a removed device

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

mtip_device_reset() returns 0 for a removed device, so the timeout handler takes that for a successful reset and reissues the commands to registers that are no longer there, forever. Return -ENODEV so they get aborted instead.

Fixes abb0ccd185c9 ("mtip32xx: Implement timeout handler")
---
 drivers/block/mtip32xx/mtip32xx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/mtip32xx/mtip32xx.c +1 -1
index 58dc0d63df32..ee5b170e767c 100644
@@ -456,7 +456,7 @@ static int mtip_device_reset(struct driver_data *dd)
 	int rv = 0;
 
 	if (mtip_check_surprise_removal(dd))
-		return 0;
+		return -ENODEV;
 
 	if (mtip_hba_reset(dd) < 0)
 		rv = -EFAULT;
Signature · 1 line
2.56.0
author maintainer
#48 lore raw

PATCH 22/98 mtip32xx: abort commands in flight on surprise removal

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

mtip_pci_remove() relies on blk_mark_disk_dead(), which only stops new IO. del_gendisk() waits for the requests the driver still owns, and nothing completes them for a removed device: the timeout handler leaves them to the service thread, which exits as soon as it sees MTIP_DDF_REMOVE_PENDING_BIT. The removal hangs.

Stop the service thread and abort the commands in flight before del_gendisk() if the device is gone.

Fixes e8b58ef09e84 ("mtip32xx: fix device removal")
---
 drivers/block/mtip32xx/mtip32xx.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
drivers/block/mtip32xx/mtip32xx.c +23 -5
index ee5b170e767c..301c3db21ca8 100644
@@ -3887,6 +3887,17 @@ static int mtip_pci_probe(struct pci_dev *pdev,
 	return rv;
 }
 
+static void mtip_stop_service_thread(struct driver_data *dd)
+{
+	if (!dd->mtip_svc_handler)
+		return;
+
+	set_bit(MTIP_PF_SVC_THD_STOP_BIT, &dd->port->flags);
+	wake_up_interruptible(&dd->port->svc_wait);
+	kthread_stop(dd->mtip_svc_handler);
+	dd->mtip_svc_handler = NULL;
+}
+
 /*
  * Called for each probed device when the device is removed or the
  * driver is unloaded.
@@ -3917,16 +3928,23 @@ static void mtip_pci_remove(struct pci_dev *pdev)
 	set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag);
 	mutex_lock(&dd->ioctl_mutex);
 
+	/*
+	 * Nobody will complete the commands a removed device has been given,
+	 * and del_gendisk() waits for them.
+	 */
+	if (dd->sr) {
+		mtip_stop_service_thread(dd);
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);
 
 	mtip_hw_debugfs_exit(dd);
 
-	if (dd->mtip_svc_handler) {
-		set_bit(MTIP_PF_SVC_THD_STOP_BIT, &dd->port->flags);
-		wake_up_interruptible(&dd->port->svc_wait);
-		kthread_stop(dd->mtip_svc_handler);
-	}
+	mtip_stop_service_thread(dd);
 
 	if (!dd->sr) {
 		/*
Signature · 1 line
2.56.0
#49 lore raw

Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal

drivers/block/mtip32xx/mtip32xx.c static void mtip_pci_remove(struct pci_dev *pdev) · line 3943

Quoted 13 lines · Jens Axboe
drivers/block/mtip32xx/mtip32xx.c +11 -0
+	/*
+	 * Nobody will complete the commands a removed device has been given,
+	 * and del_gendisk() waits for them.
+	 */
+	if (dd->sr) {
+		mtip_stop_service_thread(dd);
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);

Two issues with this ordering:

1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands can deadlock with patch 20/98 if mtip_service_thread() is blocked inside blk_mark_disk_dead() -> fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss occurs after mtip_pci_remove() has already checked if (dd->sr) (false) and entered del_gendisk(dd->disk), MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread() then exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT without handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving in-flight commands uncompleted until the request timer fires.

If mtip_service_thread() aborts in-flight commands when handling MTIP_PF_SR_BIT (before blk_mark_disk_dead()) and does not exit early on MTIP_DDF_REMOVE_PENDING_BIT while surprise-removal/timeout work is pending, mtip_stop_service_thread() can stay after del_gendisk() and both races are closed.

#50 lore raw

Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal

In reply to Jens Axboe

Review

Quoted 13 lines · Jens Axboe
(quoted diff) +11 -0
+	/*
+	 * Nobody will complete the commands a removed device has been given,
+	 * and del_gendisk() waits for them.
+	 */
+	if (dd->sr) {
+		mtip_stop_service_thread(dd);
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);

Two issues with this ordering:

1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands deadlocks with patch 20/98 if mtip_service_thread() is blocked inside blk_mark_disk_dead() -> fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss happens after mtip_pci_remove() has already checked dd->sr (false) and entered del_gendisk(dd->disk), MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread() exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT without handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving in-flight commands uncompleted forever in del_gendisk().

With the replacement proposed for patch 20/98 (where mtip_handle_surprise_removal() aborts in-flight commands before calling blk_mark_disk_dead()), we can instead keep mtip_service_thread() running until kthread_stop() after del_gendisk(dd->disk) by removing its early exit on MTIP_DDF_REMOVE_PENDING_BIT (and clearing MTIP_PF_REBUILD_BIT if remove is pending or mtip_ftl_rebuild_poll(dd) <= 0, since rebuild poll returns -EFAULT once MTIP_DDF_REMOVE_PENDING_BIT is set). That handles surprise removal both before and during del_gendisk() from the service thread without racing two tagset iterations.

Below is the replacement for 22/98 on top of the 20/98 replacement + unmodified 21/98 (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested on mtip32xx hardware):

---

drivers/block/mtip32xx/mtip32xx.c +6 -5
index 757f9b74871e..b9dfa7112c95 100644
@@ -2504,10 +2504,6 @@ static int mtip_service_thread(void *data)
 			test_bit(MTIP_PF_SVC_THD_STOP_BIT, &port->flags))
 			goto st_out;
 
-		if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT,
-				&dd->dd_flag)))
-			goto st_out;
-
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
 		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags)) {
@@ -2588,7 +2584,9 @@ restart_eh:
 		}
 
 		if (test_bit(MTIP_PF_REBUILD_BIT, &port->flags)) {
-			if (mtip_ftl_rebuild_poll(dd) == 0)
+			if (test_bit(MTIP_DDF_REMOVE_PENDING_BIT,
+				     &dd->dd_flag) ||
+			    mtip_ftl_rebuild_poll(dd) <= 0)
 				clear_bit(MTIP_PF_REBUILD_BIT, &port->flags);
 		}
 	}
@@ -3926,6 +3924,9 @@ static void mtip_pci_remove(struct pci_dev *pdev)
 	set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag);
 	mutex_lock(&dd->ioctl_mutex);
 
+	if (dd->sr && !dd->mtip_svc_handler)
+		mtip_handle_surprise_removal(dd);
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);
 
author maintainer
#51 lore raw

PATCH 23/98 blk-iolatency: flush enable_work after going offline

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blkcg_iolatency_exit() flushes enable_work before it deactivates the policy, but iolatency_pd_offline() schedules it again for the last group with a target, so the blk_iolatency is freed with the work queued. Flush it after the deactivation.

Fixes 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")
Cc Tejun Heo <[email protected]>
---
 block/blk-iolatency.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-iolatency.c +2 -1
index 2caa79a008ad..c15b1aba606c 100644
@@ -637,8 +637,9 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos)
 	struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);
 
 	timer_shutdown_sync(&blkiolat->timer);
-	flush_work(&blkiolat->enable_work);
 	blkcg_deactivate_policy(rqos->disk, &blkcg_policy_iolatency);
+	/* iolatency_pd_offline() schedules it for the last group with a target */
+	flush_work(&blkiolat->enable_work);
 	kfree(blkiolat);
 }
 
Signature · 1 line
2.56.0
maintainer Acked-by on v1 23/98
#52 lore raw

Re: [PATCH 23/98] blk-iolatency: flush enable_work after going offline

Quoted 9 lines · Jens Axboe

blkcg_iolatency_exit() flushes enable_work before it deactivates the policy, but iolatency_pd_offline() schedules it again for the last group with a target, so the blk_iolatency is freed with the work queued. Flush it after the deactivation.

Fixes 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#53 lore raw

PATCH 24/98 zram: accept lz4 dictionaries of more than 64 KiB

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

LZ4_loadDict() and LZ4_loadDictHC() only use the last 64KiB of a dictionary and return how much they used, which the zram backends take for an error. Expect that instead.

Fixes fb4f644ee8da ("zram: add dictionary support to lz4")
Fixes 1e673c8cf7f9 ("zram: add dictionary support to lz4hc")
Cc Minchan Kim <[email protected]>
Cc Sergey Senozhatsky <[email protected]>
---
 drivers/block/zram/backend_lz4.c   | 4 +++-
 drivers/block/zram/backend_lz4hc.c | 4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/block/zram/backend_lz4.c +3 -1
index 1e4ad31d39a6..15db3292e19e 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
drivers/block/zram/backend_lz4hc.c +3 -1
index d8aa01bb258f..d163ff0ae8ae 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;
 		ret = LZ4_compress_HC_continue(zctx->cstrm, req->src, req->dst,
 					       req->src_len, req->dst_len);
Signature · 1 line
2.56.0
#54 lore raw

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

On (26/10/28 10:41), Jens Axboe wrote:
[..]

Quoted 29 lines

#include <linux/kernel.h>
#include <linux/lz4.h>

drivers/block/zram/backend_lz4.c +3 -1
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
drivers/block/zram/backend_lz4hc.c +1 -0
index d8aa01bb258f..d163ff0ae8ae 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102

Quoted 7 lines
drivers/block/zram/backend_lz4hc.c +2 -1
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;

I think this should patch lz4hc_setup_params() instead. Similarly to lz4 hunk.

author maintainer
#55 lore raw

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

Quoted 30 lines · Sergey Senozhatsky

On (26/10/28 10:41), Jens Axboe wrote:
[..]

Quoted 28 lines

#include <linux/kernel.h>
#include <linux/lz4.h>

drivers/block/zram/backend_lz4.c +3 -1
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
drivers/block/zram/backend_lz4hc.c +1 -0
index d8aa01bb258f..d163ff0ae8ae 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
Quoted 11 lines

drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102

Quoted 8 lines
drivers/block/zram/backend_lz4hc.c +2 -1
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;

I think this should patch lz4hc_setup_params() instead. Similarly to lz4 hunk.

You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.

Signature · 1 line
Jens Axboe
#56 lore raw

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

Quoted 43 lines · Jens Axboe
Quoted 30 lines · Sergey Senozhatsky

On (26/10/28 10:41), Jens Axboe wrote:
[..]

Quoted 28 lines

#include <linux/kernel.h>
#include <linux/lz4.h>

drivers/block/zram/backend_lz4.c +3 -1
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
drivers/block/zram/backend_lz4hc.c +1 -0
index d8aa01bb258f..d163ff0ae8ae 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
Quoted 11 lines

drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102

Quoted 8 lines
drivers/block/zram/backend_lz4hc.c +2 -1
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;

I think this should patch lz4hc_setup_params() instead. Similarly to lz4 hunk.

Quoted 4 lines

You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.

Works for me, I'll queue up a fix (will give you credit).

author maintainer
#57 lore raw

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

Quoted 44 lines · Sergey Senozhatsky
Quoted 43 lines · Jens Axboe
Quoted 41 lines · Sergey Senozhatsky

On (26/10/28 10:41), Jens Axboe wrote:
[..]

Quoted 29 lines

#include <linux/kernel.h>
#include <linux/lz4.h>

drivers/block/zram/backend_lz4.c +3 -1
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
drivers/block/zram/backend_lz4hc.c +1 -0
index d8aa01bb258f..d163ff0ae8ae 100644
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102

Quoted 7 lines
drivers/block/zram/backend_lz4hc.c +2 -1
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;

I think this should patch lz4hc_setup_params() instead. Similarly to lz4 hunk.

Quoted 6 lines
Quoted 4 lines

You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.

Works for me, I'll queue up a fix (will give you credit).

Perfect, thank you! I'll drop this one.

Signature · 1 line
Jens Axboe
author maintainer
#58 lore raw

PATCH 25/98 blk-throttle: pass the I/O counts of a dead group on

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The recursive io_service_bytes and io_serviced files sum up the online groups plus what dead descendants left in aux_cnt. That worked while the counters lived in the blkg. Since commit 7ca464383aec they're in the throtl_grp and nobody passes them up, so the files go backwards when a child is removed. Add them to the parent in throtl_pd_offline().

Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc Tejun Heo <[email protected]>
---
 block/blk-throttle.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
block/blk-throttle.c +12 -1
index 4f10fe05744b..80a5e4b2872f 100644
@@ -1670,7 +1670,18 @@ static void tg_flush_bios(struct throtl_grp *tg)
 
 static void throtl_pd_offline(struct blkg_policy_data *pd)
 {
-	tg_flush_bios(pd_to_tg(pd));
+	struct throtl_grp *tg = pd_to_tg(pd);
+	struct throtl_grp *parent = blkg_to_tg(tg_to_blkg(tg)->parent);
+
+	tg_flush_bios(tg);
+
+	/* the recursive files of the ancestors go on counting what it did */
+	if (parent) {
+		blkg_rwstat_add_aux(&parent->stat_bytes, &tg->stat_bytes);
+		blkg_rwstat_add_aux(&parent->stat_ios, &tg->stat_ios);
+		blkg_rwstat_reset(&tg->stat_bytes);
+		blkg_rwstat_reset(&tg->stat_ios);
+	}
 }
 
 struct blkcg_policy blkcg_policy_throtl = {
Signature · 1 line
2.56.0
maintainer Acked-by on v1 25/98
#59 lore raw

Re: [PATCH 25/98] blk-throttle: pass the I/O counts of a dead group on

Quoted 10 lines · Jens Axboe

The recursive io_service_bytes and io_serviced files sum up the online groups plus what dead descendants left in aux_cnt. That worked while the counters lived in the blkg. Since commit 7ca464383aec they're in the throtl_grp and nobody passes them up, so the files go backwards when a child is removed. Add them to the parent in throtl_pd_offline().

Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#60 lore raw

PATCH 26/98 blk-throttle: make blkio.reset_stats reset I/O counts

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blkio.throttle.io_service_bytes and io_serviced were reset by blkcg_reset_stats() while the counters were part of the blkg. Now that they live in the throtl_grp, add a pd_reset_stats_fn for them.

Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc Tejun Heo <[email protected]>
---
 block/blk-throttle.c | 9 +++++++++
 1 file changed, 9 insertions(+)
block/blk-throttle.c +9 -0
index 80a5e4b2872f..c46807a0faab 100644
@@ -1684,6 +1684,14 @@ static void throtl_pd_offline(struct blkg_policy_data *pd)
 	}
 }
 
+static void throtl_pd_reset_stats(struct blkg_policy_data *pd)
+{
+	struct throtl_grp *tg = pd_to_tg(pd);
+
+	blkg_rwstat_reset(&tg->stat_bytes);
+	blkg_rwstat_reset(&tg->stat_ios);
+}
+
 struct blkcg_policy blkcg_policy_throtl = {
 	.dfl_cftypes		= throtl_files,
 	.legacy_cftypes		= throtl_legacy_files,
@@ -1693,6 +1701,7 @@ struct blkcg_policy blkcg_policy_throtl = {
 	.pd_online_fn		= throtl_pd_online,
 	.pd_offline_fn		= throtl_pd_offline,
 	.pd_free_fn		= throtl_pd_free,
+	.pd_reset_stats_fn	= throtl_pd_reset_stats,
 };
 
 static void tg_cancel_writeback_bios(struct throtl_grp *tg,
Signature · 1 line
2.56.0
maintainer Acked-by on v1 26/98
#61 lore raw

Re: [PATCH 26/98] blk-throttle: make blkio.reset_stats reset I/O counts

Quoted 8 lines · Jens Axboe

blkio.throttle.io_service_bytes and io_serviced were reset by blkcg_reset_stats() while the counters were part of the blkg. Now that they live in the throtl_grp, add a pd_reset_stats_fn for them.

Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#62 lore raw

PATCH 27/98 blk-mq: don't trust the cookie in blk_mq_poll()

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_poll() indexes queue_hw_ctx[] with bio->bi_cookie unchecked. Zone write plugging keeps the segment count of a plugged write in __bi_nr_segments, which shares a union with the cookie, and clears REQ_POLLED, but bio_poll() only looks at the cookie. A plugged polled write with more segments than there are hardware queues reads past the array. The bio is also SLAB_TYPESAFE_BY_RCU, so a cookie from a bio reused for another device can meet the wrong array as well.

Ignore a cookie that is out of range or doesn't name a poll queue.

Fixes 3e08773c3841 ("block: switch polling to be bio based")
Fixes dd291d77cc90 ("block: Introduce zone write plugging")
---
 block/blk-mq.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
block/blk-mq.c +14 -1
index c91306f8156f..5ebe2e23df5c 100644
@@ -5245,9 +5245,22 @@ static int blk_hctx_poll(struct request_queue *q, struct blk_mq_hw_ctx *hctx,
 int blk_mq_poll(struct request_queue *q, blk_qc_t cookie,
 		struct io_comp_batch *iob, unsigned int flags)
 {
+	struct blk_mq_hw_ctx *hctx;
+
 	if (!blk_mq_can_poll(q))
 		return 0;
-	return blk_hctx_poll(q, q->queue_hw_ctx[cookie], iob, flags);
+
+	/*
+	 * The cookie is no more than a hint. iocb_bio_iopoll() may have read it
+	 * from a bio that was reused for another queue right afterwards, and
+	 * zone write plugging keeps the number of segments in the same field.
+	 */
+	if (cookie >= q->nr_hw_queues)
+		return 0;
+	hctx = q->queue_hw_ctx[cookie];
+	if (hctx->type != HCTX_TYPE_POLL)
+		return 0;
+	return blk_hctx_poll(q, hctx, iob, flags);
 }
 
 int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
Signature · 1 line
2.56.0
author maintainer
#64 lore raw

PATCH 28/98 blk-iocost: check again that the disk is alive

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blkg_conf_open_bdev() checks that the disk is alive under rq_qos_mutex, but ioc_qos_write() drops the mutex to freeze the queue first. If del_gendisk() runs in between, the tag set may be gone by the time blk_mq_quiesce_queue() looks at it. Check disk_live() again after retaking the mutex.

Fixes 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
Cc Tejun Heo <[email protected]>
---
 block/blk-iocost.c | 5 +++++
 1 file changed, 5 insertions(+)
block/blk-iocost.c +5 -0
index 7b5d865d8c14..bfb0dd320c4a 100644
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}
 	if (!queue_is_mq(disk->queue)) {
 		ret = -EOPNOTSUPP;
 		goto close_bdev;
Signature · 1 line
2.56.0
#65 lore raw

Re: [PATCH 28/98] blk-iocost: check again that the disk is alive

block/blk-iocost.c static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3329

Quoted 9 lines · Jens Axboe
block/blk-iocost.c +5 -0
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}

Look at the close_bdev unwind path in ioc_qos_write():

close_bdev:

	q = ctx.bdev->bd_queue;
	blkg_conf_close_bdev(&ctx);
	blk_mq_unfreeze_queue(q, memflags);

blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because del_gendisk() (and potentially put_disk()) ran while rq_qos_mutex was dropped, ctx.bdev may hold the final reference to the bdev/gendisk. blkdev_put_no_open(ctx.bdev) will then run disk_release() -> blk_put_queue(q) and free q while it is still frozen, before the next line calls blk_mq_unfreeze_queue(q, memflags) on the freed queue.

q->rq_qos_mutex needs to be unlocked and blk_mq_unfreeze_queue(q, memflags) called before blkdev_put_no_open(ctx.bdev) drops the bdev reference.

#66 lore raw

Re: [PATCH 28/98] blk-iocost: check again that the disk is alive

In reply to Jens Axboe

Review static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3329

Quoted 9 lines · Jens Axboe
(quoted diff) +5 -0
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}

Look at the close_bdev unwind path in ioc_qos_write():

close_bdev:

	q = ctx.bdev->bd_queue;
	blkg_conf_close_bdev(&ctx);
	blk_mq_unfreeze_queue(q, memflags);

blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because del_gendisk() and put_disk() ran while rq_qos_mutex was dropped, ctx.bdev can hold the last reference to the bdev/gendisk. Calling blkg_conf_close_bdev(&ctx) before blk_mq_unfreeze_queue(q, memflags) will then run disk_release() -> blk_put_queue(q) and free q while it is still frozen, followed by a use-after-free in blk_mq_unfreeze_queue().

Since __blk_mq_unfreeze_queue() acquires q->mq_freeze_lock (which nests outside q->rq_qos_mutex), close_bdev should unlock q->rq_qos_mutex, call blk_mq_unfreeze_queue(q, memflags) while ctx.bdev still pins the disk and queue, and only then call blkdev_put_no_open(ctx.bdev).

Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

block/blk-iocost.c +7 -1
index 7b5d865d8c14..df713bbc790b 100644
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}
 	if (!queue_is_mq(disk->queue)) {
 		ret = -EOPNOTSUPP;
 		goto close_bdev;
@@ -3437,8 +3442,9 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 close_bdev:
 	q = ctx.bdev->bd_queue;
-	blkg_conf_close_bdev(&ctx);
+	mutex_unlock(&q->rq_qos_mutex);
 	blk_mq_unfreeze_queue(q, memflags);
+	blkdev_put_no_open(ctx.bdev);
 	return ret ?: nbytes;
 
 einval:
author maintainer
#67 lore raw

PATCH 29/98 blk-mq: keep sharing tags fairly after a queue update

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A hctx only gets BLK_MQ_F_TAG_QUEUE_SHARED from blk_mq_add_queue_tag_set(). The hctxs blk_mq_update_nr_hw_queues() allocates for queues that joined long ago never get it, so they aren't counted in active_queues and hctx_may_queue() lets them take all the tags. One NVMe namespace can starve the others after the first update.

Set the flag on the new hctxs.

Fixes 868f2f0b7206 ("blk-mq: dynamic h/w context count")
---
 block/blk-mq.c | 3 +++
 1 file changed, 3 insertions(+)
block/blk-mq.c +3 -0
index 5ebe2e23df5c..6d3c60b4ea2b 100644
@@ -5184,6 +5184,9 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 			goto fallback;
 		}
 		blk_mq_map_swqueue(q);
+		/* The new hctxs don't know yet that they share their tags */
+		if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
+			queue_set_hctx_shared(q, true);
 	}
 switch_back:
 	/* The blk_mq_elv_switch_back unfreezes queue for us. */
Signature · 1 line
2.56.0
author maintainer
#69 lore raw

PATCH 30/98 blk-mq: don't bring back flags of a completed request

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The timeout handler sets and clears RQF_TIMED_OUT in a request it only holds a reference to, with a plain read-modify-write, while the completion clears flags in rq_flags the same way. If the two interleave, blk_mq_finish_request() runs twice with the flags back: the zone write plug reference is dropped twice and the scheduler's ->finish_request() is called again.

Change RQF_TIMED_OUT with cmpxchg, so the handler never writes back a flag that changed under it.

Fixes e5c0ca13659e ("blk-mq: release scheduler resource when request completes")
---
 block/blk-mq.c      |  2 +-
 block/blk-timeout.c |  3 ++-
 block/blk.h         | 14 ++++++++++++++
 3 files changed, 17 insertions(+), 2 deletions(-)
block/blk-mq.c +1 -1
index 6d3c60b4ea2b..0491e9821322 100644
@@ -1639,7 +1639,7 @@ EXPORT_SYMBOL_GPL(blk_mq_queue_inflight);
 
 static void blk_mq_rq_timed_out(struct request *req)
 {
-	req->rq_flags |= RQF_TIMED_OUT;
+	blk_rq_change_timed_out(req, true);
 	if (req->q->mq_ops->timeout) {
 		enum blk_eh_timer_return ret;
 
block/blk-timeout.c +2 -1
index 1b8de0417fc1..0590720e38c7 100644
@@ -137,7 +137,8 @@ void blk_add_timer(struct request *req)
 	if (!req->timeout)
 		req->timeout = q->rq_timeout;
 
-	req->rq_flags &= ~RQF_TIMED_OUT;
+	if (req->rq_flags & RQF_TIMED_OUT)
+		blk_rq_change_timed_out(req, false);
 
 	expiry = jiffies + req->timeout;
 	WRITE_ONCE(req->deadline, expiry);
block/blk.h +14 -0
index a48665e10994..de3a1633403b 100644
@@ -365,6 +365,20 @@ static inline void bio_integrity_free(struct bio *bio)
 unsigned long blk_rq_timeout(unsigned long timeout);
 void blk_add_timer(struct request *req);
 
+/*
+ * The timeout handler changes RQF_TIMED_OUT in a request which it does not
+ * own: whoever completes it changes rq_flags at the same time, with plain
+ * loads and stores. Never write back a flag that has changed under us.
+ */
+static inline void blk_rq_change_timed_out(struct request *rq, bool set)
+{
+	req_flags_t old = READ_ONCE(rq->rq_flags), new;
+
+	do {
+		new = set ? old | RQF_TIMED_OUT : old & ~RQF_TIMED_OUT;
+	} while (!try_cmpxchg(&rq->rq_flags, &old, new));
+}
+
 enum bio_merge_status {
 	BIO_MERGE_OK,
 	BIO_MERGE_NONE,
Signature · 1 line
2.56.0
Reviewed-by on v1 30/98
#70 lore raw

Re: [PATCH 30/98] blk-mq: don't bring back flags of a completed request

block/blk.h static inline void bio_integrity_free(struct bio *bio) · line 380

Quoted 8 lines · Jens Axboe
block/blk.h +8 -0
+static inline void blk_rq_change_timed_out(struct request *rq, bool set)
+{
+	req_flags_t old = READ_ONCE(rq->rq_flags), new;
+
+	do {
+		new = set ? old | RQF_TIMED_OUT : old & ~RQF_TIMED_OUT;
+	} while (!try_cmpxchg(&rq->rq_flags, &old, new));
+}

I've seen this pattern so often lately that a generic helper might be in order.

Otherwise looks good:

author maintainer
#71 lore raw

PATCH 31/98 blk-mq: don't forget a timer the driver has restarted

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

If ->timeout() returns BLK_EH_RESET_TIMER, blk_add_timer() gives the request a new deadline that blk_mq_timeout_work() doesn't know about when it picks the next expiry. A 2s nbd read next to a 60s one expires once and then not again for a minute. Let the new deadline take part.

Fixes 320ae51feed5 ("blk-mq: new multi-queue block IO queueing mechanism")
---
 block/blk-mq.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
block/blk-mq.c +4 -1
index 0491e9821322..6d3ea97f9a00 100644
@@ -1710,8 +1710,11 @@ static bool blk_mq_handle_expired(struct request *rq, void *priv)
 {
 	struct blk_expired_data *expired = priv;
 
-	if (blk_mq_req_expired(rq, expired))
+	if (blk_mq_req_expired(rq, expired)) {
 		blk_mq_rq_timed_out(rq);
+		/* If its timer has been started again, count the new deadline */
+		blk_mq_req_expired(rq, expired);
+	}
 	return true;
 }
 
Signature · 1 line
2.56.0
#72 lore raw

Re: [PATCH 31/98] blk-mq: don't forget a timer the driver has restarted

block/blk-mq.c static bool blk_mq_handle_expired(struct request *rq, void *priv) · line 1715

Quoted 3 lines · Jens Axboe
block/blk-mq.c +2 -0
+	if (blk_mq_req_expired(rq, expired)) {
 		blk_mq_rq_timed_out(rq);
+		/* If its timer has been started again, count the new deadline */

Comment nit: if you capitalize the first word, it is a sentence and needs to end with a period. Also an overly long line.

Otherwise looks good.

author maintainer
#73 lore raw

PATCH 32/98 blk-mq-dma: initialize the dma state for P2P, too

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_rq_dma_map_iter_start() promises that the caller need not initialize the dma_iova_state, but the P2P bus address path returns before it does so. The state then holds what the last request with that tag left, and blk_rq_dma_map_coalesce() describes a transfer as one IOVA range that it isn't. Clear it on that path too.

Fixes 858299dc6160 ("block: add scatterlist-less DMA mapping helpers")
---
 block/blk-mq-dma.c | 2 ++
 1 file changed, 2 insertions(+)
block/blk-mq-dma.c +2 -0
index 88fd9cbc951f..e0e7a18790b5 100644
@@ -181,6 +181,8 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
 	switch (pci_p2pdma_state(&iter->p2pdma, dma_dev,
 				 phys_to_page(vec.paddr))) {
 	case PCI_P2PDMA_MAP_BUS_ADDR:
+		/* don't leave the state of an earlier mapping to the caller */
+		memset(state, 0, sizeof(*state));
 		return blk_dma_map_bus(iter, &vec);
 	case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
 		/*
Signature · 1 line
2.56.0
Reviewed-by on v1 32/98
#74 lore raw

Re: [PATCH 32/98] blk-mq-dma: initialize the dma state for P2P, too

Quoted 7 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_rq_dma_map_iter_start() promises that the caller need not initialize the dma_iova_state, but the P2P bus address path returns before it does so. The state then holds what the last request with that tag left, and blk_rq_dma_map_coalesce() describes a transfer as one IOVA range that it isn't. Clear it on that path too.

Looks good:

author maintainer
#75 lore raw

PATCH 33/98 blk-mq-dma: map the metadata a passthrough request has

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_rq_integrity_dma_map_iter_start() sizes the metadata from the sector count and the device format. A passthrough request has whatever userspace gave it, so too much is mapped past the allocated IOVA range and too little hands the device an unmapped length. If the data needs a second bio, the iterator moves on to it and dereferences a NULL integrity payload.

Take the size from the payload and stop at a bio without one.

Fixes fec9b16dc555 ("blk-mq-dma: add scatter-less integrity data DMA mapping")
---
 block/blk-mq-dma.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/blk-mq-dma.c +8 -0
index e0e7a18790b5..5df704bbd98a 100644
@@ -13,6 +13,10 @@ static bool __blk_map_iter_next(struct blk_map_iter *iter)
 	if (!iter->bio || !iter->bio->bi_next)
 		return false;
 
+	/* All the metadata of a passthrough request is in its first bio */
+	if (iter->is_integrity && !bio_integrity(iter->bio->bi_next))
+		return false;
+
 	iter->bio = iter->bio->bi_next;
 	if (iter->is_integrity) {
 		iter->iter = bio_integrity(iter->bio)->bip_iter;
@@ -348,6 +352,10 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req,
 					   blk_rq_sectors(req));
 	struct bio *bio = req->bio;
 
+	/* User space decides how much metadata a passthrough request has */
+	if (blk_rq_is_passthrough(req))
+		len = bio_integrity(bio)->bip_iter.bi_size;
+
 	iter->iter = (struct blk_map_iter) {
 		.bio = bio,
 		.iter = bio_integrity(bio)->bip_iter,
Signature · 1 line
2.56.0
#76 lore raw

Re: [PATCH 33/98] blk-mq-dma: map the metadata a passthrough request has

The subject line does not parse.

block/blk-mq-dma.c static bool __blk_map_iter_next(struct blk_map_iter *iter) · line 17

Quoted 9 lines

index e0e7a18790b5..5df704bbd98a 100644

block/blk-mq-dma.c +2 -0
@@ -13,6 +13,10 @@ static bool __blk_map_iter_next(struct blk_map_iter *iter)
 	if (!iter->bio || !iter->bio->bi_next)
 		return false;
 
+	/* All the metadata of a passthrough request is in its first bio */
+	if (iter->is_integrity && !bio_integrity(iter->bio->bi_next))

The comment does not make sense. We're at a layer where we don't know if this is passthrough or not. And the block layer should make sure we never merge integrity vs non-integrity requests into a single requests. If the passthrough code breaks that somehow it needs to be fixed there.

block/blk-mq-dma.c bool blk_rq_integrity_dma_map_iter_start(struct request *req, · line 357

Quoted 12 lines
block/blk-mq-dma.c +5 -0
+		return false;
+
 	iter->bio = iter->bio->bi_next;
 	if (iter->is_integrity) {
 		iter->iter = bio_integrity(iter->bio)->bip_iter;
@@ -348,6 +352,10 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req,
 					   blk_rq_sectors(req));
 	struct bio *bio = req->bio;
 
+	/* User space decides how much metadata a passthrough request has */
+	if (blk_rq_is_passthrough(req))
+		len = bio_integrity(bio)->bip_iter.bi_size;

DMA mapping helpers don't have any business looking at the passthrough flag. But this should work for kernel-generated I/O just fine as far as I can tell.

author maintainer
#77 lore raw

PATCH 34/98 block: don't enter reclaim under rq_qos_mutex

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

ioc_qos_write() and the wbt_lat_usec writers take rq_qos_mutex with the queue frozen, blkcg_deactivate_policy() does the same with blkcg_mutex, so nothing holding either may wait for IO. blkg_conf_prep() got GFP_NOIO for the blkg in commit 5d726c4dbeed, but the radix_tree_preload() right after it, the policy init functions and the GFP_KERNEL fallback in blkcg_activate_policy() can still enter reclaim there. lockdep reports the cycle on the first write to io.max.

Use GFP_NOIO for all of them.

Fixes 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
Fixes 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy")
Cc Tejun Heo <[email protected]>
---
 block/blk-cgroup.c    | 13 +++++++++----
 block/blk-iocost.c    |  5 +++--
 block/blk-iolatency.c |  3 ++-
 block/blk-throttle.c  |  3 ++-
 4 files changed, 16 insertions(+), 8 deletions(-)
block/blk-cgroup.c +9 -4
index 00da55c6ec43..5b7a32293dc6 100644
@@ -884,7 +884,11 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			parent = blkcg_parent(parent);
 		}
 
-		/* Drop locks to do new blkg allocation with GFP_KERNEL. */
+		/*
+		 * Drop the spinlock to allocate the new blkg. rq_qos_mutex and
+		 * blkcg_mutex stay held, and both are taken with the queue
+		 * frozen elsewhere: GFP_NOIO.
+		 */
 		spin_unlock_irq(&q->queue_lock);
 
 		new_blkg = blkg_alloc(pos, disk, GFP_NOIO);
@@ -893,7 +897,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			goto fail_exit;
 		}
 
-		if (radix_tree_preload(GFP_KERNEL)) {
+		if (radix_tree_preload(GFP_NOIO)) {
 			blkg_free(new_blkg);
 			ret = -ENOMEM;
 			goto fail_exit;
@@ -1648,7 +1652,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
 		if (!pd) {
 			/*
 			 * GFP_NOWAIT failed.  Free the existing one and
-			 * prealloc for @blkg w/ GFP_KERNEL.
+			 * prealloc for @blkg w/ GFP_NOIO: blkcg_mutex is held,
+			 * and a bio based queue has not been frozen above.
 			 */
 			if (pinned_blkg)
 				blkg_put(pinned_blkg);
@@ -1660,7 +1665,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
 			if (pd_prealloc)
 				pol->pd_free_fn(pd_prealloc);
 			pd_prealloc = pol->pd_alloc_fn(disk, blkg->blkcg,
-						       GFP_KERNEL);
+						       GFP_NOIO);
 			if (pd_prealloc)
 				goto retry;
 			else
block/blk-iocost.c +3 -2
index bfb0dd320c4a..99fe89f687ae 100644
@@ -2938,11 +2938,12 @@ static int blk_iocost_init(struct gendisk *disk)
 	struct ioc *ioc;
 	int i, cpu, ret;
 
-	ioc = kzalloc_obj(*ioc);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	ioc = kzalloc_obj(*ioc, GFP_NOIO);
 	if (!ioc)
 		return -ENOMEM;
 
-	ioc->pcpu_stat = alloc_percpu(struct ioc_pcpu_stat);
+	ioc->pcpu_stat = alloc_percpu_gfp(struct ioc_pcpu_stat, GFP_NOIO);
 	if (!ioc->pcpu_stat) {
 		kfree(ioc);
 		return -ENOMEM;
block/blk-iolatency.c +2 -1
index c15b1aba606c..a3bd153c67be 100644
@@ -761,7 +761,8 @@ static int blk_iolatency_init(struct gendisk *disk)
 	struct blk_iolatency *blkiolat;
 	int ret;
 
-	blkiolat = kzalloc_obj(*blkiolat);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	blkiolat = kzalloc_obj(*blkiolat, GFP_NOIO);
 	if (!blkiolat)
 		return -ENOMEM;
 
block/blk-throttle.c +2 -1
index c46807a0faab..c5e26551956a 100644
@@ -1322,7 +1322,8 @@ static int blk_throtl_init(struct gendisk *disk)
 	unsigned int memflags;
 	int ret;
 
-	td = kzalloc_node(sizeof(*td), GFP_KERNEL, q->node);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	td = kzalloc_node(sizeof(*td), GFP_NOIO, q->node);
 	if (!td)
 		return -ENOMEM;
 
Signature · 1 line
2.56.0
maintainer Acked-by on v1 34/98
#78 lore raw

Re: [PATCH 34/98] block: don't enter reclaim under rq_qos_mutex

Quoted 8 lines · Jens Axboe

ioc_qos_write() and the wbt_lat_usec writers take rq_qos_mutex with the queue frozen, blkcg_deactivate_policy() does the same with blkcg_mutex, so nothing holding either may wait for IO. blkg_conf_prep() got GFP_NOIO for the blkg in commit 5d726c4dbeed, but the radix_tree_preload() right after it, the policy init functions and the GFP_KERNEL fallback in blkcg_activate_policy() can still enter reclaim there. lockdep reports the cycle on the first write to io.max.

Quoted 7 lines

Use GFP_NOIO for all of them.

Fixes 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
Fixes 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy")
Cc Tejun Heo <[email protected]>

Thanks.

--
tejun

author maintainer
#79 lore raw

PATCH 35/98 blktrace: remove the debugfs files before the channel

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_trace_free() closes the relay channel, which frees it, and removes the dropped and msg files afterwards, while both of them read bt->rchan. Remove the files first, which waits for their readers, then close the channel.

Fixes 7f2173894f7b ("blktrace: use rbuf->stats.full as a drop indicator in relayfs")
Cc Steven Rostedt <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
---
 kernel/trace/blktrace.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
kernel/trace/blktrace.c +8 -9
index 8cd2520b4c99..a37857a2fbd3 100644
@@ -468,18 +468,17 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 
 static void blk_trace_free(struct request_queue *q, struct blk_trace *bt)
 {
-	relay_close(bt->rchan);
-
 	/*
 	 * If 'bt->dir' is not set, then both 'dropped' and 'msg' are created
-	 * under 'q->debugfs_dir', thus lookup and remove them.
+	 * under 'q->debugfs_dir', thus lookup and remove them. They use the
+	 * relay channel: wait for their users before it goes away.
 	 */
-	if (!bt->dir) {
-		debugfs_lookup_and_remove("dropped", q->debugfs_dir);
-		debugfs_lookup_and_remove("msg", q->debugfs_dir);
-	} else {
-		debugfs_remove(bt->dir);
-	}
+	debugfs_lookup_and_remove("dropped", bt->dir ?: q->debugfs_dir);
+	debugfs_lookup_and_remove("msg", bt->dir ?: q->debugfs_dir);
+
+	relay_close(bt->rchan);
+	debugfs_remove(bt->dir);
+
 	free_percpu(bt->sequence);
 	free_percpu(bt->msg_data);
 	kfree(bt);
Signature · 1 line
2.56.0
author maintainer
#80 lore raw

PATCH 36/98 blk-mq: quiesce a queue that joins a quiesced tag set

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_quiesce_tagset() only quiesces the queues on the list at that moment. A queue added before blk_mq_unquiesce_tagset() runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning in blk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.

Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be tag_list_lock, see commit 59e25ef2b413.

Fixes 414dd48e882c ("blk-mq: add tagset quiesce interface")
---
 block/blk-mq.c         | 22 ++++++++++++++++++++++
 include/linux/blk-mq.h |  6 ++++++
 2 files changed, 28 insertions(+)
block/blk-mq.c +22 -0
index 6d3ea97f9a00..0e8449424c9a 100644
@@ -336,12 +336,15 @@ void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	set->quiesce_depth++;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 
 	blk_mq_wait_quiesce_done(set);
 }
@@ -351,12 +354,16 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	if (set->quiesce_depth)
+		set->quiesce_depth--;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_unquiesce_queue(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 }
 EXPORT_SYMBOL_GPL(blk_mq_unquiesce_tagset);
 
@@ -4346,6 +4353,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 				     struct request_queue *q)
 {
+	unsigned int i;
+
 	mutex_lock(&set->tag_list_lock);
 
 	/*
@@ -4359,7 +4368,18 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 	}
 	if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
 		queue_set_hctx_shared(q, true);
+
+	/*
+	 * A queue that joins a quiesced tag set is quiesced like the others,
+	 * and blk_mq_unquiesce_tagset() finds what it is going to undo.
+	 */
+	spin_lock(&set->quiesce_lock);
 	list_add_tail_rcu(&q->tag_set_list, &set->tag_list);
+	if (!blk_queue_skip_tagset_quiesce(q)) {
+		for (i = 0; i < set->quiesce_depth; i++)
+			blk_mq_quiesce_queue_nowait(q);
+	}
+	spin_unlock(&set->quiesce_lock);
 
 	mutex_unlock(&set->tag_list_lock);
 }
@@ -4928,6 +4948,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 
 	mutex_init(&set->tag_list_lock);
 	INIT_LIST_HEAD(&set->tag_list);
+	spin_lock_init(&set->quiesce_lock);
+	set->quiesce_depth = 0;
 
 	return 0;
 
include/linux/blk-mq.h +6 -0
index 5e130b282b42..aa3b35405edc 100644
@@ -523,6 +523,10 @@ enum hctx_type {
  * @tag_list_lock: Serializes tag_list accesses.
  * @tag_list:	   List of the request queues that use this tag set. See also
  *		   request_queue.tag_set_list.
+ * @quiesce_lock:  Serializes blk_mq_[un]quiesce_tagset() against each other
+ *		   and against a request queue being added to @tag_list.
+ * @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without
+ *		   blk_mq_unquiesce_tagset().
  * @srcu:	   Use as lock when type of the request queue is blocking
  *		   (BLK_MQ_F_BLOCKING).
  * @update_nr_hwq_lock:
@@ -548,6 +552,8 @@ struct blk_mq_tag_set {
 
 	struct mutex		tag_list_lock;
 	struct list_head	tag_list;
+	spinlock_t		quiesce_lock;
+	unsigned int		quiesce_depth;
 	struct srcu_struct	*srcu;
 
 	struct rw_semaphore	update_nr_hwq_lock;
Signature · 1 line
2.56.0
#82 lore raw

Re: [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set

In reply to Jens Axboe

Hi Rust for Linux folks,

I only added a small subset of you plus the mailing list, if I missed anybody, please bring them in as necessary.

Quoted 8 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_quiesce_tagset() only quiesces the queues on the list at that moment. A queue added before blk_mq_unquiesce_tagset() runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning in blk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.

Quoted 7 lines

Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be tag_list_lock, see commit 59e25ef2b413.

Fixes 414dd48e882c ("blk-mq: add tagset quiesce interface")

After this patch landed in -next as commit 71ec3c7d6db8 ("blk-mq:
quiesce a queue that joins a quiesced tag set"), I am seeing the following build error:

  $ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper allmodconfig rust/kernel.o
  error[E0277]: the trait bound `bindings::__BindgenOpaqueArray8<[u8; 72]>: pin_init::Zeroable` is not satisfied
      --> rust/kernel/block/mq/tag_set.rs:41:49
       |
    41 |         let tag_set: bindings::blk_mq_tag_set = pin_init::zeroed();
       |                                                 ^^^^^^^^^^^^^^^^^^ the trait `pin_init::Zeroable` is not implemented for `bindings::__BindgenOpaqueArray8<[u8; 72]>`
       |
       = help: the following other types implement trait `pin_init::Zeroable`:
                 ()
                 (A, B, C, D, E, F, G, H, I, J)
                 (B, C, D, E, F, G, H, I, J)
                 (C, D, E, F, G, H, I, J)
                 (D, E, F, G, H, I, J)
                 (E, F, G, H, I, J)
                 (F, G, H, I, J)
                 (G, H, I, J)
               and 6359 others
       = note: required for `bindings::spinlock` to implement `pin_init::Zeroable`
       = note: 1 redundant requirement hidden
       = note: required for `blk_mq_tag_set` to implement `pin_init::Zeroable`
  note: required by a bound in `pin_init::zeroed`
      --> rust/pin-init/src/lib.rs:1688:24
       |
  1688 | pub const fn zeroed<T: Zeroable>() -> T {
       |                        ^^^^^^^^ required by this bound in `zeroed`

error: aborting due to 1 previous error

For more information about this error, try rustc --explain E0277. make[4]: *** [rust/Makefile:799: rust/kernel.o] Error 1
...

Quoted 106 lines
---
 block/blk-mq.c         | 22 ++++++++++++++++++++++
 include/linux/blk-mq.h |  6 ++++++
 2 files changed, 28 insertions(+)
block/blk-mq.c +22 -0
index 6d3ea97f9a00..0e8449424c9a 100644
@@ -336,12 +336,15 @@ void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	set->quiesce_depth++;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 
 	blk_mq_wait_quiesce_done(set);
 }
@@ -351,12 +354,16 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	if (set->quiesce_depth)
+		set->quiesce_depth--;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_unquiesce_queue(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 }
 EXPORT_SYMBOL_GPL(blk_mq_unquiesce_tagset);
 
@@ -4346,6 +4353,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 				     struct request_queue *q)
 {
+	unsigned int i;
+
 	mutex_lock(&set->tag_list_lock);
 
 	/*
@@ -4359,7 +4368,18 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 	}
 	if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
 		queue_set_hctx_shared(q, true);
+
+	/*
+	 * A queue that joins a quiesced tag set is quiesced like the others,
+	 * and blk_mq_unquiesce_tagset() finds what it is going to undo.
+	 */
+	spin_lock(&set->quiesce_lock);
 	list_add_tail_rcu(&q->tag_set_list, &set->tag_list);
+	if (!blk_queue_skip_tagset_quiesce(q)) {
+		for (i = 0; i < set->quiesce_depth; i++)
+			blk_mq_quiesce_queue_nowait(q);
+	}
+	spin_unlock(&set->quiesce_lock);
 
 	mutex_unlock(&set->tag_list_lock);
 }
@@ -4928,6 +4948,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 
 	mutex_init(&set->tag_list_lock);
 	INIT_LIST_HEAD(&set->tag_list);
+	spin_lock_init(&set->quiesce_lock);
+	set->quiesce_depth = 0;
 
 	return 0;
 
include/linux/blk-mq.h +6 -0
index 5e130b282b42..aa3b35405edc 100644
@@ -523,6 +523,10 @@ enum hctx_type {
  * @tag_list_lock: Serializes tag_list accesses.
  * @tag_list:	   List of the request queues that use this tag set. See also
  *		   request_queue.tag_set_list.
+ * @quiesce_lock:  Serializes blk_mq_[un]quiesce_tagset() against each other
+ *		   and against a request queue being added to @tag_list.
+ * @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without
+ *		   blk_mq_unquiesce_tagset().
  * @srcu:	   Use as lock when type of the request queue is blocking
  *		   (BLK_MQ_F_BLOCKING).
  * @update_nr_hwq_lock:
@@ -548,6 +552,8 @@ struct blk_mq_tag_set {
 
 	struct mutex		tag_list_lock;
 	struct list_head	tag_list;
+	spinlock_t		quiesce_lock;
+	unsigned int		quiesce_depth;
 	struct srcu_struct	*srcu;
 
 	struct rw_semaphore	update_nr_hwq_lock;
Signature · 1 line
2.56.0
Signature · 2 lines
Cheers,
Nathan
#83 lore raw

Re: [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set

Quoted 52 lines · Nathan Chancellor

Hi Rust for Linux folks,

I only added a small subset of you plus the mailing list, if I missed anybody, please bring them in as necessary.

Quoted 8 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_quiesce_tagset() only quiesces the queues on the list at that moment. A queue added before blk_mq_unquiesce_tagset() runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning in blk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.

Quoted 7 lines

Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be tag_list_lock, see commit 59e25ef2b413.

Fixes 414dd48e882c ("blk-mq: add tagset quiesce interface")

After this patch landed in -next as commit 71ec3c7d6db8 ("blk-mq:
quiesce a queue that joins a quiesced tag set"), I am seeing the following build error:

  $ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper allmodconfig rust/kernel.o
  error[E0277]: the trait bound `bindings::__BindgenOpaqueArray8<[u8; 72]>: pin_init::Zeroable` is not satisfied
      --> rust/kernel/block/mq/tag_set.rs:41:49
       |
    41 |         let tag_set: bindings::blk_mq_tag_set = pin_init::zeroed();
       |                                                 ^^^^^^^^^^^^^^^^^^ the trait `pin_init::Zeroable` is not implemented for `bindings::__BindgenOpaqueArray8<[u8; 72]>`
       |
       = help: the following other types implement trait `pin_init::Zeroable`:
                 ()
                 (A, B, C, D, E, F, G, H, I, J)
                 (B, C, D, E, F, G, H, I, J)
                 (C, D, E, F, G, H, I, J)
                 (D, E, F, G, H, I, J)
                 (E, F, G, H, I, J)
                 (F, G, H, I, J)
                 (G, H, I, J)
               and 6359 others
       = note: required for `bindings::spinlock` to implement `pin_init::Zeroable`
       = note: 1 redundant requirement hidden
       = note: required for `blk_mq_tag_set` to implement `pin_init::Zeroable`
  note: required by a bound in `pin_init::zeroed`
      --> rust/pin-init/src/lib.rs:1688:24
       |
  1688 | pub const fn zeroed<T: Zeroable>() -> T {
       |                        ^^^^^^^^ required by this bound in `zeroed`
Quoted 6 lines

error: aborting due to 1 previous error

For more information about this error, try rustc --explain E0277. make[4]: *** [rust/Makefile:799: rust/kernel.o] Error 1
...

Thanks for reporting. This was fixed over here [1].

Best regards,
Andreas Hindborg

[1] https://lore.kernel.org/r/[email protected]

author maintainer
#84 lore raw

PATCH 37/98 blk-mq: exit the hctxs of a queue under tag_list_lock

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_update_nr_hw_queues() exits and replaces the hctxs of every queue on set->tag_list under tag_list_lock. blk_mq_exit_queue() exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits. del_gendisk() is kept away by update_nr_hwq_lock, disk_release() of a never added disk and blk_mq_destroy_queue() are not, and nvme_alloc_ns() drops such a disk when it finds the queues frozen by a reset.

Hold tag_list_lock for both steps.

Fixes 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")
---
 block/blk-mq.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-mq.c +8 -2
index 0e8449424c9a..37c8f68e23c4 100644
@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
+
 	list_del_rcu(&q->tag_set_list);
 	if (list_is_singular(&set->tag_list)) {
 		/* just transitioned to unshared */
@@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 		/* update existing queue */
 		blk_mq_update_tag_set_shared(set, false);
 	}
-	mutex_unlock(&set->tag_list_lock);
 }
 
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
+	/*
+	 * As long as the queue is on set->tag_list, its hctxs belong to
+	 * blk_mq_update_nr_hw_queues() as well, which exits and replaces them.
+	 */
+	mutex_lock(&set->tag_list_lock);
 	/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */
 	blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
 	/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */
 	blk_mq_del_queue_tag_set(q);
+	mutex_unlock(&set->tag_list_lock);
 }
 
 static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
Signature · 1 line
2.56.0
#85 lore raw

Re: [PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock

Quoted 11 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_update_nr_hw_queues() exits and replaces the hctxs of every queue on set->tag_list under tag_list_lock. blk_mq_exit_queue() exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits. del_gendisk() is kept away by update_nr_hwq_lock, disk_release() of a never added disk and blk_mq_destroy_queue() are not, and nvme_alloc_ns() drops such a disk when it finds the queues frozen by a reset.

Quoted 1 line

Hold tag_list_lock for both steps.

I'm a bit worried about cpu hotplug lock interaction as that always lead to the craziest imaginable lock chains, but otherwise this looks ok.

Quoted 51 lines
Fixes 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")
---
 block/blk-mq.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-mq.c +8 -2
index 0e8449424c9a..37c8f68e23c4 100644
@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
+
 	list_del_rcu(&q->tag_set_list);
 	if (list_is_singular(&set->tag_list)) {
 		/* just transitioned to unshared */
@@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 		/* update existing queue */
 		blk_mq_update_tag_set_shared(set, false);
 	}
-	mutex_unlock(&set->tag_list_lock);
 }
 
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
+	/*
+	 * As long as the queue is on set->tag_list, its hctxs belong to
+	 * blk_mq_update_nr_hw_queues() as well, which exits and replaces them.
+	 */
+	mutex_lock(&set->tag_list_lock);
 	/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */
 	blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
 	/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */
 	blk_mq_del_queue_tag_set(q);
+	mutex_unlock(&set->tag_list_lock);
 }
 
 static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
Signature · 1 line
2.56.0

---end quoted text---

Reviewed-by
#86 lore raw

Re: [PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock

Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_update_nr_hw_queues() exits and replaces the hctxs of every queue on set->tag_list under tag_list_lock. blk_mq_exit_queue() exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits. del_gendisk() is kept away by update_nr_hwq_lock, disk_release() of a never added disk and blk_mq_destroy_queue() are not, and nvme_alloc_ns() drops such a disk when it finds the queues frozen by a reset.

Quoted 6 lines
Quoted 1 line

Hold tag_list_lock for both steps.

I'm a bit worried about cpu hotplug lock interaction as that always lead to the craziest imaginable lock chains, but otherwise this looks ok.

Yes, CPU hotplug lock interactions have caused numerous lockdep splats in the past. However, looking at this change, the lock chains appear to remain consistent. I reviewed all possible paths leading to hctx exit and cpu_hotplug_lock and those appears to be consistent.

It seems the new order is consistent across all relevant paths:

blk_mq_update_nr_hw_queues(): update_nr_hwq_lock (write) -> tag_list_lock.

Registered-disk teardown: update_nr_hwq_lock (read) -> tag_list_lock.

Never-registered disk release and blk_mq_destroy_queue(): tag_list_lock only.

HCTX exit callbacks already run under tag_list_lock during hardware-queue resizing, so the new exit path does not introduce a new locking context.

CPU-hotplug removal remains tag_list_lock -> blk_mq_cpuhp_lock -> cpu_hotplug_lock, the same order already used by the resize path; there is no reverse acquisition.

So With that,

Quoted 52 lines
Quoted 51 lines
Fixes 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")
---
  block/blk-mq.c | 10 ++++++++--
  1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-mq.c +8 -2
index 0e8449424c9a..37c8f68e23c4 100644
@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
  {
  	struct blk_mq_tag_set *set = q->tag_set;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
+
  	list_del_rcu(&q->tag_set_list);
  	if (list_is_singular(&set->tag_list)) {
  		/* just transitioned to unshared */
@@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
  		/* update existing queue */
  		blk_mq_update_tag_set_shared(set, false);
  	}
-	mutex_unlock(&set->tag_list_lock);
  }
 
  static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q)
  {
  	struct blk_mq_tag_set *set = q->tag_set;
 
+	/*
+	 * As long as the queue is on set->tag_list, its hctxs belong to
+	 * blk_mq_update_nr_hw_queues() as well, which exits and replaces them.
+	 */
+	mutex_lock(&set->tag_list_lock);
  	/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */
  	blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
  	/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */
  	blk_mq_del_queue_tag_set(q);
+	mutex_unlock(&set->tag_list_lock);
  }
 
  static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
Signature · 1 line
2.56.0

---end quoted text---

author maintainer
#87 lore raw

PATCH 38/98 blk-mq: set up a queue under tag_list_lock

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_init_allocated_queue() only takes tag_list_lock at the end, when it adds the queue to the set. Before that it sizes the queue by set->nr_hw_queues, hands out set->tags[], maps by set->map[], and blk_mq_map_swqueue() frees tags of the set. An update clearing the maps at that time makes the new queue free tags that all the other queues share. An NVMe controller reset during a namespace scan does exactly that.

Take the lock before the first look at the set. An update can be part of error handling, so the allocations under the lock must not wait for IO:
memalloc_noio_save(), as in add_disk_fwnode().

Fixes 868f2f0b7206 ("blk-mq: dynamic h/w context count")
---
 block/blk-mq.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
block/blk-mq.c +18 -3
index 37c8f68e23c4..458ef0970527 100644
@@ -4355,7 +4355,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 {
 	unsigned int i;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
 
 	/*
 	 * Check to see if we're transitioning to shared (from 1 to 2 queues).
@@ -4380,8 +4380,6 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	spin_unlock(&set->quiesce_lock);
-
-	mutex_unlock(&set->tag_list_lock);
 }
 
 /* All allocations will be freed in release handler of q->mq_kobj */
@@ -4658,6 +4656,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
 int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 		struct request_queue *q)
 {
+	unsigned int memflags;
+
 	/* mark the queue as mq asap */
 	q->mq_ops = set->ops;
 
@@ -4676,6 +4676,16 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	INIT_LIST_HEAD(&q->unused_hctx_list);
 	spin_lock_init(&q->unused_hctx_lock);
 
+	/*
+	 * Keep blk_mq_update_nr_hw_queues() away until the queue is on
+	 * set->tag_list: it changes the number of hardware queues, the tags and
+	 * the maps, and blk_mq_map_swqueue() allocates and frees tags of the set.
+	 * An update can be part of the error handling of a driver, so it must
+	 * not have to wait for somebody who waits for I/O.
+	 */
+	memflags = memalloc_noio_save();
+	mutex_lock(&set->tag_list_lock);
+
 	blk_mq_realloc_hw_ctxs(set, q);
 	if (!q->nr_hw_queues)
 		goto err_hctxs;
@@ -4696,9 +4706,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 	blk_mq_map_swqueue(q);
 	blk_mq_add_queue_tag_set(set, q);
+
+	mutex_unlock(&set->tag_list_lock);
+	memalloc_noio_restore(memflags);
 	return 0;
 
 err_hctxs:
+	mutex_unlock(&set->tag_list_lock);
+	memalloc_noio_restore(memflags);
 	blk_mq_release(q);
 err_exit:
 	q->mq_ops = NULL;
Signature · 1 line
2.56.0
#88 lore raw

Re: [PATCH 38/98] blk-mq: set up a queue under tag_list_lock

Quoted 53 lines · Jens Axboe

From: Josef Bacik <[email protected]>

blk_mq_init_allocated_queue() only takes tag_list_lock at the end, when it adds the queue to the set. Before that it sizes the queue by set->nr_hw_queues, hands out set->tags[], maps by set->map[], and blk_mq_map_swqueue() frees tags of the set. An update clearing the maps at that time makes the new queue free tags that all the other queues share. An NVMe controller reset during a namespace scan does exactly that.

Take the lock before the first look at the set. An update can be part of error handling, so the allocations under the lock must not wait for IO:
memalloc_noio_save(), as in add_disk_fwnode().

Fixes 868f2f0b7206 ("blk-mq: dynamic h/w context count")
---
 block/blk-mq.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
block/blk-mq.c +3 -3
index 37c8f68e23c4..458ef0970527 100644
@@ -4355,7 +4355,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 {
 	unsigned int i;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
 
 	/*
 	 * Check to see if we're transitioning to shared (from 1 to 2 queues).
@@ -4380,8 +4380,6 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	spin_unlock(&set->quiesce_lock);
-
-	mutex_unlock(&set->tag_list_lock);
 }
 
 /* All allocations will be freed in release handler of q->mq_kobj */
@@ -4658,6 +4656,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
 int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 		struct request_queue *q)
 {
+	unsigned int memflags;
+
 	/* mark the queue as mq asap */
 	q->mq_ops = set->ops;
 

block/blk-mq.c int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, · line 4682

Quoted 7 lines
block/blk-mq.c +4 -0
@@ -4676,6 +4676,16 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	INIT_LIST_HEAD(&q->unused_hctx_list);
 	spin_lock_init(&q->unused_hctx_lock);
 
+	/*
+	 * Keep blk_mq_update_nr_hw_queues() away until the queue is on
+	 * set->tag_list: it changes the number of hardware queues, the tags and
+	 * the maps, and blk_mq_map_swqueue() allocates and frees tags of the set.

Overly long line inside an otherwise nicely formatted block comment :(

Otherwise similarly a slight worry about lock chains, otherwise this looks fine.

author maintainer
#89 lore raw

PATCH 39/98 rnbd-srv: fix use-after-free on two CLOSE messages

In reply to Jens Axboe

From: Chris Mason <[email protected]>

process_msg_close() looks the sess_dev up with a kref, drops it right away, and only then takes the session mutex for the teardown. A session has several connections handled by different workers, so two CLOSE messages for the same ID both finish the lookup before either one tears the sess_dev down, and the second one works on freed memory. Any host that can open a device on the server can do that.

Take the mutex first and look the ID up with xa_load() under it. The second CLOSE then finds nothing.

Fixes 2de6c8de192b ("block/rnbd: server: main functionality")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>
---
 drivers/block/rnbd/rnbd-srv.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -7
index 1e3de82fd70b..1cfa2c580759 100644
@@ -360,14 +360,15 @@ static void process_msg_close(struct rnbd_srv_session *srv_sess,
 
 	trace_process_msg_close(srv_sess, close_msg);
 
-	sess_dev = rnbd_get_sess_dev(le32_to_cpu(close_msg->device_id),
-				      srv_sess);
-	if (IS_ERR(sess_dev))
-		return;
-
-	rnbd_put_sess_dev(sess_dev);
+	/*
+	 * Each teardown of a sess_dev runs under srv_sess->lock and erases the
+	 * ID, so an entry that is found under the lock is alive.
+	 */
 	mutex_lock(&srv_sess->lock);
-	rnbd_srv_destroy_dev_session_sysfs(sess_dev);
+	sess_dev = xa_load(&srv_sess->index_idr,
+			   le32_to_cpu(close_msg->device_id));
+	if (sess_dev)
+		rnbd_srv_destroy_dev_session_sysfs(sess_dev);
 	mutex_unlock(&srv_sess->lock);
 }
 
Signature · 1 line
2.56.0
Acked-by on v1 39/98
#90 lore raw

Re: [PATCH 39/98] rnbd-srv: fix use-after-free on two CLOSE messages

Quoted 10 lines · Jens Axboe

From: Chris Mason <[email protected]>

process_msg_close() looks the sess_dev up with a kref, drops it right away, and only then takes the session mutex for the teardown. A session has several connections handled by different workers, so two CLOSE messages for the same ID both finish the lookup before either one tears the sess_dev down, and the second one works on freed memory. Any host that can open a device on the server can do that.

Quoted 8 lines

Take the mutex first and look the ID up with xa_load() under it. The second CLOSE then finds nothing.

Fixes 2de6c8de192b ("block/rnbd: server: main functionality")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>

Thx!

Quoted 34 lines
---
 drivers/block/rnbd/rnbd-srv.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -8
index 1e3de82fd70b..1cfa2c580759 100644
@@ -360,14 +360,15 @@ static void process_msg_close(struct rnbd_srv_session *srv_sess,
 
        trace_process_msg_close(srv_sess, close_msg);
 
-       sess_dev = rnbd_get_sess_dev(le32_to_cpu(close_msg->device_id),
-                                     srv_sess);
-       if (IS_ERR(sess_dev))
-               return;
-
-       rnbd_put_sess_dev(sess_dev);
+       /*
+        * Each teardown of a sess_dev runs under srv_sess->lock and erases the
+        * ID, so an entry that is found under the lock is alive.
+        */
        mutex_lock(&srv_sess->lock);
-       rnbd_srv_destroy_dev_session_sysfs(sess_dev);
+       sess_dev = xa_load(&srv_sess->index_idr,
+                          le32_to_cpu(close_msg->device_id));
+       if (sess_dev)
+               rnbd_srv_destroy_dev_session_sysfs(sess_dev);
        mutex_unlock(&srv_sess->lock);
 }
 
--

2.56.0

author maintainer
#91 lore raw

PATCH 40/98 rnbd-srv: publish the sess_dev at the end of OPEN

In reply to Jens Axboe

From: Chris Mason <[email protected]>

rnbd_sess_dev_alloc() stores the empty sess_dev in the XArray as its first step. An IO message with the new ID from another connection can find it before bdev_file is set, or hold a reference while a failing OPEN frees it after a mere RCU grace period. Both are reachable from any host that can open a device.

Reserve the ID with a NULL entry and store the sess_dev once nothing can fail anymore. The error path still erases the ID, and no longer needs synchronize_rcu().

Fixes 2de6c8de192b ("block/rnbd: server: main functionality")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>
---
 drivers/block/rnbd/rnbd-srv.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -2
index 1cfa2c580759..1c80e8fe0267 100644
@@ -431,7 +431,7 @@ static struct rnbd_srv_sess_dev
 	if (!sess_dev)
 		return ERR_PTR(-ENOMEM);
 
-	error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, sess_dev,
+	error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, NULL,
 			 xa_limit_32b, GFP_NOWAIT);
 	if (error < 0) {
 		pr_warn("Allocating idr failed, err: %d\n", error);
@@ -802,6 +802,13 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 	list_add(&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
 	mutex_unlock(&srv_dev->lock);
 
+	/*
+	 * Nothing fails from here on, so let I/O messages find the sess_dev.
+	 * The ID is reserved, so this store allocates nothing.
+	 */
+	xa_store(&srv_sess->index_idr, srv_sess_dev->device_id, srv_sess_dev,
+		 GFP_KERNEL);
+
 	rnbd_srv_info(srv_sess_dev, "Opened device '%s'\n", srv_dev->name);
 
 	kfree(full_path);
@@ -813,7 +820,6 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 
 free_srv_sess_dev:
 	xa_erase(&srv_sess->index_idr, srv_sess_dev->device_id);
-	synchronize_rcu();
 	kfree(srv_sess_dev);
 srv_dev_put:
 	if (open_msg->access_mode != RNBD_ACCESS_RO) {
Signature · 1 line
2.56.0
Acked-by on v1 40/98
#92 lore raw

Re: [PATCH 40/98] rnbd-srv: publish the sess_dev at the end of OPEN

Quoted 13 lines · Jens Axboe

From: Chris Mason <[email protected]>

rnbd_sess_dev_alloc() stores the empty sess_dev in the XArray as its first step. An IO message with the new ID from another connection can find it before bdev_file is set, or hold a reference while a failing OPEN frees it after a mere RCU grace period. Both are reachable from any host that can open a device.

Reserve the ID with a NULL entry and store the sess_dev once nothing can fail anymore. The error path still erases the ID, and no longer needs synchronize_rcu().

Quoted 5 lines
Fixes 2de6c8de192b ("block/rnbd: server: main functionality")
Cc "Md. Haris Iqbal" <[email protected]>
Cc Jack Wang <[email protected]>

Thx!

Quoted 42 lines
---
 drivers/block/rnbd/rnbd-srv.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -3
index 1cfa2c580759..1c80e8fe0267 100644
@@ -431,7 +431,7 @@ static struct rnbd_srv_sess_dev
        if (!sess_dev)
                return ERR_PTR(-ENOMEM);
 
-       error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, sess_dev,
+       error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, NULL,
                         xa_limit_32b, GFP_NOWAIT);
        if (error < 0) {
                pr_warn("Allocating idr failed, err: %d\n", error);
@@ -802,6 +802,13 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
        list_add(&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
        mutex_unlock(&srv_dev->lock);
 
+       /*
+        * Nothing fails from here on, so let I/O messages find the sess_dev.
+        * The ID is reserved, so this store allocates nothing.
+        */
+       xa_store(&srv_sess->index_idr, srv_sess_dev->device_id, srv_sess_dev,
+                GFP_KERNEL);
+
        rnbd_srv_info(srv_sess_dev, "Opened device '%s'\n", srv_dev->name);
 
        kfree(full_path);
@@ -813,7 +820,6 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 
 free_srv_sess_dev:
        xa_erase(&srv_sess->index_idr, srv_sess_dev->device_id);
-       synchronize_rcu();
        kfree(srv_sess_dev);
 srv_dev_put:
        if (open_msg->access_mode != RNBD_ACCESS_RO) {
--

2.56.0

author maintainer
#93 lore raw

PATCH 41/98 zram: check a bio's range against the device size

In reply to Jens Axboe

From: Chris Mason <[email protected]>

bio_check_eod() counts whole sectors, so a bio shorter than a sector, or one whose tail is shorter than a sector, passes it even at or past the end of the device. zram takes such a piece to the slot of its sector and reads, locks and modifies an entry outside the table, or dereferences a NULL table on a device without a size. valid_io_request() refused these until commit 9fe95babc742 removed it, and any in-kernel builder of bios that takes size and sector from a peer can feed them in.

Check the range at the top of zram_submit_bio(), with the size rounded up to sectors, and fail with an IO error like bio_check_eod() does.

Fixes 9fe95babc742 ("zram: remove valid_io_request")
Cc Minchan Kim <[email protected]>
Cc Sergey Senozhatsky <[email protected]>
---
 drivers/block/zram/zram_drv.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
drivers/block/zram/zram_drv.c +22 -0
index 024402438bd1..2555057a48b5 100644
@@ -2794,6 +2794,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)
 	bio_endio(bio);
 }
 
+/*
+ * bio_check_eod() counts whole sectors: it does not look at a bio that is
+ * shorter than a sector, nor at a tail that is shorter than one. zram takes
+ * such a piece to the slot of its sector, so round the size up. An empty bio
+ * has to start inside the device as well: the loops for reads and writes run
+ * once before their test.
+ */
+static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
+			SECTOR_SHIFT;
+
+	return start < nr_sectors && len <= nr_sectors - start;
+}
+
 /*
  * Handler function for all zram I/O requests.
  */
@@ -2801,6 +2818,11 @@ static void zram_submit_bio(struct bio *bio)
 {
 	struct zram *zram = bio->bi_bdev->bd_disk->private_data;
 
+	if (unlikely(!zram_bio_in_range(zram, bio))) {
+		bio_io_error(bio);
+		return;
+	}
+
 	switch (bio_op(bio)) {
 	case REQ_OP_READ:
 		zram_bio_read(zram, bio);
Signature · 1 line
2.56.0
#94 lore raw

Re: [PATCH 41/98] zram: check a bio's range against the device size

Quoted 6 lines · Jens Axboe
drivers/block/zram/zram_drv.c +6 -0
+/*
+ * bio_check_eod() counts whole sectors: it does not look at a bio that is
+ * shorter than a sector, nor at a tail that is shorter than one. zram takes
+ * such a piece to the slot of its sector, so round the size up. An empty bio
+ * has to start inside the device as well: the loops for reads and writes run
+ * once before their test.

drivers/block/zram/zram_drv.c static void zram_bio_write(struct zram *zram, struct bio *bio) · line 2812

Quoted 10 lines
drivers/block/zram/zram_drv.c +10 -0
+ */
+static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
+			SECTOR_SHIFT;
+
+	return start < nr_sectors && len <= nr_sectors - start;
+}

There are two problems with zram_bio_in_range():

1. It explicitly allows an empty REQ_OP_READ or REQ_OP_WRITE bio (bi_size == 0) as long as start < nr_sectors. Both zram_bio_read() and zram_bio_write() use a do { ... } while (iter.bi_size) loop whose first statement is bio_iter_iovec(bio, iter), which immediately dereferences NULL bio->bi_io_vec on an empty bio.

2. It allows sub-4K / unaligned bios as long as they fall within nr_sectors. However, ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on PAGE_SIZE == 4096 kernels is_partial_io() is hardcoded to return false:
- zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index, bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes of bv_page.
- zram_bvec_write() calls zram_write_page(zram, bvec->bv_page, index), clobbering the rest of the 4 KiB zram slot and compressing up to 4095 bytes of adjacent kernel page memory into the slot where a subsequent read will return it.

Shouldn't this reinstate valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE alignment checks on bi_sector and bi_size (as well as rejecting bi_size == 0)?

#95 lore raw

Re: [PATCH 41/98] zram: check a bio's range against the device size

In reply to Jens Axboe

Review

Quoted 9 lines · Jens Axboe
(quoted diff) +9 -0
+static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
+			SECTOR_SHIFT;
+
+	return start < nr_sectors && len <= nr_sectors - start;
+}

There are two problems with zram_bio_in_range():

1. It allows an empty REQ_OP_READ or REQ_OP_WRITE bio (bi_size == 0) as long as start < nr_sectors. Both zram_bio_read() and zram_bio_write() use a do { ... } while (iter.bi_size) loop whose first statement is bio_iter_iovec(bio, iter), dereferencing NULL bio->bi_io_vec on an empty bio. (Empty REQ_OP_FLUSH bios are already completed in submit_bio_noacct() because zram does not set a write cache.)

2. It allows sub-4K / unaligned bios as long as they fall within nr_sectors. ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on PAGE_SIZE == 4096 kernels is_partial_io() is hardcoded to return false:
- zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index, bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes of bv_page.
- zram_bvec_write() calls zram_write_page(zram, bvec->bv_page, index), clobbering the rest of the 4 KiB zram slot and compressing up to 4095 bytes of adjacent kernel page memory into the slot where a subsequent read can read it back.

Reinstating valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE alignment checks on bi_sector and bi_size alongside rejecting bi_size == 0 and out-of-range bios closes all of these cases.

Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

drivers/block/zram/zram_drv.c +22 -0
index ab064c84b092..8d4e3791b424 100644
@@ -2796,6 +2796,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)
 	bio_endio(bio);
 }
 
+static bool zram_valid_io_request(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	unsigned int size = bio->bi_iter.bi_size;
+
+	if (!size)
+		return false;
+	if (start & (ZRAM_SECTOR_PER_LOGICAL_BLOCK - 1))
+		return false;
+	if (size & (ZRAM_LOGICAL_BLOCK_SIZE - 1))
+		return false;
+
+	return start < nr_sectors &&
+	       (size >> SECTOR_SHIFT) <= nr_sectors - start;
+}
+
 /*
  * Handler function for all zram I/O requests.
  */
@@ -2803,6 +2820,11 @@ static void zram_submit_bio(struct bio *bio)
 {
 	struct zram *zram = bio->bi_bdev->bd_disk->private_data;
 
+	if (unlikely(!zram_valid_io_request(zram, bio))) {
+		bio_io_error(bio);
+		return;
+	}
+
 	switch (bio_op(bio)) {
 	case REQ_OP_READ:
 		zram_bio_read(zram, bio);
author maintainer
#96 lore raw

PATCH 42/98 block: partitions/mac: compute the sectors in 64 bits

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

start_block and block_count are converted to 512 byte sectors in 32-bit arithmetic, although put_partition() takes a sector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
---
 block/partitions/mac.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/partitions/mac.c +2 -2
index df03ca428e15..f372392af95b 100644
@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)
 		if (be16_to_cpu(part->signature) != MAC_PARTITION_MAGIC)
 			break;
 		put_partition(state, slot,
-			be32_to_cpu(part->start_block) * (secsize/512),
-			be32_to_cpu(part->block_count) * (secsize/512));
+			(sector_t)be32_to_cpu(part->start_block) * (secsize/512),
+			(sector_t)be32_to_cpu(part->block_count) * (secsize/512));
 
 		if (!strncasecmp(part->type, "Linux_RAID", 10))
 			state->parts[slot].flags = ADDPART_FLAG_RAID;
Signature · 1 line
2.56.0
#97 lore raw

Re: [PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits

Quoted 19 lines · Jens Axboe

From: Josef Bacik <[email protected]>

start_block and block_count are converted to 512 byte sectors in 32-bit arithmetic, although put_partition() takes a sector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
---
 block/partitions/mac.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/partitions/mac.c +0 -0
index df03ca428e15..f372392af95b 100644

block/partitions/mac.c int mac_partition(struct parsed_partitions *state) · line 101

Quoted 7 lines
block/partitions/mac.c +2 -2
@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)
 		if (be16_to_cpu(part->signature) != MAC_PARTITION_MAGIC)
 			break;
 		put_partition(state, slot,
-			be32_to_cpu(part->start_block) * (secsize/512),
-			be32_to_cpu(part->block_count) * (secsize/512));
+			(sector_t)be32_to_cpu(part->start_block) * (secsize/512),
+			(sector_t)be32_to_cpu(part->block_count) * (secsize/512));

PLease just put this into a helper..

author maintainer
#98 lore raw

Re: [PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits

Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Jens Axboe

From: Josef Bacik <[email protected]>

start_block and block_count are converted to 512 byte sectors in 32-bit arithmetic, although put_partition() takes a sector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
---
 block/partitions/mac.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 14 lines

block/partitions/mac.c int mac_partition(struct parsed_partitions *state) · line 101

Quoted 12 lines
block/partitions/mac.c +2 -2
index df03ca428e15..f372392af95b 100644
@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)
 		if (be16_to_cpu(part->signature) != MAC_PARTITION_MAGIC)
 			break;
 		put_partition(state, slot,
-			be32_to_cpu(part->start_block) * (secsize/512),
-			be32_to_cpu(part->block_count) * (secsize/512));
+			(sector_t)be32_to_cpu(part->start_block) * (secsize/512),
+			(sector_t)be32_to_cpu(part->block_count) * (secsize/512));

PLease just put this into a helper..

For this and others, we can add helpers later.

Signature · 1 line
Jens Axboe
author maintainer
#99 lore raw

PATCH 43/98 blk-mq: free the bitmaps of tags in the SRCU callback

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the iterators under tags_srcu are gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks, blk_mq_tagset_busy_iter() may be about to walk tags that __blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.

Free the bitmaps with the rest in the callback.

Fixes ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
---
 block/blk-mq-tag.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq-tag.c +4 -4
index aaea2e2c2f23..e3edf18697af 100644
@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)
 		kmemleak_free(page_address(page));
 		__free_pages(page, page->private);
 	}
+	/* An iterator may walk the bitmaps for as long as it may see the tags */
+	sbitmap_queue_free(&tags->bitmap_tags);
+	sbitmap_queue_free(&tags->breserved_tags);
 	kfree(tags);
 }
 
 void blk_mq_free_tags(struct blk_mq_tags *tags)
 {
-	sbitmap_queue_free(&tags->bitmap_tags);
-	sbitmap_queue_free(&tags->breserved_tags);
-
 	/* if tags pages is not allocated yet, free tags directly */
 	if (list_empty(&tags->page_list)) {
-		kfree(tags);
+		blk_mq_free_tags_callback(&tags->rcu_head);
 		return;
 	}
 
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 43/98
#100 lore raw

Re: [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback

Quoted 35 lines · Jens Axboe

From: Josef Bacik <[email protected]>

Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the iterators under tags_srcu are gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks, blk_mq_tagset_busy_iter() may be about to walk tags that __blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.

Free the bitmaps with the rest in the callback.

Fixes ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
---
  block/blk-mq-tag.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq-tag.c +3 -0
index aaea2e2c2f23..e3edf18697af 100644
@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)
  		kmemleak_free(page_address(page));
  		__free_pages(page, page->private);
  	}
+	/* An iterator may walk the bitmaps for as long as it may see the tags */
+	sbitmap_queue_free(&tags->bitmap_tags);
+	sbitmap_queue_free(&tags->breserved_tags);
  	kfree(tags);
  }
 
  void blk_mq_free_tags(struct blk_mq_tags *tags)
  {

block/blk-mq-tag.c static void blk_mq_free_tags_callback(struct rcu_head *head) · line 639

Quoted 10 lines
block/blk-mq-tag.c +1 -4
-	sbitmap_queue_free(&tags->bitmap_tags);
-	sbitmap_queue_free(&tags->breserved_tags);
-
  	/* if tags pages is not allocated yet, free tags directly */
  	if (list_empty(&tags->page_list)) {
-		kfree(tags);
+		blk_mq_free_tags_callback(&tags->rcu_head);
  		return;
  	}
 

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#101 lore raw

Re: [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback

In reply to Jens Axboe
Quoted 24 lines · Jens Axboe

From: Josef Bacik <[email protected]>

Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the iterators under tags_srcu are gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks, blk_mq_tagset_busy_iter() may be about to walk tags that __blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.

Free the bitmaps with the rest in the callback.

Fixes ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
---
 block/blk-mq-tag.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq-tag.c +0 -0
index aaea2e2c2f23..e3edf18697af 100644

block/blk-mq-tag.c static void blk_mq_free_tags_callback(struct rcu_head *head) · line 626

Quoted 4 lines
block/blk-mq-tag.c +1 -0
@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)
 		kmemleak_free(page_address(page));
 		__free_pages(page, page->private);
 	}
+	/* An iterator may walk the bitmaps for as long as it may see the tags */

Missing sentence ending and overly long line. Otherwise looks good.

author maintainer
#102 lore raw

PATCH 44/98 block: apply the segment size to the integrity buffer

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The integrity buffer is a single segment and must not exceed max_segment_size. max_integrity_io_size() applies that limit to the data size instead, which is off by a factor of 64 for 8 bytes of PI per 512 bytes of data. A SCSI host that doesn't set a maximum segment size gets 64KiB, lpfc and qla2xxx among them, so a DIX disk behind one of those does no IO above 64KiB.

Limit the buffer and derive the data size from that, which makes it 4MiB for that case.

Fixes ec7f31b2a2d3 ("block: make bio auto-integrity deadlock safe")
---
 include/linux/blkdev.h | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
include/linux/blkdev.h +5 -4
index d003a9d2d1f6..1a965fb5ae6f 100644
@@ -1819,11 +1819,12 @@ static inline int bio_split_rw_at(struct bio *bio,
  */
 static inline unsigned int max_integrity_io_size(struct queue_limits *lim)
 {
-	u64 max_intervals;
+	unsigned int buf_size = min_t(unsigned int, lim->max_segment_size,
+				      BLK_INTEGRITY_MAX_SIZE);
 
-	max_intervals = BLK_INTEGRITY_MAX_SIZE / lim->integrity.metadata_size;
-	return min_t(u64, lim->max_segment_size,
-		max_intervals << lim->integrity.interval_exp);
+	return min_t(u64, UINT_MAX,
+		(u64)(buf_size / lim->integrity.metadata_size) <<
+			lim->integrity.interval_exp);
 }
 
 #define DEFINE_IO_COMP_BATCH(name)	struct io_comp_batch name = { }
Signature · 1 line
2.56.0
Reviewed-by on v1 44/98
#103 lore raw

Re: [PATCH 44/98] block: apply the segment size to the integrity buffer

Quoted 8 lines · Jens Axboe

From: Josef Bacik <[email protected]>

The integrity buffer is a single segment and must not exceed max_segment_size. max_integrity_io_size() applies that limit to the data size instead, which is off by a factor of 64 for 8 bytes of PI per 512 bytes of data. A SCSI host that doesn't set a maximum segment size gets 64KiB, lpfc and qla2xxx among them, so a DIX disk behind one of those does no IO above 64KiB.

Looks good:

author maintainer
#104 lore raw

PATCH 45/98 nvmet: don't truncate the length of big discards

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_bdev_execute_discard() shifts the 32-bit nlb of a DSM range into sectors before widening it to sector_t, so on a 4k namespace the top bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards nothing, and both complete with success. The target announces no DMRSL, and hosts other than Linux (and nvme dsm passthrough) do send ranges that big.

Widen before the shift.

Fixes a07b4970f464 ("nvmet: add a generic NVMe target")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/io-cmd-bdev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/io-cmd-bdev.c +2 -1
index f2d9e8901df4..ed2e2c89e3e9 100644
@@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)
 		if (status)
 			break;
 
-		nr_sects = le32_to_cpu(range.nlb) << (ns->blksize_shift - 9);
+		nr_sects = (sector_t)le32_to_cpu(range.nlb) <<
+			   (ns->blksize_shift - 9);
 		__blkdev_issue_discard(ns->bdev,
 				nvmet_lba_to_sect(ns, range.slba), nr_sects,
 				GFP_KERNEL, &bio);
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 45/98
#105 lore raw

Re: [PATCH 45/98] nvmet: don't truncate the length of big discards

Quoted 26 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_bdev_execute_discard() shifts the 32-bit nlb of a DSM range into sectors before widening it to sector_t, so on a 4k namespace the top bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards nothing, and both complete with success. The target announces no DMRSL, and hosts other than Linux (and nvme dsm passthrough) do send ranges that big.

Widen before the shift.

Fixes a07b4970f464 ("nvmet: add a generic NVMe target")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/io-cmd-bdev.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/io-cmd-bdev.c +0 -0
index f2d9e8901df4..ed2e2c89e3e9 100644

drivers/nvme/target/io-cmd-bdev.c static void nvmet_bdev_execute_discard(struct nvmet_req *req) · line 396

Quoted 9 lines
drivers/nvme/target/io-cmd-bdev.c +2 -1
@@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)
  		if (status)
  			break;
 
-		nr_sects = le32_to_cpu(range.nlb) << (ns->blksize_shift - 9);
+		nr_sects = (sector_t)le32_to_cpu(range.nlb) <<
+			   (ns->blksize_shift - 9);
  		__blkdev_issue_discard(ns->bdev,
  				nvmet_lba_to_sect(ns, range.slba), nr_sects,
  				GFP_KERNEL, &bio);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#106 lore raw

PATCH 46/98 nvme-loop: delete the controller if a reset fails

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

If the reconnect of a reset fails, nvme_loop_reset_ctrl_work() tears the controller down by hand with nvme_uninit_ctrl(). Since that was split from nvme_stop_ctrl() it no longer removes the namespaces, which hold references to the controller. A reset while the target no longer accepts the host (allow_any_host 0) is enough to get there: a sysfs reset hangs in kernfs_drain(), an ioctl reset leaves a controller stuck in CONNECTING with its block devices, the cntlid can't be reused, and module unload deletes the controller a second time and hits a use-after-free.

Do what the other fabrics drivers do and call nvme_delete_ctrl(). The delete path quiesces the admin queue and only unquiesces it if the queue is live, which it isn't after a failed reset, so unquiesce it unconditionally before destroying it so a stuck request can't block the freeze.

Fixes d09f2b45f346 ("nvme: split nvme_uninit_ctrl into stop and uninit")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/loop.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/target/loop.c +3 -2
index fcb1f8186fdd..93ba6357a850 100644
@@ -422,6 +422,8 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)
 static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl)
 {
 	nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl));
+	/* Left quiesced if a failed reset has left no admin queue behind */
+	nvme_unquiesce_admin_queue(ctrl);
 	nvme_remove_admin_tag_set(ctrl);
 }
 
@@ -485,9 +487,8 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 	nvme_cancel_admin_tagset(&ctrl->ctrl);
 	nvme_loop_destroy_admin_queue(ctrl);
 out_disable:
-	nvme_remove_admin_tag_set(&ctrl->ctrl);
 	dev_warn(ctrl->ctrl.device, "Removing after reset failure\n");
-	nvme_uninit_ctrl(&ctrl->ctrl);
+	nvme_delete_ctrl(&ctrl->ctrl);
 }
 
 static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
Signature · 1 line
2.56.0
author maintainer
#107 lore raw

PATCH 47/98 nvme: refuse metadata sizes above 255 bytes

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The metadata size of an LBA format is a 16-bit field, but blk_integrity->metadata_size is an unsigned char. nvme_init_integrity()
truncates: a 264-byte format tells the block layer 8, so the integrity buffer is 8 bytes per block while the controller transfers 264, and a read overflows it. 256 bytes becomes 0 and every request fails in nvme_setup_rw().

Treat such a format like the other unsupported ones: capacity 0, passthrough only.

Fixes a48f041d91bf ("block: Reduce the size of struct blk_integrity")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/core.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +4 -0
index af71d02b8476..69d3ba98d50b 100644
@@ -1867,6 +1867,10 @@ static bool nvme_init_integrity(struct nvme_ns_head *head,
 	    !(head->features & NVME_NS_METADATA_SUPPORTED))
 		return nvme_ns_has_pi(head);
 
+	/* struct blk_integrity has eight bits for the metadata size. */
+	if (head->ms > U8_MAX)
+		return false;
+
 	switch (head->pi_type) {
 	case NVME_NS_DPS_PI_TYPE3:
 		switch (head->guard_type) {
Signature · 1 line
2.56.0
maintainer
#108 lore raw

Re: [PATCH 47/98] nvme: refuse metadata sizes above 255 bytes

drivers/nvme/host/core.c static bool nvme_init_integrity(struct nvme_ns_head *head, · line 1872

Quoted 3 lines · Jens Axboe
drivers/nvme/host/core.c +3 -0
+	/* struct blk_integrity has eight bits for the metadata size. */
+	if (head->ms > U8_MAX)
+		return false;

I've never seen a device with 256 or greater metadata, but if the specs allow it, should the block layer accomodate? It's just the queue_limits, so widening the type isn't a big deal; it fits in the existing padding anyway so the struct doesn't even have to grow.

There's potential issues with logical_block_metadata_cap, but that's just user space reporting and doesn't affect the kernel behavior. Seems better than throttling the capacity to 0 for an unsupported format that we can totally support.

author maintainer
#109 lore raw

PATCH 48/98 nvmet-tcp: rewind the iterator after non-data records

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A TLS record that isn't application data (an Alert, say) lands in whatever buffer was waiting for a PDU header, command data or a digest. nvmet_tcp_tls_record_ok() fails the receive and the bytes aren't counted, but the iterator has already advanced. tls_alert_recv() then expects the iterator at the start of the alert, and if the alert filled the kvec it has stepped off the end of it:

  BUG: KASAN: stack-out-of-bounds in tls_alert_recv+0x33/0x170
   nvmet_tcp_tls_record_ok+0x139/0x1a0
   nvmet_tcp_try_recv_pdu+0x37f/0x12a0

Six bytes of a header followed by an alert from the host is all it takes. For command data the following bytes are stored behind the dropped record and the command runs before its data is complete.

Rewind the iterator by what was received.

Fixes a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/nvme/target/tcp.c +7 -5
index c38ea3501ea5..a0d942c34d3a 100644
@@ -1174,7 +1174,7 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 }
 
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
-		struct msghdr *msg, char *cbuf)
+		struct msghdr *msg, char *cbuf, int len)
 {
 	struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;
 	u8 ctype, level, description;
@@ -1187,6 +1187,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 	case TLS_RECORD_TYPE_DATA:
 		break;
 	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);
 		if (level == TLS_ALERT_LEVEL_FATAL) {
 			pr_err("queue %d: TLS Alert desc %u\n",
@@ -1200,6 +1201,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		break;
 	default:
 		/* discard this record type */
+		iov_iter_revert(&msg->msg_iter, len);
 		pr_err("queue %d: TLS record %d unhandled\n",
 		       queue->idx, ctype);
 		ret = -EAGAIN;
@@ -1228,7 +1230,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1290,7 +1292,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			return len;
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
 				return ret;
 		}
@@ -1331,7 +1333,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1803,7 +1805,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 		return len;
 	}
 
-	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 	if (ret < 0)
 		return ret;
 
Signature · 1 line
2.56.0
#110 lore raw

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

Quoted 3 lines · Jens Axboe
drivers/nvme/target/tcp.c +1 -0
 	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);

...

drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1295

Quoted 4 lines
drivers/nvme/target/tcp.c +1 -1
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);

Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT is still broken in two ways here:

1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC initialized with iov_iter_bvec(), whereas tls_alert_recv() (net/handshake/alert.c) unconditionally reads union member msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases bvec->bv_page (struct page *), so tls_alert_recv() reads the first two bytes of struct page instead of the received alert payload and silently ignores fatal alerts when byte 0 != TLS_ALERT_LEVEL_FATAL. (Also, up to 2 bytes of the command's SG page were overwritten before iov_iter_revert().)

2. If queue->left == 1 when a 2-byte TLS alert arrives, sock_recvmsg() returns len == 1, and tls_alert_recv() reads data[0] and data[1] without checking len >= 2.

tls_alert_recv() either needs to handle non-kvec iterators and len < 2 properly, or nvmet-tcp should treat any TLS alert record during live PDU/data transfer as fatal without calling tls_alert_recv() on cmd->recv_msg.

maintainer
#111 lore raw

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

Quoted 23 lines · Bart Van Assche
Quoted 3 lines · Jens Axboe

case TLS_RECORD_TYPE_ALERT:

drivers/nvme/target/tcp.c +1 -0
+		iov_iter_revert(&msg->msg_iter, len);
  		tls_alert_recv(queue->sock->sk, msg, &level, &description);

...

drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1295

Quoted 4 lines
  		if (queue->tls_pskid) {
  			ret = nvmet_tcp_tls_record_ok(cmd->queue,
drivers/nvme/target/tcp.c +1 -1
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);

Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT is still broken in two ways here:

1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC initialized with iov_iter_bvec(), whereas tls_alert_recv() (net/handshake/alert.c) unconditionally reads union member msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases bvec->bv_page (struct page *), so tls_alert_recv() reads the first two bytes of struct page instead of the received alert payload and silently ignores fatal alerts when byte 0 != TLS_ALERT_LEVEL_FATAL. (Also, up to 2 bytes of the command's SG page were overwritten before iov_iter_revert().)

Quoted 9 lines

2. If queue->left == 1 when a 2-byte TLS alert arrives, sock_recvmsg() returns len == 1, and tls_alert_recv() reads data[0] and data[1] without checking len >= 2.

tls_alert_recv() either needs to handle non-kvec iterators and len < 2 properly, or nvmet-tcp should treat any TLS alert record during live PDU/data transfer as fatal without calling tls_alert_recv() on cmd->recv_msg.

There is a larger rework under way for handling TLS alerts, so we should defer this until the rework is merged.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#112 lore raw

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

In reply to Jens Axboe
Quoted 3 lines · Jens Axboe
(quoted diff) +1 -0
 	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);

...

Review

Quoted 4 lines
(quoted diff) +1 -1
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);

Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT is still broken in two ways here:

1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC initialized with iov_iter_bvec(), whereas tls_alert_recv() in net/handshake/alert.c unconditionally reads union member msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases bvec->bv_page (struct page *), so tls_alert_recv() reads the first two bytes of struct page instead of the received alert payload and ignores fatal alerts whenever byte 0 != TLS_ALERT_LEVEL_FATAL.
2. If queue->left == 1 when a 2-byte TLS alert arrives (or a peer sends a 1-byte malformed alert record), sock_recvmsg() returns len == 1, and tls_alert_recv() reads data[0] and data[1] without checking that 2 bytes were received.

To fix this inside nvmet-tcp without cross-tree changes to net/handshake/alert.c, nvmet_tcp_tls_record_ok() can extract the alert bytes from a local ITER_SOURCE copy of the rewound msg->msg_iter via copy_from_iter_full(), pull the second byte from queue->sock when len == 1 and !(msg->msg_flags & (MSG_EOR | MSG_PEEK)), reject < 2-byte alerts with -EPROTO, and pass a local 2-byte ITER_KVEC msghdr to tls_alert_recv() so its tracepoint still works.

Note: patch 51/98 will need a trivial context adjustment around the nvmet_tcp_tls_alert_recv() call in nvmet_tcp_tls_record_ok() (and propagating MSG_EOR from rem_msg lets 51/98's nvmet_tcp_tls_drop_rest() check work unchanged).

Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

drivers/nvme/target/tcp.c +57 -6
index c38ea3501ea5..e6eadfc35d01 100644
@@ -1173,8 +1173,54 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 	return false;
 }
 
+static int nvmet_tcp_tls_alert_recv(struct nvmet_tcp_queue *queue,
+		struct msghdr *msg, int len, u8 *level, u8 *description)
+{
+	struct iov_iter iter = msg->msg_iter;
+	size_t n = min_t(size_t, len, 2);
+	u8 alert[2];
+	struct kvec kvec = {
+		.iov_base = alert,
+		.iov_len = sizeof(alert),
+	};
+	struct msghdr alert_msg = {};
+
+	iter.data_source = ITER_SOURCE;
+	if (!copy_from_iter_full(alert, n, &iter))
+		return -EPROTO;
+
+	if (n < sizeof(alert)) {
+		char cbuf[CMSG_LEN(sizeof(char))] = {};
+		struct kvec rem = {
+			.iov_base = alert + n,
+			.iov_len = sizeof(alert) - n,
+		};
+		struct msghdr rem_msg = {
+			.msg_control = cbuf,
+			.msg_controllen = sizeof(cbuf),
+			.msg_flags = MSG_DONTWAIT,
+		};
+		int ret;
+
+		if (msg->msg_flags & (MSG_EOR | MSG_PEEK))
+			return -EPROTO;
+		ret = kernel_recvmsg(queue->sock, &rem_msg, &rem, 1,
+				     rem.iov_len, rem_msg.msg_flags);
+		if (ret != rem.iov_len ||
+		    tls_get_record_type(queue->sock->sk,
+					(struct cmsghdr *)cbuf) !=
+		    TLS_RECORD_TYPE_ALERT)
+			return -EPROTO;
+		msg->msg_flags |= rem_msg.msg_flags & MSG_EOR;
+	}
+
+	iov_iter_kvec(&alert_msg.msg_iter, ITER_DEST, &kvec, 1, sizeof(alert));
+	tls_alert_recv(queue->sock->sk, &alert_msg, level, description);
+	return 0;
+}
+
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
-		struct msghdr *msg, char *cbuf)
+		struct msghdr *msg, char *cbuf, int len)
 {
 	struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;
 	u8 ctype, level, description;
@@ -1187,7 +1233,11 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 	case TLS_RECORD_TYPE_DATA:
 		break;
 	case TLS_RECORD_TYPE_ALERT:
-		tls_alert_recv(queue->sock->sk, msg, &level, &description);
+		iov_iter_revert(&msg->msg_iter, len);
+		ret = nvmet_tcp_tls_alert_recv(queue, msg, len, &level,
+					       &description);
+		if (ret)
+			break;
 		if (level == TLS_ALERT_LEVEL_FATAL) {
 			pr_err("queue %d: TLS Alert desc %u\n",
 			       queue->idx, description);
@@ -1200,6 +1250,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		break;
 	default:
 		/* discard this record type */
+		iov_iter_revert(&msg->msg_iter, len);
 		pr_err("queue %d: TLS record %d unhandled\n",
 		       queue->idx, ctype);
 		ret = -EAGAIN;
@@ -1228,7 +1279,7 @@ recv:
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1290,7 +1341,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			return len;
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
 				return ret;
 		}
@@ -1331,7 +1382,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1803,7 +1854,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 		return len;
 	}
 
-	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 	if (ret < 0)
 		return ret;
 
author maintainer
#113 lore raw

PATCH 49/98 nvmet-tcp: ask for the TLS record type on every receive

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_alloc_cmd() points recv_msg.msg_control at recv_cbuf once. The first receive that stores a record type there advances msg_control and leaves msg_controllen at 0 for the lifetime of the command. From then on nvmet_tcp_try_recv_data() sees the record type of that first
receive: if it was an alert, all later data is dropped as alerts and the queue stalls, if it was data, a later alert can't be reported and tls_sw_recvmsg() fails with -EIO.

Set up the control buffer before each receive, like the other receive paths do.

Fixes a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/target/tcp.c +5 -0
index a0d942c34d3a..261672ab5c1f 100644
@@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 	int len, ret;
 
 	while (msg_data_left(&cmd->recv_msg)) {
+		if (queue->tls_pskid) {
+			/* The last receive has used it up */
+			cmd->recv_msg.msg_control = cmd->recv_cbuf;
+			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
+		}
 		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
 		if (len <= 0)
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 49/98
#114 lore raw

Re: [PATCH 49/98] nvmet-tcp: ask for the TLS record type on every receive

Quoted 29 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_alloc_cmd() points recv_msg.msg_control at recv_cbuf once. The first receive that stores a record type there advances msg_control and leaves msg_controllen at 0 for the lifetime of the command. From then on nvmet_tcp_try_recv_data() sees the record type of that first
receive: if it was an alert, all later data is dropped as alerts and the queue stalls, if it was data, a later alert can't be reported and tls_sw_recvmsg() fails with -EIO.

Set up the control buffer before each receive, like the other receive paths do.

Fixes a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/tcp.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/nvme/target/tcp.c +0 -0
index a0d942c34d3a..261672ab5c1f 100644
@@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
  	int len, ret;

drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1296

Quoted 10 lines
drivers/nvme/target/tcp.c +5 -0
 
  	while (msg_data_left(&cmd->recv_msg)) {
+		if (queue->tls_pskid) {
+			/* The last receive has used it up */
+			cmd->recv_msg.msg_control = cmd->recv_cbuf;
+			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
+		}
  		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
  			cmd->recv_msg.msg_flags);
  		if (len <= 0)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#115 lore raw

PATCH 50/98 nvme-tcp: wait for an in-flight TLS handshake callback

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

If the wait for the TLS handshake times out or is interrupted, nvme_tcp_start_tls() cancels the request and returns. tls_handshake_cancel() returns false if the handshake already completed, in which case nvme_tcp_tls_done() may still be running while nvme_tcp_alloc_queue() tears the queue down. On the first connect that frees the controller, including the queue array the callback writes to.

If the cancel lost, wait for the callback to signal completion before returning, like the SUNRPC callers do.

Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +6 -1
index 7190eb10c248..d735994ee517 100644
@@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
 		dev_err(nctrl->device,
 			"queue %d: TLS handshake failed, error %d\n",
 			qid, ret);
-		tls_handshake_cancel(queue->sock->sk);
+		/*
+		 * If the cancellation lost to handshake_complete(),
+		 * nvme_tcp_tls_done() is in flight and still uses the queue.
+		 */
+		if (!tls_handshake_cancel(queue->sock->sk))
+			wait_for_completion(&queue->tls_complete);
 	} else {
 		if (queue->tls_err) {
 			dev_err(nctrl->device,
Signature · 1 line
2.56.0
maintainer
#116 lore raw

Re: [PATCH 50/98] nvme-tcp: wait for an in-flight TLS handshake callback

Quoted 30 lines · Jens Axboe

From: Josef Bacik <[email protected]>

If the wait for the TLS handshake times out or is interrupted, nvme_tcp_start_tls() cancels the request and returns. tls_handshake_cancel() returns false if the handshake already completed, in which case nvme_tcp_tls_done() may still be running while nvme_tcp_alloc_queue() tears the queue down. On the first connect that frees the controller, including the queue array the callback writes to.

If the cancel lost, wait for the callback to signal completion before returning, like the SUNRPC callers do.

Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
  drivers/nvme/host/tcp.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +0 -0
index 7190eb10c248..d735994ee517 100644
@@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
  		dev_err(nctrl->device,
  			"queue %d: TLS handshake failed, error %d\n",
  			qid, ret);

drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1838

Quoted 10 lines
drivers/nvme/host/tcp.c +6 -1
-		tls_handshake_cancel(queue->sock->sk);
+		/*
+		 * If the cancellation lost to handshake_complete(),
+		 * nvme_tcp_tls_done() is in flight and still uses the queue.
+		 */
+		if (!tls_handshake_cancel(queue->sock->sk))
+			wait_for_completion(&queue->tls_complete);
  	} else {
  		if (queue->tls_err) {
  			dev_err(nctrl->device,

This has already been reported to the linux-nvme list
(https://lore.kernel.org/linux-nvme/[email protected]/)
Please drop.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#117 lore raw

PATCH 51/98 nvmet-tcp: look again after a short read with TLS

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The receive helpers return -EAGAIN both for a short read and for a dropped non-data TLS record, and nvmet_tcp_try_recv() takes that as "socket empty" and waits for the next data_ready. On plain TCP that's right. tls_sw_recvmsg() stops where the record type changes though, so what follows an alert or a handshake record may already be queued and nobody looks at it until the host sends something else.

Return 1 in those cases so nvmet_tcp_try_recv() counts it against the budget and tries again until the socket itself says -EAGAIN. While at it, drain the rest of a dropped record using MSG_EOR instead of pulling it in PDU-sized pieces, and ratelimit the messages since the host controls how many there are.

Fixes a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 54 +++++++++++++++++++++++++++++++--------
 1 file changed, 43 insertions(+), 11 deletions(-)
drivers/nvme/target/tcp.c +43 -11
index 261672ab5c1f..a2a5f39d2bb7 100644
@@ -1173,6 +1173,24 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 	return false;
 }
 
+/*
+ * The buffer took as much of the record as the PDU still wants, which may be
+ * one byte.  The rest comes with the type of the record, never with that
+ * which follows: throw it away at once.
+ */
+static void nvmet_tcp_tls_drop_rest(struct nvmet_tcp_queue *queue)
+{
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT,
+	};
+
+	iov_iter_discard(&msg.msg_iter, ITER_DEST, TLS_MAX_PAYLOAD_SIZE);
+	sock_recvmsg(queue->sock, &msg, msg.msg_flags);
+}
+
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		struct msghdr *msg, char *cbuf, int len)
 {
@@ -1190,26 +1208,38 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);
 		if (level == TLS_ALERT_LEVEL_FATAL) {
-			pr_err("queue %d: TLS Alert desc %u\n",
-			       queue->idx, description);
+			pr_err_ratelimited("queue %d: TLS Alert desc %u\n",
+					   queue->idx, description);
 			ret = -ENOTCONN;
 		} else {
-			pr_warn("queue %d: TLS Alert desc %u\n",
-			       queue->idx, description);
+			pr_warn_ratelimited("queue %d: TLS Alert desc %u\n",
+					    queue->idx, description);
 			ret = -EAGAIN;
 		}
 		break;
 	default:
 		/* discard this record type */
 		iov_iter_revert(&msg->msg_iter, len);
-		pr_err("queue %d: TLS record %d unhandled\n",
-		       queue->idx, ctype);
+		pr_err_ratelimited("queue %d: TLS record %d unhandled\n",
+				   queue->idx, ctype);
 		ret = -EAGAIN;
 		break;
 	}
+	if (ret == -EAGAIN && !(msg->msg_flags & MSG_EOR))
+		nvmet_tcp_tls_drop_rest(queue);
 	return ret;
 }
 
+/*
+ * A receive from a TLS socket ends with the record, or where the type of the
+ * records changes.  There may be more behind a record that has been dropped,
+ * and nobody will tell: have nvmet_tcp_try_recv() look again.
+ */
+static int nvmet_tcp_tls_again(int ret)
+{
+	return ret == -EAGAIN ? 1 : ret;
+}
+
 static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 {
 	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
@@ -1224,6 +1254,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		msg.msg_control = cbuf;
 		msg.msg_controllen = sizeof(cbuf);
+		msg.msg_flags &= ~MSG_EOR;
 	}
 	len = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
@@ -1232,13 +1263,13 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
-			return ret;
+			return nvmet_tcp_tls_again(ret);
 	}
 
 	queue->offset += len;
 	queue->left -= len;
 	if (queue->left)
-		return -EAGAIN;
+		return queue->tls_pskid && len ? 1 : -EAGAIN;
 
 	if (queue->offset == sizeof(struct nvme_tcp_hdr)) {
 		u8 hdgst = nvmet_tcp_hdgst_len(queue);
@@ -1290,6 +1321,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			/* The last receive has used it up */
 			cmd->recv_msg.msg_control = cmd->recv_cbuf;
 			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
+			cmd->recv_msg.msg_flags &= ~MSG_EOR;
 		}
 		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
@@ -1299,7 +1331,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
 					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
-				return ret;
+				return nvmet_tcp_tls_again(ret);
 		}
 
 		cmd->pdu_recv += len;
@@ -1340,13 +1372,13 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
-			return ret;
+			return nvmet_tcp_tls_again(ret);
 	}
 
 	queue->offset += len;
 	queue->left -= len;
 	if (queue->left)
-		return -EAGAIN;
+		return queue->tls_pskid && len ? 1 : -EAGAIN;
 
 	if (queue->data_digest && cmd->exp_ddgst != cmd->recv_ddgst) {
 		pr_err("queue %d: cmd %d pdu (%d) data digest error: recv %#x expected %#x\n",
Signature · 1 line
2.56.0
author maintainer
#118 lore raw

PATCH 52/98 nvme-tcp: keep the socket while the queue is polled

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Quiescing keeps ->queue_rq() out, not ->poll(): bio_poll() enters even a frozen queue. So nvme_tcp_poll() can run while error recovery or a reset stops and frees the queues, and keeps running afterwards for the requeued requests whose bios still carry a cookie. The NVME_TCP_Q_LIVE check doesn't help, queue->sock->sk is read before it and nvme_tcp_free_queue() has set queue->sock to NULL:

  BUG: KASAN: null-ptr-deref in nvme_tcp_poll+0x42/0x1c0

The poller dies holding a queue reference and the controller never reconnects.

nvme_tcp_stop_queue() clears the bit under queue_lock. Trylock it in nvme_tcp_poll() too and check the bit and the socket under it; whoever holds it is either polling or stopping this queue. That means the lock can't live with the queue, so initialize it with the controller.

Fixes 1a9460cef571 ("nvme-tcp: support simple polling")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)
drivers/nvme/host/tcp.c +22 -7
index d735994ee517..98c55fba53c6 100644
@@ -1530,7 +1530,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->queue_lock);
 	mutex_destroy(&queue->pf_cache_lock);
 }
 
@@ -1855,7 +1854,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	int ret, rcv_pdu_size;
 	struct file *sock_file;
 
-	mutex_init(&queue->queue_lock);
 	queue->ctrl = ctrl;
 	init_llist_head(&queue->req_list);
 	INIT_LIST_HEAD(&queue->send_list);
@@ -1993,7 +1991,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->queue_lock);
 	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
@@ -2922,18 +2919,33 @@ static void nvme_tcp_map_queues(struct blk_mq_tag_set *set)
 static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
 {
 	struct nvme_tcp_queue *queue = hctx->driver_data;
-	struct sock *sk = queue->sock->sk;
-	int ret;
+	struct sock *sk;
+	int ret = 0;
 
 	if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
 		return 0;
 
+	/*
+	 * Nothing in the block layer keeps a poller out while the queue is
+	 * torn down.  The lock keeps the socket until we are through; if
+	 * somebody else has it, he polls, or the queue is being stopped.
+	 */
+	if (!mutex_trylock(&queue->queue_lock))
+		return 0;
+	if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
+		goto unlock;
+
+	sk = queue->sock->sk;
 	set_bit(NVME_TCP_Q_POLLING, &queue->flags);
 	if (sk_can_busy_loop(sk) && skb_queue_empty_lockless(&sk->sk_receive_queue))
 		sk_busy_loop(sk, true);
 	ret = nvme_tcp_try_recv(queue);
 	clear_bit(NVME_TCP_Q_POLLING, &queue->flags);
-	return ret < 0 ? ret : queue->nr_cqe;
+	if (ret >= 0)
+		ret = queue->nr_cqe;
+unlock:
+	mutex_unlock(&queue->queue_lock);
+	return ret;
 }
 
 static int nvme_tcp_get_address(struct nvme_ctrl *ctrl, char *buf, int size)
@@ -3020,7 +3032,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
 	struct nvme_tcp_ctrl *ctrl;
-	int ret;
+	int ret, i;
 
 	ctrl = kzalloc_obj(*ctrl);
 	if (!ctrl)
@@ -3089,6 +3101,9 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		ret = -ENOMEM;
 		goto out_free_ctrl;
 	}
+	/* A poller may try the lock of a queue that has been freed */
+	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+		mutex_init(&ctrl->queues[i].queue_lock);
 
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
Signature · 1 line
2.56.0
author maintainer
#119 lore raw

PATCH 53/98 nvme-tcp: don't get stuck on a TLS control record

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The host receives with read_sock(), and tls_sw_read_sock() won't hand out a non-data record: it puts it back and returns -EINVAL. nvme_tcp_io_work() bails on the error and finds the same record again next time, so nothing behind it is ever read. One warning alert or NewSessionTicket after the ICResp kills the queue: commands time out and the controller resets. The Linux target sends none, but others may.

Fetch such a record with recvmsg() instead. A fatal alert starts error recovery, anything else is dropped whole, with a log line, and the queue carries on. io_work and a poller may both see the record, so a bit makes sure only one of them deals with it, and that one retries read_sock() afterwards.

Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 86 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +86 -1
index 98c55fba53c6..87ecee0665c7 100644
@@ -95,6 +95,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
 	NVME_TCP_Q_IO_CPU_SET	= 3,
+	NVME_TCP_Q_TLS_CTRL	= 4,
 };
 
 enum nvme_tcp_recv_state {
@@ -1418,7 +1419,7 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
 	return ret;
 }
 
-static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
+static int nvme_tcp_read_sock(struct nvme_tcp_queue *queue)
 {
 	struct socket *sock = queue->sock;
 	struct sock *sk = sock->sk;
@@ -1431,6 +1432,90 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	queue->nr_cqe = 0;
 	consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
 	release_sock(sk);
+	return consumed;
+}
+
+/*
+ * read_sock() leaves a TLS record that is no data where it is, in front of
+ * all that follows.  Take it out of the way.
+ */
+static int nvme_tcp_recv_tls_ctrl(struct nvme_tcp_queue *queue)
+{
+	struct nvme_ctrl *ctrl = &queue->ctrl->ctrl;
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT,
+	};
+	u8 ctype, level, description, data[2] = {};
+	struct kvec iov = { .iov_base = data, .iov_len = sizeof(data) };
+	int len;
+
+	/*
+	 * io_work and a poller may both have met the record, and recvmsg()
+	 * would take data as well.  Only one of them goes on, and he looks
+	 * again: nobody else removes such a record from now on.
+	 */
+	if (test_and_set_bit_lock(NVME_TCP_Q_TLS_CTRL, &queue->flags))
+		return 0;
+	len = nvme_tcp_read_sock(queue);
+	if (len != -EINVAL || !queue->rd_enabled)
+		goto out;
+
+	len = kernel_recvmsg(queue->sock, &msg, &iov, 1, iov.iov_len,
+			     msg.msg_flags);
+	if (len < 0)
+		goto out;
+
+	ctype = tls_get_record_type(queue->sock->sk, (struct cmsghdr *)cbuf);
+	switch (ctype) {
+	case 0:
+	case TLS_RECORD_TYPE_DATA:
+		goto fail;
+	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg.msg_iter, len);
+		tls_alert_recv(queue->sock->sk, &msg, &level, &description);
+		dev_warn_ratelimited(ctrl->device,
+				     "queue %d: TLS Alert level %u desc %u\n",
+				     nvme_tcp_queue_id(queue), level,
+				     description);
+		if (level == TLS_ALERT_LEVEL_FATAL)
+			goto fail;
+		break;
+	default:
+		dev_warn_ratelimited(ctrl->device,
+				     "queue %d: TLS record %u dropped\n",
+				     nvme_tcp_queue_id(queue), ctype);
+		break;
+	}
+
+	if (!(msg.msg_flags & MSG_EOR)) {
+		msg.msg_control = cbuf;
+		msg.msg_controllen = sizeof(cbuf);
+		iov_iter_discard(&msg.msg_iter, ITER_DEST,
+				 TLS_MAX_PAYLOAD_SIZE);
+		sock_recvmsg(queue->sock, &msg, MSG_DONTWAIT);
+	}
+	len = 1;
+out:
+	clear_bit_unlock(NVME_TCP_Q_TLS_CTRL, &queue->flags);
+	return len;
+
+fail:
+	queue->rd_enabled = false;
+	nvme_tcp_error_recovery(ctrl);
+	len = -ENOTCONN;
+	goto out;
+}
+
+static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
+{
+	int consumed = nvme_tcp_read_sock(queue);
+
+	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
+	    queue->rd_enabled)
+		consumed = nvme_tcp_recv_tls_ctrl(queue);
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
Signature · 1 line
2.56.0
maintainer
#120 lore raw

Re: [PATCH 53/98] nvme-tcp: don't get stuck on a TLS control record

Quoted 21 lines · Jens Axboe

From: Josef Bacik <[email protected]>

The host receives with read_sock(), and tls_sw_read_sock() won't hand out a non-data record: it puts it back and returns -EINVAL. nvme_tcp_io_work() bails on the error and finds the same record again next time, so nothing behind it is ever read. One warning alert or NewSessionTicket after the ICResp kills the queue: commands time out and the controller resets. The Linux target sends none, but others may.

Fetch such a record with recvmsg() instead. A fatal alert starts error recovery, anything else is dropped whole, with a log line, and the queue carries on. io_work and a poller may both see the record, so a bit makes sure only one of them deals with it, and that one retries read_sock() afterwards.

Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Quoted 4 lines
---
  drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 86 insertions(+), 1 deletion(-)

TLS record handling is currently being reworked, so please drop.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#121 lore raw

PATCH 54/98 nvme-tcp: start error recovery after a TLS KeyUpdate

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Once the controller sends a KeyUpdate, kTLS fails every receive with -EKEYEXPIRED until a new key is installed, and nobody on the host side does that. nvme_tcp_io_work() returns on the error and the queue is dead until a command or the keep-alive times out.

The connection is unusable at that point, say so and start error recovery right away.

Fixes 0471b1093e3a ("tls: block decryption when a rekey is pending")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/nvme/host/tcp.c +8 -0
index 87ecee0665c7..64f563686938 100644
@@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
 	    queue->rd_enabled)
 		consumed = nvme_tcp_recv_tls_ctrl(queue);
+	if (consumed == -EKEYEXPIRED && queue->rd_enabled) {
+		/* Nobody is going to install the new key */
+		dev_err(queue->ctrl->ctrl.device,
+			"queue %d: TLS KeyUpdate is not supported\n",
+			nvme_tcp_queue_id(queue));
+		queue->rd_enabled = false;
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
+	}
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
Signature · 1 line
2.56.0
maintainer
#122 lore raw

Re: [PATCH 54/98] nvme-tcp: start error recovery after a TLS KeyUpdate

Quoted 29 lines · Jens Axboe

From: Josef Bacik <[email protected]>

Once the controller sends a KeyUpdate, kTLS fails every receive with -EKEYEXPIRED until a new key is installed, and nobody on the host side does that. nvme_tcp_io_work() returns on the error and the queue is dead until a command or the keep-alive times out.

The connection is unusable at that point, say so and start error recovery right away.

Fixes 0471b1093e3a ("tls: block decryption when a rekey is pending")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
  drivers/nvme/host/tcp.c | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/nvme/host/tcp.c +1 -0
index 87ecee0665c7..64f563686938 100644
@@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
  	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
  	    queue->rd_enabled)
  		consumed = nvme_tcp_recv_tls_ctrl(queue);
+	if (consumed == -EKEYEXPIRED && queue->rd_enabled) {

drivers/nvme/host/tcp.c static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) · line 1529

Quoted 10 lines
drivers/nvme/host/tcp.c +7 -0
+		/* Nobody is going to install the new key */
+		dev_err(queue->ctrl->ctrl.device,
+			"queue %d: TLS KeyUpdate is not supported\n",
+			nvme_tcp_queue_id(queue));
+		queue->rd_enabled = false;
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
+	}
  	return consumed == -EAGAIN ? 0 : consumed;
  }
 

Already been reported (https://lore.kernel.org/linux-nvme/[email protected]/).
Please drop.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#123 lore raw

PATCH 55/98 nvme-tcp: don't act on PDUs for a command being sent

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A request belongs to the send path from the moment it's put on req_list until its last byte is handed to the socket, but the receive path acts on whatever the controller sends for it. A response completes the request while the send side is still reading from its bio:

  BUG: KASAN: slab-use-after-free in nvme_tcp_try_send_data+0x175/0x6b0
   nvme_tcp_try_send+0x4af/0x600
   nvme_tcp_io_work+0x249/0x380

and an R2T resets and requeues it, since the queued check only looks at the two lists and a request being sent is on neither, so it ends up completed twice. An honest controller doesn't do that, but its answer can legitimately overtake a sender that has handed over the last byte and not yet cleared queue->request.

Make such a PDU wait: don't consume the bytes that complete its header, so it's seen again by the next receive. For that nvme_tcp_req_busy() has to find the request in one of the three places at all times, so nvme_tcp_fetch_request() sets queue->request before taking it off the lists. io_work reruns when the sender was io_work or the socket was full, waits for send_mutex when the submitter was sending, and a poller just polls again.

Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 73 insertions(+), 14 deletions(-)
drivers/nvme/host/tcp.c +73 -14
index 64f563686938..06c205703cf5 100644
@@ -123,6 +123,7 @@ struct nvme_tcp_queue {
 	size_t			data_remaining;
 	size_t			ddgst_remaining;
 	unsigned int		nr_cqe;
+	bool			rcv_parked;
 
 	/* send state */
 	struct nvme_tcp_request *request;
@@ -465,11 +466,29 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)
 			return NULL;
 	}
 
+	WRITE_ONCE(queue->request, req);
+	/* nvme_tcp_req_busy() must find the request in one place at any time */
+	smp_wmb();
 	list_del_init(&req->entry);
 	init_llist_node(&req->lentry);
 	return req;
 }
 
+/*
+ * Does the send path still hold @req? As long as it does the request must
+ * neither be completed nor be queued once more, whatever the controller says.
+ */
+static bool nvme_tcp_req_busy(struct nvme_tcp_queue *queue,
+			      struct nvme_tcp_request *req)
+{
+	if (llist_on_list(&req->lentry) || !list_empty(&req->entry))
+		return true;
+	/* pairs with the barrier in nvme_tcp_fetch_request() */
+	smp_rmb();
+	/* pairs with the release in nvme_tcp_done_send_req() */
+	return smp_load_acquire(&queue->request) == req;
+}
+
 #define NVME_TCP_CRC_SEED (~0)
 
 static inline void nvme_tcp_ddgst_update(u32 *crcp,
@@ -672,6 +691,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 	}
 
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
+
 	if (req->status == cpu_to_le16(NVME_SC_SUCCESS))
 		req->status = cqe->status;
 
@@ -707,6 +729,9 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
 	}
 
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
+
 	if (!blk_rq_payload_bytes(rq) || !req->curr_bio || !req->data_len) {
 		dev_err(queue->ctrl->ctrl.device,
 			"queue %d tag %#x unexpected data\n",
@@ -800,6 +825,8 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
 		return -ENOENT;
 	}
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
 
 	if (unlikely(rq_data_dir(rq) != WRITE)) {
 		dev_err(queue->ctrl->ctrl.device,
@@ -829,14 +856,6 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
 		return -EPROTO;
 	}
 
-	if (llist_on_list(&req->lentry) ||
-	    !list_empty(&req->entry)) {
-		dev_err(queue->ctrl->ctrl.device,
-			"req %d unexpected r2t while processing request\n",
-			rq->tag);
-		return -EPROTO;
-	}
-
 	req->pdu_len = 0;
 	req->h2cdata_left = r2t_length;
 	req->h2cdata_offset = r2t_offset;
@@ -939,17 +958,32 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
 	switch (hdr->type) {
 	case nvme_tcp_c2h_data:
-		return nvme_tcp_handle_c2h_data(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_c2h_data(queue, (void *)queue->pdu);
+		break;
 	case nvme_tcp_rsp:
-		nvme_tcp_init_recv_ctx(queue);
-		return nvme_tcp_handle_comp(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_comp(queue, (void *)queue->pdu);
+		break;
 	case nvme_tcp_r2t:
-		nvme_tcp_init_recv_ctx(queue);
-		return nvme_tcp_handle_r2t(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_r2t(queue, (void *)queue->pdu);
+		break;
 	default:
 		goto unsupported_pdu;
 	}
 
+	if (unlikely(ret == -EBUSY)) {
+		/*
+		 * The command is still being sent. Leave the end of the header
+		 * in the socket and look at it again when that is done.
+		 */
+		queue->pdu_remaining += rcv_len;
+		queue->pdu_offset -= rcv_len;
+		*offset -= rcv_len;
+		*len += rcv_len;
+	} else if (hdr->type != nvme_tcp_c2h_data) {
+		nvme_tcp_init_recv_ctx(queue);
+	}
+	return ret;
+
 unsupported_pdu:
 	dev_err(queue->ctrl->ctrl.device,
 		"unsupported pdu type (%d)\n", hdr->type);
@@ -1110,6 +1144,10 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
 		default:
 			result = -EFAULT;
 		}
+		if (unlikely(result == -EBUSY)) {
+			queue->rcv_parked = true;
+			return consumed - len;
+		}
 		if (result) {
 			dev_err(queue->ctrl->ctrl.device,
 				"receive failed:  %d\n", result);
@@ -1182,7 +1220,8 @@ static void nvme_tcp_state_change(struct sock *sk)
 
 static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue)
 {
-	queue->request = NULL;
+	/* pairs with nvme_tcp_req_busy() */
+	smp_store_release(&queue->request, NULL);
 }
 
 static void nvme_tcp_fail_request(struct nvme_tcp_request *req)
@@ -1430,6 +1469,7 @@ static int nvme_tcp_read_sock(struct nvme_tcp_queue *queue)
 	rd_desc.count = 1;
 	lock_sock(sk);
 	queue->nr_cqe = 0;
+	queue->rcv_parked = false;
 	consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
 	release_sock(sk);
 	return consumed;
@@ -1527,6 +1567,22 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
+/*
+ * A PDU has been left in the socket because its command is still being sent.
+ * A sender in another context does not wake us when it is done: wait for it.
+ * If there is something left to send then, either the socket is full and tells
+ * us when it has room, or io_work has been queued.
+ */
+static bool nvme_tcp_recv_again(struct nvme_tcp_queue *queue)
+{
+	bool again;
+
+	mutex_lock(&queue->send_mutex);
+	again = !queue->request && !nvme_tcp_queue_has_pending(queue);
+	mutex_unlock(&queue->send_mutex);
+	return again;
+}
+
 static void nvme_tcp_io_work(struct work_struct *w)
 {
 	struct nvme_tcp_queue *queue =
@@ -1552,6 +1608,9 @@ static void nvme_tcp_io_work(struct work_struct *w)
 		else if (unlikely(result < 0))
 			return;
 
+		if (unlikely(queue->rcv_parked) && nvme_tcp_recv_again(queue))
+			pending = true;
+
 		/* did we get some space after spending time in recv? */
 		if (nvme_tcp_queue_has_pending(queue) &&
 		    sk_stream_is_writeable(queue->sock->sk))
Signature · 1 line
2.56.0
author maintainer
#124 lore raw

PATCH 56/98 nvme-tcp: don't queue the AER while it is being sent

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Every AER completion runs nvme_async_event_work(), which has nvme_tcp_submit_async_event() set up and queue ctrl->async_req again. A controller that completes the AER twice can do that while the request is still on req_list or send_list, or being sent. Added to req_list twice the list becomes a ring and nvme_tcp_process_req_list() never ends:

  WARNING: lib/list_debug.c:35 at __list_add_valid_or_report+0xd8/0xf0
   nvme_tcp_try_send+0x51b/0x600

Leave the AER alone while the send path holds it, there's one on its way already. Initialize the list nodes at allocation so the check is valid from the start; nvme_tcp_fetch_request() leaves them that way.

Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +11 -4
index 06c205703cf5..be3938c3cb6f 100644
@@ -766,11 +766,14 @@ static int nvme_tcp_handle_comp(struct nvme_tcp_queue *queue,
 	 * for them but rather special case them here.
 	 */
 	if (unlikely(nvme_is_aen_req(nvme_tcp_queue_id(queue),
-				     cqe->command_id)))
+				     cqe->command_id))) {
+		if (nvme_tcp_req_busy(queue, &queue->ctrl->async_req))
+			return -EBUSY;
 		nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status,
 				&cqe->result);
-	else
+	} else {
 		ret = nvme_tcp_process_nvme_cqe(queue, cqe);
+	}
 
 	return ret;
 }
@@ -1646,6 +1649,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
 		return -ENOMEM;
 
 	async->queue = &ctrl->queues[0];
+	init_llist_node(&async->lentry);
+	INIT_LIST_HEAD(&async->entry);
 	return 0;
 }
 
@@ -2884,6 +2889,10 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
 	struct nvme_command *cmd = &pdu->cmd;
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 
+	/* A controller which completes it twice makes us come here twice */
+	if (nvme_tcp_req_busy(queue, &ctrl->async_req))
+		return;
+
 	memset(pdu, 0, sizeof(*pdu));
 	pdu->hdr.type = nvme_tcp_cmd;
 	if (queue->hdr_digest)
@@ -2900,8 +2909,6 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
 	ctrl->async_req.offset = 0;
 	ctrl->async_req.curr_bio = NULL;
 	ctrl->async_req.data_len = 0;
-	init_llist_node(&ctrl->async_req.lentry);
-	INIT_LIST_HEAD(&ctrl->async_req.entry);
 
 	nvme_tcp_queue_request(&ctrl->async_req, true);
 }
Signature · 1 line
2.56.0
author maintainer
#125 lore raw

PATCH 57/98 nvme-tcp: don't send SIGPIPE to whoever submits I/O

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvme_tcp_queue_request() tries to send inline, so the first PDU of a command goes out in the context of the submitting task, and none of the sends pass MSG_NOSIGNAL. Once a connection is down and error recovery hasn't stopped the queue yet, tcp_sendmsg() returns -EPIPE and sk_stream_error() sends SIGPIPE to current. A lost connection that the driver would recover from transparently instead kills whatever is doing I/O on the namespace: with 16 dd processes and a connection cut every other second, a few die within a minute.

Pass MSG_NOSIGNAL, like nbd, drbd, ceph, cifs and sunrpc do.

Fixes db5ad6b7f8cd ("nvme-tcp: try to send request in queue_rq context")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/nvme/host/tcp.c +10 -5
index be3938c3cb6f..c9531e653a2c 100644
@@ -1249,7 +1249,8 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 	while (true) {
 		struct bio_vec bvec;
 		struct msghdr msg = {
-			.msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES,
+			.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL |
+				     MSG_SPLICE_PAGES,
 		};
 		struct page *page = nvme_tcp_req_cur_page(req);
 		size_t offset = nvme_tcp_req_cur_offset(req);
@@ -1308,7 +1309,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	struct nvme_tcp_queue *queue = req->queue;
 	struct nvme_tcp_cmd_pdu *pdu = nvme_tcp_req_cmd_pdu(req);
 	struct bio_vec bvec;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, };
+	struct msghdr msg = {
+		.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SPLICE_PAGES,
+	};
 	bool inline_data = nvme_tcp_has_inline_data(req);
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) + hdgst - req->offset;
@@ -1349,7 +1352,9 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)
 	struct nvme_tcp_queue *queue = req->queue;
 	struct nvme_tcp_data_pdu *pdu = nvme_tcp_req_data_pdu(req);
 	struct bio_vec bvec;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, };
+	struct msghdr msg = {
+		.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL | MSG_MORE,
+	};
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) - req->offset + hdgst;
 	int ret;
@@ -1384,7 +1389,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 	size_t offset = req->offset;
 	u32 h2cdata_left = req->h2cdata_left;
 	int ret;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL };
 	struct kvec iov = {
 		.iov_base = (u8 *)&req->ddgst + req->offset,
 		.iov_len = NVME_TCP_DIGEST_LENGTH - req->offset
@@ -1696,7 +1701,7 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 	struct nvme_tcp_icresp_pdu *icresp;
 	char cbuf[CMSG_LEN(sizeof(char))] = {};
 	u8 ctype;
-	struct msghdr msg = {};
+	struct msghdr msg = { .msg_flags = MSG_NOSIGNAL };
 	struct kvec iov;
 	bool ctrl_hdgst, ctrl_ddgst;
 	u32 maxh2cdata;
Signature · 1 line
2.56.0
author maintainer
#126 lore raw

PATCH 58/98 nvme-fabrics: reconnect if the association is gone

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A reconnect connects the admin queue, gets a controller ID, and connects the I/O queues with it. If the controller goes away in between, the Connect of an I/O queue fails with Connect Invalid Parameters pointing at the cntlid, with DNR set:

nvme nvme0: Connect Invalid Data Parameter, cntlid: 1 nvme nvme0: Failed reconnect attempt 1/600 nvme nvme0: Removing controller (16770)...

nvmf_should_reconnect() honours DNR and the host deletes the controller with 599 attempts to go. DNR is right in that resending that command is pointless, but the next attempt would create a new association with a new cntlid. This is not a verdict on the host like a failed admin Connect is.

Treat it as a lost association and return -ENOTCONN, so the retries are bounded by ctrl_loss_tmo like after any other loss.

Fixes adfde7ed0b30 ("nvme-fabrics: short-circuit reconnect retries")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/fabrics.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/nvme/host/fabrics.c +13 -2
index 59f823dfbbcc..96d5ae6e1c39 100644
@@ -542,8 +542,19 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 			NVME_SUBMIT_RESERVED |
 			NVME_SUBMIT_NOWAIT);
 	if (ret) {
-		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
-				       &cmd, data);
+		result = le32_to_cpu(res.u32);
+		nvmf_log_connect_error(ctrl, ret, result, &cmd, data);
+		/*
+		 * The controller ID we got from the admin queue's Connect is
+		 * not known any more: the association has gone since. DNR is
+		 * right about this command and says nothing about the next
+		 * association. Don't let nvmf_should_reconnect() see it.
+		 */
+		if (ret > 0 &&
+		    (ret & NVME_SCT_SC_MASK) == NVME_SC_CONNECT_INVALID_PARAM &&
+		    result >> 16 && (result & 0xffff) ==
+				offsetof(struct nvmf_connect_data, cntlid))
+			ret = -ENOTCONN;
 		goto out_free_data;
 	}
 	result = le32_to_cpu(res.u32);
Signature · 1 line
2.56.0
author maintainer
#127 lore raw

PATCH 59/98 nvmet-auth: serialize the commands of a controller

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The host and controller keys, the DH context and the hash of DH-HMAC-CHAP live in the controller, and the authentication commands of all its queues use them without a lock. A host can send AUTH_Negotiate on the admin queue at any time, and nvmet_setup_auth() then frees and reloads the keys while another queue computes a response from them:

  BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250
   nvmet_auth_host_hash+0x10e/0x530
   nvmet_execute_auth_send+0xa4c/0x1a00
  Freed by task 194:
   nvmet_setup_auth+0x166/0x3a0
   nvmet_execute_auth_send+0x3e7/0x1a00

Neither queue needs to have authenticated for that.

Add a controller-wide mutex and hold it for the duration of an authentication command, dropping it before the request is completed. nvmet_alloc_ctrl() takes it around nvmet_setup_auth() too, as the controller is already on the subsystem list by then and an I/O queue can connect.

[axboe: upstream already has the per-queue auth_lock and expiry handling
from 53f6efe00cea, so only the controller lock is added here. It nests inside sq->auth_lock.]

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 drivers/nvme/target/core.c             | 4 ++++
 drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++
 drivers/nvme/target/nvmet.h            | 1 +
 3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +4 -0
index 8353950f346f..5bd78bb83cf9 100644
@@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 	if (!ctrl)
 		goto out_put_subsystem;
 	mutex_init(&ctrl->lock);
+	mutex_init(&ctrl->auth_lock);
 
 	ctrl->port = args->port;
 	ctrl->ops = args->ops;
@@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
 	nvmet_start_keep_alive_timer(ctrl);
 
+	/* An I/O queue may connect as soon as the controller is on the list. */
+	mutex_lock(&ctrl->auth_lock);
 	dhchap_status = nvmet_setup_auth(ctrl, args->sq, false);
+	mutex_unlock(&ctrl->auth_lock);
 	if (dhchap_status) {
 		pr_err("Failed to setup authentication, dhchap status %u\n",
 		       dhchap_status);
drivers/nvme/target/fabrics-cmd-auth.c +6 -0
index 1fe21f2672f1..9779e664b65f 100644
@@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	u8 dhchap_status;
 
 	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
 	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	}
 	if (unlikely(tl != req->transfer_len)) {
 		pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl);
+		mutex_unlock(&ctrl->auth_lock);
 		mutex_unlock(&req->sq->auth_lock);
 		nvmet_check_transfer_len(req, tl);
 		return;
@@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 		nvmet_ctrl_fatal_error(ctrl);
 
 complete:
+	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
 	nvmet_req_complete(req, status);
 }
@@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	u16 status = 0;
 
 	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
 	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	}
 	if (unlikely(al != req->transfer_len)) {
 		pr_debug("%s: transfer length mismatch (%u)\n", __func__, al);
+		mutex_unlock(&ctrl->auth_lock);
 		mutex_unlock(&req->sq->auth_lock);
 		nvmet_check_transfer_len(req, al);
 		return;
@@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 		nvmet_auth_sq_free(req->sq);
 		nvmet_ctrl_fatal_error(ctrl);
 	}
+	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
 	nvmet_req_complete(req, status);
 }
drivers/nvme/target/nvmet.h +1 -0
index 9cec19e6f485..f6a9f2e1c0ed 100644
@@ -307,6 +307,7 @@ struct nvmet_ctrl {
 	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
 	bool			pi_support;
 	bool			concat;
+	struct mutex		auth_lock; /* one AUTH command at a time */
 #ifdef CONFIG_NVME_TARGET_AUTH
 	struct nvme_dhchap_key	*host_key;
 	struct nvme_dhchap_key	*ctrl_key;
Signature · 1 line
2.56.0
maintainer
#128 lore raw

Re: [PATCH 59/98] nvmet-auth: serialize the commands of a controller

Quoted 120 lines · Jens Axboe

From: Josef Bacik <[email protected]>

The host and controller keys, the DH context and the hash of DH-HMAC-CHAP live in the controller, and the authentication commands of all its queues use them without a lock. A host can send AUTH_Negotiate on the admin queue at any time, and nvmet_setup_auth() then frees and reloads the keys while another queue computes a response from them:

   BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250
    nvmet_auth_host_hash+0x10e/0x530
    nvmet_execute_auth_send+0xa4c/0x1a00
   Freed by task 194:
    nvmet_setup_auth+0x166/0x3a0
    nvmet_execute_auth_send+0x3e7/0x1a00

Neither queue needs to have authenticated for that.

Add a controller-wide mutex and hold it for the duration of an authentication command, dropping it before the request is completed. nvmet_alloc_ctrl() takes it around nvmet_setup_auth() too, as the controller is already on the subsystem list by then and an I/O queue can connect.

[axboe: upstream already has the per-queue auth_lock and expiry handling
from 53f6efe00cea, so only the controller lock is added here. It nests inside sq->auth_lock.]

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
  drivers/nvme/target/core.c             | 4 ++++
  drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++
  drivers/nvme/target/nvmet.h            | 1 +
  3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +4 -0
index 8353950f346f..5bd78bb83cf9 100644
@@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
  	if (!ctrl)
  		goto out_put_subsystem;
  	mutex_init(&ctrl->lock);
+	mutex_init(&ctrl->auth_lock);
 
  	ctrl->port = args->port;
  	ctrl->ops = args->ops;
@@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
  	nvmet_start_keep_alive_timer(ctrl);
 
+	/* An I/O queue may connect as soon as the controller is on the list. */
+	mutex_lock(&ctrl->auth_lock);
  	dhchap_status = nvmet_setup_auth(ctrl, args->sq, false);
+	mutex_unlock(&ctrl->auth_lock);
  	if (dhchap_status) {
  		pr_err("Failed to setup authentication, dhchap status %u\n",
  		       dhchap_status);
drivers/nvme/target/fabrics-cmd-auth.c +6 -0
index 1fe21f2672f1..9779e664b65f 100644
@@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  	u8 dhchap_status;
 
  	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
  	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
  		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  	}
  	if (unlikely(tl != req->transfer_len)) {
  		pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl);
+		mutex_unlock(&ctrl->auth_lock);
  		mutex_unlock(&req->sq->auth_lock);
  		nvmet_check_transfer_len(req, tl);
  		return;
@@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  		nvmet_ctrl_fatal_error(ctrl);
 
  complete:
+	mutex_unlock(&ctrl->auth_lock);
  	mutex_unlock(&req->sq->auth_lock);
  	nvmet_req_complete(req, status);
  }
@@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
  	u16 status = 0;
 
  	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
  	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
  		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
  	}
  	if (unlikely(al != req->transfer_len)) {
  		pr_debug("%s: transfer length mismatch (%u)\n", __func__, al);
+		mutex_unlock(&ctrl->auth_lock);
  		mutex_unlock(&req->sq->auth_lock);
  		nvmet_check_transfer_len(req, al);
  		return;
@@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
  		nvmet_auth_sq_free(req->sq);
  		nvmet_ctrl_fatal_error(ctrl);
  	}
+	mutex_unlock(&ctrl->auth_lock);
  	mutex_unlock(&req->sq->auth_lock);
  	nvmet_req_complete(req, status);
  }
drivers/nvme/target/nvmet.h +0 -0
index 9cec19e6f485..f6a9f2e1c0ed 100644

drivers/nvme/target/nvmet.h struct nvmet_ctrl { · line 313

Quoted 7 lines
drivers/nvme/target/nvmet.h +1 -0
@@ -307,6 +307,7 @@ struct nvmet_ctrl {
  	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
  	bool			pi_support;
  	bool			concat;
+	struct mutex		auth_lock; /* one AUTH command at a time */
  #ifdef CONFIG_NVME_TARGET_AUTH
  	struct nvme_dhchap_key	*host_key;
  	struct nvme_dhchap_key	*ctrl_key;

Hmm. Not particularly happy with this.
In the end, it renders sq->auth_lock pretty much pointless (the only
place where it remains on its own is in nvmet_auth_expiry_work() ).
And it would trigger if a authentication reset while another authentication (eg on the I/O queues) is still ongoing. With this patch we would switch the authentication variables
in the middle of the authentication protocol run for the I/O
queue, most likely causing that authentication to fail.

We should rather fail ongoing authentications whenever we call nvmet_auth_setup().

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#129 lore raw

PATCH 60/98 nvmet-auth: keep the negotiated hash per queue

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Every queue runs its own DH-HMAC-CHAP transaction, but the hash that AUTH_Negotiate settles on is stored in the controller. A host that doesn't offer the configured hash gets a fallback, and that changes ctrl->shash_id under the transactions of the other queues, whose challenges have the length of the old hash:

qid 1: AUTH_Negotiate (SHA-256), 32 byte challenge qid 0: AUTH_Negotiate (SHA-512 only)
qid 1: DH-HMAC-CHAP_Reply with a hash length of 64

  BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530

No secret is needed for that.

Store the negotiated hash in the queue and use it for the rest of the transaction. ctrl->shash_id stays the configured preference.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/auth.c             | 35 +++++++++++++-------------
 drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++-----
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 25 insertions(+), 23 deletions(-)
drivers/nvme/target/auth.c +18 -17
index 1680e5e6839c..0f64fd4ade59 100644
@@ -299,14 +299,14 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 	if (IS_ERR(transformed_key))
 		return PTR_ERR(transformed_key);
 
-	ret = nvme_auth_hmac_init(&hmac, ctrl->shash_id, transformed_key->key,
-				  transformed_key->len);
+	ret = nvme_auth_hmac_init(&hmac, req->sq->shash_id,
+				  transformed_key->key, transformed_key->len);
 	if (ret)
 		goto out_free_response;
 
-	if (shash_len != nvme_auth_hmac_hash_len(ctrl->shash_id)) {
+	if (shash_len != nvme_auth_hmac_hash_len(req->sq->shash_id)) {
 		pr_err("%s: hash len mismatch (len %u digest %zu)\n", __func__,
-		       shash_len, nvme_auth_hmac_hash_len(ctrl->shash_id));
+		       shash_len, nvme_auth_hmac_hash_len(req->sq->shash_id));
 		ret = -EINVAL;
 		goto out_free_response;
 	}
@@ -317,7 +317,7 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			ret = -ENOMEM;
 			goto out_free_response;
 		}
-		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+		ret = nvme_auth_augmented_challenge(req->sq->shash_id,
 						    req->sq->dhchap_skey,
 						    req->sq->dhchap_skey_len,
 						    req->sq->dhchap_c1,
@@ -371,14 +371,14 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 	if (IS_ERR(transformed_key))
 		return PTR_ERR(transformed_key);
 
-	ret = nvme_auth_hmac_init(&hmac, ctrl->shash_id, transformed_key->key,
-				  transformed_key->len);
+	ret = nvme_auth_hmac_init(&hmac, req->sq->shash_id,
+				  transformed_key->key, transformed_key->len);
 	if (ret)
 		goto out_free_response;
 
-	if (shash_len != nvme_auth_hmac_hash_len(ctrl->shash_id)) {
+	if (shash_len != nvme_auth_hmac_hash_len(req->sq->shash_id)) {
 		pr_err("%s: hash len mismatch (len %u digest %zu)\n", __func__,
-		       shash_len, nvme_auth_hmac_hash_len(ctrl->shash_id));
+		       shash_len, nvme_auth_hmac_hash_len(req->sq->shash_id));
 		ret = -EINVAL;
 		goto out_free_response;
 	}
@@ -389,7 +389,7 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 			ret = -ENOMEM;
 			goto out_free_response;
 		}
-		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+		ret = nvme_auth_augmented_challenge(req->sq->shash_id,
 						    req->sq->dhchap_skey,
 						    req->sq->dhchap_skey_len,
 						    req->sq->dhchap_c2,
@@ -456,7 +456,8 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	int ret;
 
-	req->sq->dhchap_skey_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	req->sq->dhchap_skey_len =
+		nvme_auth_hmac_hash_len(req->sq->shash_id);
 	req->sq->dhchap_skey = kzalloc(req->sq->dhchap_skey_len, GFP_KERNEL);
 	if (!req->sq->dhchap_skey)
 		return -ENOMEM;
@@ -464,7 +465,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 					pkey, pkey_size,
 					req->sq->dhchap_skey,
 					req->sq->dhchap_skey_len,
-					ctrl->shash_id);
+					req->sq->shash_id);
 	if (ret)
 		pr_debug("failed to compute session key, err %d\n", ret);
 #ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
@@ -478,7 +479,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 
 void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 {
-	int hash_len = nvme_auth_hmac_hash_len(sq->ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(sq->shash_id);
 	u8 *psk, *tls_psk;
 	char *digest;
 	size_t psk_len;
@@ -487,7 +488,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 	struct key *tls_key = NULL;
 #endif
 
-	ret = nvme_auth_generate_psk(sq->ctrl->shash_id,
+	ret = nvme_auth_generate_psk(sq->shash_id,
 				     sq->dhchap_skey,
 				     sq->dhchap_skey_len,
 				     sq->dhchap_c1, sq->dhchap_c2,
@@ -497,7 +498,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 			__func__, sq->ctrl->cntlid, sq->qid, ret);
 		return;
 	}
-	ret = nvme_auth_generate_digest(sq->ctrl->shash_id, psk, psk_len,
+	ret = nvme_auth_generate_digest(sq->shash_id, psk, psk_len,
 					sq->ctrl->subsys->subsysnqn,
 					sq->ctrl->hostnqn, &digest);
 	if (ret) {
@@ -505,7 +506,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 			__func__, sq->ctrl->cntlid, sq->qid, ret);
 		goto out_free_psk;
 	}
-	ret = nvme_auth_derive_tls_psk(sq->ctrl->shash_id, psk, psk_len,
+	ret = nvme_auth_derive_tls_psk(sq->shash_id, psk, psk_len,
 				       digest, &tls_psk);
 	if (ret) {
 		pr_warn("%s: ctrl %d qid %d failed to derive TLS PSK, error %d\n",
@@ -515,7 +516,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 #ifdef CONFIG_NVME_TARGET_TCP_TLS
 	tls_key = nvme_tls_psk_refresh(NULL, sq->ctrl->hostnqn,
 				       sq->ctrl->subsys->subsysnqn,
-				       sq->ctrl->shash_id, tls_psk, psk_len,
+				       sq->shash_id, tls_psk, psk_len,
 				       digest);
 	if (IS_ERR(tls_key)) {
 		pr_warn("%s: ctrl %d qid %d failed to refresh key, error %ld\n",
drivers/nvme/target/fabrics-cmd-auth.c +6 -6
index 9779e664b65f..841fd2683702 100644
@@ -111,8 +111,9 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 		pr_debug("%s: ctrl %d qid %d: no usable hash found, falling back to %s\n",
 			 __func__, ctrl->cntlid, req->sq->qid,
 			 nvme_auth_hmac_name(fallback_hash_id));
-		ctrl->shash_id = fallback_hash_id;
+		hash_id = fallback_hash_id;
 	}
+	req->sq->shash_id = hash_id;
 
 	dhgid = -1;
 	fallback_dhgid = -1;
@@ -454,7 +455,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 	struct nvmf_auth_dhchap_challenge_data *data = d;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	int ret = 0;
-	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(req->sq->shash_id);
 	int data_size = sizeof(*data) + hash_len;
 
 	if (ctrl->dh_tfm)
@@ -469,7 +470,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
 	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
 	data->t_id = cpu_to_le16(req->sq->dhchap_tid);
-	data->hashid = ctrl->shash_id;
+	data->hashid = req->sq->shash_id;
 	data->hl = hash_len;
 	data->seqnum = cpu_to_le32(req->sq->dhchap_s1);
 	req->sq->dhchap_c1 = kmalloc(data->hl, GFP_KERNEL);
@@ -493,7 +494,7 @@ static int nvmet_auth_success1(struct nvmet_req *req, void *d, int al)
 {
 	struct nvmf_auth_dhchap_success1_data *data = d;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
-	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(req->sq->shash_id);
 
 	WARN_ON(al < sizeof(*data));
 	memset(data, 0, sizeof(*data));
@@ -530,7 +531,6 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al)
 
 u32 nvmet_auth_receive_data_len(struct nvmet_req *req)
 {
-	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	u32 al = le32_to_cpu(req->cmd->auth_receive.al);
 	u32 min_len;
 
@@ -544,7 +544,7 @@ u32 nvmet_auth_receive_data_len(struct nvmet_req *req)
 	case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1:
 		min_len = sizeof(struct nvmf_auth_dhchap_success1_data);
 		if (req->sq->dhchap_c2)
-			min_len += nvme_auth_hmac_hash_len(ctrl->shash_id);
+			min_len += nvme_auth_hmac_hash_len(req->sq->shash_id);
 		break;
 	default:
 		min_len = sizeof(struct nvmf_auth_dhchap_failure_data);
drivers/nvme/target/nvmet.h +1 -0
index f6a9f2e1c0ed..46501d0757c0 100644
@@ -167,6 +167,7 @@ struct nvmet_sq {
 	bool			auth_expiry_active;
 	u16			dhchap_tid;
 	u8			sc_c;
+	u8			shash_id;
 	u8			dhchap_status;
 	u8			dhchap_step;
 	u8			*dhchap_c1;
Signature · 1 line
2.56.0
maintainer
#130 lore raw

Re: [PATCH 60/98] nvmet-auth: keep the negotiated hash per queue

Quoted 26 lines · Jens Axboe

From: Josef Bacik <[email protected]>

Every queue runs its own DH-HMAC-CHAP transaction, but the hash that AUTH_Negotiate settles on is stored in the controller. A host that doesn't offer the configured hash gets a fallback, and that changes ctrl->shash_id under the transactions of the other queues, whose challenges have the length of the old hash:

qid 1: AUTH_Negotiate (SHA-256), 32 byte challenge qid 0: AUTH_Negotiate (SHA-512 only)
qid 1: DH-HMAC-CHAP_Reply with a hash length of 64

   BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530

No secret is needed for that.

Store the negotiated hash in the queue and use it for the rest of the transaction. ctrl->shash_id stays the configured preference.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 6 lines
---
  drivers/nvme/target/auth.c             | 35 +++++++++++++-------------
  drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++-----
  drivers/nvme/target/nvmet.h            |  1 +
  3 files changed, 25 insertions(+), 23 deletions(-)

Weird. But spec seems to be okay with that, I guess. But then we don't really need to the shash_id from the controller, so it should be removed from there.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#131 lore raw

PATCH 61/98 nvmet-auth: authenticate the I/O queues again

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Since commit c91a20129185 the target neither asks a host to authenticate an I/O queue nor checks that it did: nvmet_check_auth_status() is true for everything but the admin queue. Nothing else ties an I/O queue to the admin queue's authentication. The Connect of an I/O queue names its controller by subsystem NQN, host NQN and cntlid, so anyone who can reach the port and knows the NQNs can attach a queue to an authenticated controller and do I/O. Property Set isn't subject to authentication either, so an unauthenticated host can enable its own controller and connect an I/O queue to that.

Ask for authentication in the Connect response of an I/O queue and refuse its commands until it has passed, as before that commit. The Linux host authenticates every queue that is asked to. A queue with a TLS session is exempt, its peer held a PSK the port accepts, and since commit ecf4d2d88351 such a controller does get a host key when the admin queue re-authenticates.

Fixes c91a20129185 ("nvmet-auth: authenticate on admin queue only")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/auth.c        | 14 +++++++-------
 drivers/nvme/target/fabrics-cmd.c |  4 ----
 2 files changed, 7 insertions(+), 11 deletions(-)
drivers/nvme/target/auth.c +7 -7
index 0f64fd4ade59..9ae8293cc734 100644
@@ -275,13 +275,13 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
 
 bool nvmet_check_auth_status(struct nvmet_req *req)
 {
-	if (req->sq->ctrl->host_key) {
-		if (req->sq->qid > 0)
-			return true;
-		if (!req->sq->authenticated)
-			return false;
-	}
-	return true;
+	struct nvmet_sq *sq = req->sq;
+
+	if (!sq->ctrl->host_key || sq->authenticated)
+		return true;
+
+	/* Whoever set up the TLS session of an I/O queue knew the PSK. */
+	return sq->qid && nvmet_queue_tls_keyid(sq);
 }
 
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
drivers/nvme/target/fabrics-cmd.c +0 -4
index b451029ab8f4..38311187d27a 100644
@@ -251,10 +251,6 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq)
 	bool needs_auth = nvmet_has_auth(ctrl, sq);
 	key_serial_t keyid = nvmet_queue_tls_keyid(sq);
 
-	/* Do not authenticate I/O queues */
-	if (sq->qid)
-		needs_auth = false;
-
 	if (keyid)
 		pr_debug("%s: ctrl %d qid %d should %sauthenticate, tls psk %08x\n",
 			 __func__, ctrl->cntlid, sq->qid,
Signature · 1 line
2.56.0
author maintainer
#132 lore raw

PATCH 62/98 nvmet: don't mix queues with and without TLS

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

Since commit fa2e0f8bbc68 a controller whose admin queue uses TLS gets no host key, the PSK stands in for DH-HMAC-CHAP. But a port with TREQ "not required" takes plain connections too, as does any other port the subsystem is linked to, and the Connect of an I/O queue names its controller by NQNs and cntlid only. So anyone who reaches such a port can add a plain queue to a TLS controller, and with no host key nothing is asked of it. The other way round, a TLS queue skips DH-HMAC-CHAP on a controller whose other queues had to pass it.

A host uses TLS for all queues or none. Refuse an I/O queue Connect that differs from the admin queue in that.

Fixes fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/core.c        | 1 +
 drivers/nvme/target/fabrics-cmd.c | 9 +++++++++
 drivers/nvme/target/nvmet.h       | 1 +
 3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +1 -0
index 5bd78bb83cf9..333739f14c2b 100644
@@ -1646,6 +1646,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
 	ctrl->port = args->port;
 	ctrl->ops = args->ops;
+	ctrl->tls = args->sq && nvmet_queue_tls_keyid(args->sq);
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	/* By default, set loop targets to clear IDS by default */
drivers/nvme/target/fabrics-cmd.c +9 -0
index 38311187d27a..ddb732f62981 100644
@@ -377,6 +377,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 		goto out_ctrl_put;
 	}
 
+	/* All queues of a controller use TLS, or none of them does. */
+	if (ctrl->tls != !!nvmet_queue_tls_keyid(req->sq)) {
+		pr_warn("ctrl %d: queue %d with%s TLS\n", ctrl->cntlid, qid,
+			ctrl->tls ? "out" : "");
+		status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
+		req->cqe->result.u32 = IPO_IATTR_CONNECT_DATA(cntlid);
+		goto out_ctrl_put;
+	}
+
 	status = nvmet_install_queue(ctrl, req);
 	if (status)
 		goto out_ctrl_put;
drivers/nvme/target/nvmet.h +1 -0
index 46501d0757c0..cc7fface77de 100644
@@ -308,6 +308,7 @@ struct nvmet_ctrl {
 	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
 	bool			pi_support;
 	bool			concat;
+	bool			tls;
 	struct mutex		auth_lock; /* one AUTH command at a time */
 #ifdef CONFIG_NVME_TARGET_AUTH
 	struct nvme_dhchap_key	*host_key;
Signature · 1 line
2.56.0
maintainer
#133 lore raw

Re: [PATCH 62/98] nvmet: don't mix queues with and without TLS

Quoted 63 lines · Jens Axboe

From: Josef Bacik <[email protected]>

Since commit fa2e0f8bbc68 a controller whose admin queue uses TLS gets no host key, the PSK stands in for DH-HMAC-CHAP. But a port with TREQ "not required" takes plain connections too, as does any other port the subsystem is linked to, and the Connect of an I/O queue names its controller by NQNs and cntlid only. So anyone who reaches such a port can add a plain queue to a TLS controller, and with no host key nothing is asked of it. The other way round, a TLS queue skips DH-HMAC-CHAP on a controller whose other queues had to pass it.

A host uses TLS for all queues or none. Refuse an I/O queue Connect that differs from the admin queue in that.

Fixes fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/core.c        | 1 +
  drivers/nvme/target/fabrics-cmd.c | 9 +++++++++
  drivers/nvme/target/nvmet.h       | 1 +
  3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +1 -0
index 5bd78bb83cf9..333739f14c2b 100644
@@ -1646,6 +1646,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
  	ctrl->port = args->port;
  	ctrl->ops = args->ops;
+	ctrl->tls = args->sq && nvmet_queue_tls_keyid(args->sq);
 
  #ifdef CONFIG_NVME_TARGET_PASSTHRU
  	/* By default, set loop targets to clear IDS by default */
drivers/nvme/target/fabrics-cmd.c +9 -0
index 38311187d27a..ddb732f62981 100644
@@ -377,6 +377,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
  		goto out_ctrl_put;
  	}
 
+	/* All queues of a controller use TLS, or none of them does. */
+	if (ctrl->tls != !!nvmet_queue_tls_keyid(req->sq)) {
+		pr_warn("ctrl %d: queue %d with%s TLS\n", ctrl->cntlid, qid,
+			ctrl->tls ? "out" : "");
+		status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
+		req->cqe->result.u32 = IPO_IATTR_CONNECT_DATA(cntlid);
+		goto out_ctrl_put;
+	}
+
  	status = nvmet_install_queue(ctrl, req);
  	if (status)
  		goto out_ctrl_put;
drivers/nvme/target/nvmet.h +0 -0
index 46501d0757c0..cc7fface77de 100644

drivers/nvme/target/nvmet.h struct nvmet_ctrl { · line 314

Quoted 7 lines
drivers/nvme/target/nvmet.h +1 -0
@@ -308,6 +308,7 @@ struct nvmet_ctrl {
  	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
  	bool			pi_support;
  	bool			concat;
+	bool			tls;
  	struct mutex		auth_lock; /* one AUTH command at a time */
  #ifdef CONFIG_NVME_TARGET_AUTH
  	struct nvme_dhchap_key	*host_key;

Hmm. Technically TLS is a feature of the TCP transport and only applicable there. We should delegate this step to the TCP transport.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#134 lore raw

PATCH 63/98 nvmet-auth: refuse a reply without a DH value

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

With a DH group configured, nvmet_auth_host_hash() augments the challenge with the queue's session key, but nvmet_auth_reply() only computes that key if the DH-HMAC-CHAP_Reply carries a DH value. Without one, whatever the queue holds is used: nothing on the first transaction, so the DH exchange is skipped, and after nvmet_auth_sq_free() a NULL pointer with the hash length still set:

  BUG: KASAN: null-ptr-deref in __hmac_sha256_preparekey+0xa5/0x250
   nvme_auth_hmac+0xb7/0x160
   nvmet_auth_host_hash+0x2a5/0x530

Refuse such a reply, as the host refuses a challenge without a DH value, and clear the length along with the pointer.

Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/auth.c             | 1 +
 drivers/nvme/target/fabrics-cmd-auth.c | 3 +++
 2 files changed, 4 insertions(+)
drivers/nvme/target/auth.c +1 -0
index 9ae8293cc734..d1af4a921e7a 100644
@@ -237,6 +237,7 @@ void nvmet_auth_sq_free(struct nvmet_sq *sq)
 	sq->dhchap_c2 = NULL;
 	kfree(sq->dhchap_skey);
 	sq->dhchap_skey = NULL;
+	sq->dhchap_skey_len = 0;
 }
 
 void nvmet_auth_sq_destroy(struct nvmet_sq *sq)
drivers/nvme/target/fabrics-cmd-auth.c +3 -0
index 841fd2683702..a559fec3aa65 100644
@@ -180,6 +180,9 @@ static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)
 		if (nvmet_auth_ctrl_sesskey(req, data->rval + 2 * data->hl,
 					    dhvlen) < 0)
 			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+	} else if (ctrl->dh_gid != NVME_AUTH_DHGROUP_NULL) {
+		/* No DH value, no session key for the augmented challenge. */
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
 	}
 
 	response = kmalloc(data->hl, GFP_KERNEL);
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 63/98
#135 lore raw

Re: [PATCH 63/98] nvmet-auth: refuse a reply without a DH value

Quoted 23 lines · Jens Axboe

From: Josef Bacik <[email protected]>

With a DH group configured, nvmet_auth_host_hash() augments the challenge with the queue's session key, but nvmet_auth_reply() only computes that key if the DH-HMAC-CHAP_Reply carries a DH value. Without one, whatever the queue holds is used: nothing on the first transaction, so the DH exchange is skipped, and after nvmet_auth_sq_free() a NULL pointer with the hash length still set:

   BUG: KASAN: null-ptr-deref in __hmac_sha256_preparekey+0xa5/0x250
    nvme_auth_hmac+0xb7/0x160
    nvmet_auth_host_hash+0x2a5/0x530

Refuse such a reply, as the host refuses a challenge without a DH value, and clear the length along with the pointer.

Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 5 lines
---
  drivers/nvme/target/auth.c             | 1 +
  drivers/nvme/target/fabrics-cmd-auth.c | 3 +++
  2 files changed, 4 insertions(+)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#136 lore raw

PATCH 64/98 nvmet-auth: keep the old keys if the new ones fail

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

AUTH_Negotiate on the admin queue has nvmet_setup_auth() reload the host's keys, and it frees the old host key first. If decoding the new secret fails, or an allocation does, the controller is left without a host key, which means no authentication at all: nvmet_check_auth_status() passes everything and I/O queue Connects no longer ask for it. configfs accepts any string starting with "DHHC-1:" and a digit, so one typo while changing a secret opens the namespaces to any connected admin queue.

Decode the new keys first and only then replace the old ones.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/auth.c | 69 ++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 41 deletions(-)
drivers/nvme/target/auth.c +28 -41
index d1af4a921e7a..5809ce32a2e4 100644
@@ -135,6 +135,7 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset)
 	int ret = 0;
 	struct nvmet_host_link *p;
 	struct nvmet_host *host = NULL;
+	struct nvme_dhchap_key *host_key, *ctrl_key = NULL;
 
 	down_read(&nvmet_config_sem);
 	if (nvmet_is_disc_subsys(ctrl->subsys))
@@ -172,55 +173,41 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset)
 		goto out_unlock;
 	}
 
-	if (host->dhchap_hash_id == ctrl->shash_id) {
-		pr_debug("Re-use existing hash ID %d\n",
-			 ctrl->shash_id);
-	} else {
-		ctrl->shash_id = host->dhchap_hash_id;
-	}
-
-	/* Skip the 'DHHC-1:XX:' prefix */
-	nvme_auth_free_key(ctrl->host_key);
-	ctrl->host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
-					       host->dhchap_key_hash);
-	if (IS_ERR(ctrl->host_key)) {
+	/*
+	 * Skip the 'DHHC-1:XX:' prefix. Keep the old keys until the new ones
+	 * are there: a controller without a host key needs no authentication.
+	 */
+	host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
+					 host->dhchap_key_hash);
+	if (IS_ERR(host_key)) {
 		ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
-		ctrl->host_key = NULL;
-		goto out_free_hash;
+		goto out_unlock;
 	}
+	if (host->dhchap_ctrl_secret) {
+		ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
+						 host->dhchap_ctrl_key_hash);
+		if (IS_ERR(ctrl_key)) {
+			nvme_auth_free_key(host_key);
+			ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+			goto out_unlock;
+		}
+	}
+	nvme_auth_free_key(ctrl->host_key);
+	ctrl->host_key = host_key;
+	nvme_auth_free_key(ctrl->ctrl_key);
+	ctrl->ctrl_key = ctrl_key;
+	ctrl->shash_id = host->dhchap_hash_id;
 #ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
 	pr_debug("%s: using hash %s key %*ph\n", __func__,
 		 ctrl->host_key->hash > 0 ?
 		 nvme_auth_hmac_name(ctrl->host_key->hash) : "none",
 		 (int)ctrl->host_key->len, ctrl->host_key->key);
+	if (ctrl->ctrl_key)
+		pr_debug("%s: using ctrl hash %s key %*ph\n", __func__,
+			 ctrl->ctrl_key->hash > 0 ?
+			 nvme_auth_hmac_name(ctrl->ctrl_key->hash) : "none",
+			 (int)ctrl->ctrl_key->len, ctrl->ctrl_key->key);
 #endif
-	nvme_auth_free_key(ctrl->ctrl_key);
-	if (!host->dhchap_ctrl_secret) {
-		ctrl->ctrl_key = NULL;
-		goto out_unlock;
-	}
-
-	ctrl->ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
-					       host->dhchap_ctrl_key_hash);
-	if (IS_ERR(ctrl->ctrl_key)) {
-		ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
-		ctrl->ctrl_key = NULL;
-		goto out_free_hash;
-	}
-#ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
-	pr_debug("%s: using ctrl hash %s key %*ph\n", __func__,
-		 ctrl->ctrl_key->hash > 0 ?
-		 nvme_auth_hmac_name(ctrl->ctrl_key->hash) : "none",
-		 (int)ctrl->ctrl_key->len, ctrl->ctrl_key->key);
-#endif
-out_free_hash:
-	if (ret) {
-		if (ctrl->host_key) {
-			nvme_auth_free_key(ctrl->host_key);
-			ctrl->host_key = NULL;
-		}
-		ctrl->shash_id = 0;
-	}
 out_unlock:
 	up_read(&nvmet_config_sem);
 
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 64/98
#137 lore raw

Re: [PATCH 64/98] nvmet-auth: keep the old keys if the new ones fail

Quoted 20 lines · Jens Axboe

From: Josef Bacik <[email protected]>

AUTH_Negotiate on the admin queue has nvmet_setup_auth() reload the host's keys, and it frees the old host key first. If decoding the new secret fails, or an allocation does, the controller is left without a host key, which means no authentication at all: nvmet_check_auth_status() passes everything and I/O queue Connects no longer ask for it. configfs accepts any string starting with "DHHC-1:" and a digit, so one typo while changing a secret opens the namespaces to any connected admin queue.

Decode the new keys first and only then replace the old ones.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 4 lines
---
  drivers/nvme/target/auth.c | 69 ++++++++++++++++----------------------
  1 file changed, 28 insertions(+), 41 deletions(-)

That discussion again. I give in.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#138 lore raw

PATCH 65/98 nvmet-auth: take only AUTH_Negotiate for a negotiation

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_execute_auth_send() hands a common-type message to nvmet_auth_negotiate() if its ID is AUTH_Negotiate or matches the step the queue is in. The second case is wrong: the only other common-type message a host sends is AUTH_Failure2, and no queue waits in that step. A host can name the step on purpose though, and its message is then treated as a new AUTH_Negotiate without the work that goes with a new transaction, like reloading the keys on the admin queue.

Fail every other common-type message, as AUTH_Failure2 already is.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/fabrics-cmd-auth.c +2 -1
index a559fec3aa65..5ab943256f38 100644
@@ -350,8 +350,9 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 			}
 			req->sq->dhchap_step =
 				NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
-		} else if (data->auth_id != req->sq->dhchap_step)
+		} else {
 			goto done_failure1;
+		}
 		/* Validate negotiation parameters */
 		dhchap_status = nvmet_auth_negotiate(req, d, tl);
 		if (dhchap_status == 0)
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 65/98
#139 lore raw

Re: [PATCH 65/98] nvmet-auth: take only AUTH_Negotiate for a negotiation

Quoted 28 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_execute_auth_send() hands a common-type message to nvmet_auth_negotiate() if its ID is AUTH_Negotiate or matches the step the queue is in. The second case is wrong: the only other common-type message a host sends is AUTH_Failure2, and no queue waits in that step. A host can name the step on purpose though, and its message is then treated as a new AUTH_Negotiate without the work that goes with a new transaction, like reloading the keys on the admin queue.

Fail every other common-type message, as AUTH_Failure2 already is.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/fabrics-cmd-auth.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/fabrics-cmd-auth.c +0 -0
index a559fec3aa65..5ab943256f38 100644

drivers/nvme/target/fabrics-cmd-auth.c void nvmet_execute_auth_send(struct nvmet_req *req) · line 358

Quoted 10 lines
drivers/nvme/target/fabrics-cmd-auth.c +2 -1
@@ -350,8 +350,9 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  			}
  			req->sq->dhchap_step =
  				NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
-		} else if (data->auth_id != req->sq->dhchap_step)
+		} else {
  			goto done_failure1;
+		}
  		/* Validate negotiation parameters */
  		dhchap_status = nvmet_auth_negotiate(req, d, tl);
  		if (dhchap_status == 0)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#140 lore raw

PATCH 66/98 nvmet-auth: free the old data on a new AUTH_Negotiate

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A host may restart the negotiation at any step. The challenge and the session key of the abandoned transaction stay in the queue, and nvmet_auth_challenge() and nvmet_auth_ctrl_sesskey() overwrite the pointers with new allocations, leaking one or two hash-sized buffers per round for anyone who can reach the port. The controller challenge stays too, and if the new transaction brings none but asks for the controller's response, the old one is hashed with the new hash length:

  BUG: KASAN: slab-out-of-bounds in nvmet_auth_ctrl_hash+0x2d5/0x4b0

Free the old transaction's data when a new one begins.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/target/fabrics-cmd-auth.c +1 -0
index 5ab943256f38..665c56d29d07 100644
@@ -336,6 +336,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 			/* Restart negotiation */
 			pr_debug("%s: ctrl %d qid %d reset negotiation\n",
 				 __func__, ctrl->cntlid, req->sq->qid);
+			nvmet_auth_sq_free(req->sq);
 			if (!req->sq->qid) {
 				dhchap_status = nvmet_setup_auth(ctrl, req->sq,
 								 true);
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 66/98
#141 lore raw

Re: [PATCH 66/98] nvmet-auth: free the old data on a new AUTH_Negotiate

Quoted 30 lines · Jens Axboe

From: Josef Bacik <[email protected]>

A host may restart the negotiation at any step. The challenge and the session key of the abandoned transaction stay in the queue, and nvmet_auth_challenge() and nvmet_auth_ctrl_sesskey() overwrite the pointers with new allocations, leaking one or two hash-sized buffers per round for anyone who can reach the port. The controller challenge stays too, and if the new transaction brings none but asks for the controller's response, the old one is hashed with the new hash length:

   BUG: KASAN: slab-out-of-bounds in nvmet_auth_ctrl_hash+0x2d5/0x4b0

Free the old transaction's data when a new one begins.

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/fabrics-cmd-auth.c | 1 +
  1 file changed, 1 insertion(+)
drivers/nvme/target/fabrics-cmd-auth.c +0 -0
index 5ab943256f38..665c56d29d07 100644

drivers/nvme/target/fabrics-cmd-auth.c void nvmet_execute_auth_send(struct nvmet_req *req) · line 342

Quoted 7 lines
drivers/nvme/target/fabrics-cmd-auth.c +1 -0
@@ -336,6 +336,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  			/* Restart negotiation */
  			pr_debug("%s: ctrl %d qid %d reset negotiation\n",
  				 __func__, ctrl->cntlid, req->sq->qid);
+			nvmet_auth_sq_free(req->sq);
  			if (!req->sq->qid) {
  				dhchap_status = nvmet_setup_auth(ctrl, req->sq,
  								 true);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#142 lore raw

PATCH 67/98 nvmet-auth: fix the check of the offered DH groups

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_auth_negotiate() matches the first DH group in the host's list that is NOT the configured one, and looks for a fallback among those that are. So a host offering only groups the target isn't configured for passes and gets a challenge for a group it never offered, while a host offering only the NULL group is refused by a target configured for NULL, since there's no KPP algorithm to fall back to. A fallback can't work anyway, the DH context and public key belong to the configured group.

Accept if the configured group is in the list, fail with "DH group unusable" otherwise.

Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 27 +++++---------------------
 1 file changed, 5 insertions(+), 22 deletions(-)
drivers/nvme/target/fabrics-cmd-auth.c +5 -22
index 665c56d29d07..8c48140eb542 100644
@@ -47,7 +47,7 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmf_auth_dhchap_negotiate_data *data = d;
-	int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid;
+	int i, hash_id = 0, fallback_hash_id = 0;
 
 	if (tl < sizeof(*data) +
 			sizeof(struct nvmf_auth_dhchap_protocol_descriptor))
@@ -115,32 +115,15 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 	}
 	req->sq->shash_id = hash_id;
 
-	dhgid = -1;
-	fallback_dhgid = -1;
 	for (i = 0; i < data->auth_protocol[0].dhchap.dhlen; i++) {
-		int tmp_dhgid = data->auth_protocol[0].dhchap.idlist[i + 30];
-
-		if (tmp_dhgid != ctrl->dh_gid) {
-			dhgid = tmp_dhgid;
+		if (data->auth_protocol[0].dhchap.idlist[i + 30] == ctrl->dh_gid)
 			break;
-		}
-		if (fallback_dhgid < 0) {
-			const char *kpp = nvme_auth_dhgroup_kpp(tmp_dhgid);
-
-			if (crypto_has_kpp(kpp, 0, 0))
-				fallback_dhgid = tmp_dhgid;
-		}
 	}
-	if (dhgid < 0) {
-		if (fallback_dhgid < 0) {
-			pr_debug("%s: ctrl %d qid %d: no usable DH group found\n",
-				 __func__, ctrl->cntlid, req->sq->qid);
-			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
-		}
+	if (i == data->auth_protocol[0].dhchap.dhlen) {
 		pr_debug("%s: ctrl %d qid %d: configured DH group %s not found\n",
 			 __func__, ctrl->cntlid, req->sq->qid,
-			 nvme_auth_dhgroup_name(fallback_dhgid));
-		ctrl->dh_gid = fallback_dhgid;
+			 nvme_auth_dhgroup_name(ctrl->dh_gid));
+		return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
 	}
 	if (ctrl->dh_gid == NVME_AUTH_DHGROUP_NULL && ctrl->concat) {
 		pr_debug("%s: ctrl %d qid %d: NULL DH group invalid "
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 67/98
#143 lore raw

Re: [PATCH 67/98] nvmet-auth: fix the check of the offered DH groups

Quoted 21 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_auth_negotiate() matches the first DH group in the host's list that is NOT the configured one, and looks for a fallback among those that are. So a host offering only groups the target isn't configured for passes and gets a challenge for a group it never offered, while a host offering only the NULL group is refused by a target configured for NULL, since there's no KPP algorithm to fall back to. A fallback can't work anyway, the DH context and public key belong to the configured group.

Accept if the configured group is in the list, fail with "DH group unusable" otherwise.

Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 4 lines
---
  drivers/nvme/target/fabrics-cmd-auth.c | 27 +++++---------------------
  1 file changed, 5 insertions(+), 22 deletions(-)

Yeah, the fallback idea was not great.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#144 lore raw

PATCH 68/98 nvmet: accept I/O queues on the controller's port only

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_ctrl_find_get() looks the subsystem up through the port of the request, but then takes the controller from the subsystem list whatever port it was created on. A controller belongs to one port and one
transport: ctrl->ops is that of the admin queue, and both ->install_queue() and ->delete_ctrl() assume the queues are their own. So an I/O queue arriving on another port of the same subsystem, possibly on another transport, is installed into a controller it has no part in, nvmet_tcp_install_queue() writes to it as a TCP queue, and an error on it deletes that controller. The Connect comes before any authentication.

Skip controllers of other ports, and compare the transport too since a port's transport can be changed while it's disabled.

Fixes a07b4970f464 ("nvmet: add a generic NVMe target")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/core.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/target/core.c +5 -0
index 333739f14c2b..d01f8abed66c 100644
@@ -1507,6 +1507,11 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn,
 				pr_warn("hostnqn mismatch.\n");
 				continue;
 			}
+			/* All queues of a controller come in by its port. */
+			if (ctrl->port != req->port || ctrl->ops != req->ops) {
+				pr_warn("port mismatch.\n");
+				continue;
+			}
 			if (!kref_get_unless_zero(&ctrl->ref))
 				continue;
 
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 68/98
#145 lore raw

Re: [PATCH 68/98] nvmet: accept I/O queues on the controller's port only

Quoted 31 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_ctrl_find_get() looks the subsystem up through the port of the request, but then takes the controller from the subsystem list whatever port it was created on. A controller belongs to one port and one
transport: ctrl->ops is that of the admin queue, and both ->install_queue() and ->delete_ctrl() assume the queues are their own. So an I/O queue arriving on another port of the same subsystem, possibly on another transport, is installed into a controller it has no part in, nvmet_tcp_install_queue() writes to it as a TCP queue, and an error on it deletes that controller. The Connect comes before any authentication.

Skip controllers of other ports, and compare the transport too since a port's transport can be changed while it's disabled.

Fixes a07b4970f464 ("nvmet: add a generic NVMe target")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/core.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/nvme/target/core.c +0 -0
index 333739f14c2b..d01f8abed66c 100644
@@ -1507,6 +1507,11 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn,
  				pr_warn("hostnqn mismatch.\n");

drivers/nvme/target/core.c struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn, · line 1517

Quoted 10 lines
drivers/nvme/target/core.c +5 -0
  				continue;
  			}
+			/* All queues of a controller come in by its port. */
+			if (ctrl->port != req->port || ctrl->ops != req->ops) {
+				pr_warn("port mismatch.\n");
+				continue;
+			}
  			if (!kref_get_unless_zero(&ctrl->ref))
  				continue;
 

Ouch.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#146 lore raw

PATCH 69/98 nvmet: confine fatal errors of unauthenticated queues

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The Connect of an I/O queue names its controller by NQNs and cntlid and needs no secret, DH-HMAC-CHAP comes after it. But a fatal error on any queue, a failed authentication or a transport error like an unknown PDU type, takes the whole controller down. So anyone who can reach the port and knows the NQNs can attach a queue to an authenticated host's controller and kick that host out, repeatedly.

Add a ->disconnect_queue() transport op and, for an I/O queue that hasn't passed the authentication asked of it, disconnect just that queue. Transports without the op (fc, loop) take the controller down as before. A queue that had authenticated and fails to do so again is the host's own, there the controller goes down as before too.

[axboe: resolved against the per-queue auth_lock and expiry work that
is upstream now]

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/auth.c             |  9 ++++++---
 drivers/nvme/target/core.c             | 16 ++++++++++++++++
 drivers/nvme/target/fabrics-cmd-auth.c | 20 ++++++++++++++++----
 drivers/nvme/target/nvmet.h            |  7 +++++++
 drivers/nvme/target/rdma.c             |  9 ++++++++-
 drivers/nvme/target/tcp.c              | 11 ++++++++++-
 6 files changed, 63 insertions(+), 9 deletions(-)
drivers/nvme/target/auth.c +6 -3
index 5809ce32a2e4..335cf8f3f79e 100644
@@ -261,10 +261,8 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
 #endif
 }
 
-bool nvmet_check_auth_status(struct nvmet_req *req)
+bool nvmet_sq_auth_status(struct nvmet_sq *sq)
 {
-	struct nvmet_sq *sq = req->sq;
-
 	if (!sq->ctrl->host_key || sq->authenticated)
 		return true;
 
@@ -272,6 +270,11 @@ bool nvmet_check_auth_status(struct nvmet_req *req)
 	return sq->qid && nvmet_queue_tls_keyid(sq);
 }
 
+bool nvmet_check_auth_status(struct nvmet_req *req)
+{
+	return nvmet_sq_auth_status(req->sq);
+}
+
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int shash_len)
 {
drivers/nvme/target/core.c +16 -0
index d01f8abed66c..eea67fba6f92 100644
@@ -1828,6 +1828,22 @@ void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);
 
+/*
+ * A fatal error on one queue.  The Connect command of an I/O queue needs no
+ * secret, so until the queue has authenticated it may be anybody's: do not
+ * let it take down the controller of the host whose NQN it has named.
+ */
+void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+{
+	struct nvmet_ctrl *ctrl = sq->ctrl;
+
+	if (sq->qid && ctrl->ops->disconnect_queue && !nvmet_sq_auth_status(sq))
+		ctrl->ops->disconnect_queue(sq);
+	else
+		nvmet_ctrl_fatal_error(ctrl);
+}
+EXPORT_SYMBOL_GPL(nvmet_sq_fatal_error);
+
 ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 		char *traddr, size_t traddr_len)
 {
drivers/nvme/target/fabrics-cmd-auth.c +16 -4
index 8c48140eb542..cb6be285b3c4 100644
@@ -33,6 +33,14 @@ static void nvmet_auth_expired_work(struct work_struct *work)
 	mutex_unlock(&sq->auth_lock);
 }
 
+static void nvmet_auth_fatal_error(struct nvmet_sq *sq, bool passed)
+{
+	if (passed)
+		nvmet_ctrl_fatal_error(sq->ctrl);
+	else
+		nvmet_sq_fatal_error(sq);
+}
+
 void nvmet_auth_sq_init(struct nvmet_sq *sq)
 {
 	/* Initialize in-band authentication */
@@ -253,11 +261,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	void *d;
 	u32 tl;
 	u16 status = 0;
+	bool passed;
 	u8 dhchap_status;
 
 	mutex_lock(&req->sq->auth_lock);
 	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
 	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
 		req->error_loc =
@@ -430,7 +440,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	/* Final states, clear up variables */
 	nvmet_auth_sq_free(req->sq);
 	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
 
 complete:
 	mutex_unlock(&ctrl->auth_lock);
@@ -551,10 +561,12 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	void *d;
 	u32 al;
 	u16 status = 0;
+	bool passed;
 
 	mutex_lock(&req->sq->auth_lock);
 	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
 	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
 		req->error_loc =
@@ -641,7 +653,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 		nvmet_auth_sq_free(req->sq);
 	else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
 		nvmet_auth_sq_free(req->sq);
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
 	}
 	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
drivers/nvme/target/nvmet.h +7 -0
index cc7fface77de..e354a71514ae 100644
@@ -441,6 +441,7 @@ struct nvmet_fabrics_ops {
 	int (*add_port)(struct nvmet_port *port);
 	void (*remove_port)(struct nvmet_port *port);
 	void (*delete_ctrl)(struct nvmet_ctrl *ctrl);
+	void (*disconnect_queue)(struct nvmet_sq *sq);
 	void (*disc_traddr)(struct nvmet_req *req,
 			struct nvmet_port *port, char *traddr);
 	ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
@@ -622,6 +623,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);
 int nvmet_sq_init(struct nvmet_sq *sq, struct nvmet_cq *cq);
 
 void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl);
+void nvmet_sq_fatal_error(struct nvmet_sq *sq);
 
 void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new);
 
@@ -989,6 +991,7 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
 void nvmet_auth_sq_free(struct nvmet_sq *sq);
 void nvmet_auth_sq_destroy(struct nvmet_sq *sq);
 int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id);
+bool nvmet_sq_auth_status(struct nvmet_sq *sq);
 bool nvmet_check_auth_status(struct nvmet_req *req);
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int hash_len);
@@ -1015,6 +1018,10 @@ static inline void nvmet_auth_sq_init(struct nvmet_sq *sq)
 static inline void nvmet_destroy_auth(struct nvmet_ctrl *ctrl) {};
 static inline void nvmet_auth_sq_free(struct nvmet_sq *sq) {};
 static inline void nvmet_auth_sq_destroy(struct nvmet_sq *sq) {};
+static inline bool nvmet_sq_auth_status(struct nvmet_sq *sq)
+{
+	return true;
+}
 static inline bool nvmet_check_auth_status(struct nvmet_req *req)
 {
 	return true;
drivers/nvme/target/rdma.c +8 -1
index ffa1e172c259..c4b848f84465 100644
@@ -687,7 +687,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_error_comp(struct nvmet_rdma_queue *queue)
 {
 	if (queue->nvme_sq.ctrl) {
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
 	} else {
 		/*
 		 * we didn't setup the controller yet in case
@@ -1848,6 +1848,12 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,
 	return ret;
 }
 
+static void nvmet_rdma_disconnect_queue(struct nvmet_sq *sq)
+{
+	nvmet_rdma_queue_disconnect(container_of(sq, struct nvmet_rdma_queue,
+						 nvme_sq));
+}
+
 static void nvmet_rdma_delete_ctrl(struct nvmet_ctrl *ctrl)
 {
 	struct nvmet_rdma_queue *queue, *n;
@@ -2078,6 +2084,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
 	.remove_port		= nvmet_rdma_remove_port,
 	.queue_response		= nvmet_rdma_queue_response,
 	.delete_ctrl		= nvmet_rdma_delete_ctrl,
+	.disconnect_queue	= nvmet_rdma_disconnect_queue,
 	.disc_traddr		= nvmet_rdma_disc_port_addr,
 	.host_traddr		= nvmet_rdma_host_port_addr,
 	.get_mdts		= nvmet_rdma_get_mdts,
drivers/nvme/target/tcp.c +10 -1
index a2a5f39d2bb7..c8f0c52a1b95 100644
@@ -412,7 +412,7 @@ static void nvmet_tcp_socket_error(struct nvmet_tcp_queue *queue, int status)
 	if (status == -EPIPE || status == -ECONNRESET || !queue->nvme_sq.ctrl)
 		kernel_sock_shutdown(queue->sock, SHUT_RDWR);
 	else
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
 }
 
 static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
@@ -2223,6 +2223,14 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 	kfree(port);
 }
 
+static void nvmet_tcp_disconnect_queue(struct nvmet_sq *sq)
+{
+	struct nvmet_tcp_queue *queue =
+		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
+
+	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
+}
+
 static void nvmet_tcp_delete_ctrl(struct nvmet_ctrl *ctrl)
 {
 	struct nvmet_tcp_queue *queue;
@@ -2300,6 +2308,7 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = {
 	.remove_port		= nvmet_tcp_remove_port,
 	.queue_response		= nvmet_tcp_queue_response,
 	.delete_ctrl		= nvmet_tcp_delete_ctrl,
+	.disconnect_queue	= nvmet_tcp_disconnect_queue,
 	.install_queue		= nvmet_tcp_install_queue,
 	.disc_traddr		= nvmet_tcp_disc_port_addr,
 	.host_traddr		= nvmet_tcp_host_port_addr,
Signature · 1 line
2.56.0
maintainer
#147 lore raw

Re: [PATCH 69/98] nvmet: confine fatal errors of unauthenticated queues

Quoted 257 lines · Jens Axboe

From: Josef Bacik <[email protected]>

The Connect of an I/O queue names its controller by NQNs and cntlid and needs no secret, DH-HMAC-CHAP comes after it. But a fatal error on any queue, a failed authentication or a transport error like an unknown PDU type, takes the whole controller down. So anyone who can reach the port and knows the NQNs can attach a queue to an authenticated host's controller and kick that host out, repeatedly.

Add a ->disconnect_queue() transport op and, for an I/O queue that hasn't passed the authentication asked of it, disconnect just that queue. Transports without the op (fc, loop) take the controller down as before. A queue that had authenticated and fails to do so again is the host's own, there the controller goes down as before too.

[axboe: resolved against the per-queue auth_lock and expiry work that
is upstream now]

Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc Hannes Reinecke <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/auth.c             |  9 ++++++---
  drivers/nvme/target/core.c             | 16 ++++++++++++++++
  drivers/nvme/target/fabrics-cmd-auth.c | 20 ++++++++++++++++----
  drivers/nvme/target/nvmet.h            |  7 +++++++
  drivers/nvme/target/rdma.c             |  9 ++++++++-
  drivers/nvme/target/tcp.c              | 11 ++++++++++-
  6 files changed, 63 insertions(+), 9 deletions(-)
drivers/nvme/target/auth.c +6 -3
index 5809ce32a2e4..335cf8f3f79e 100644
@@ -261,10 +261,8 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
  #endif
  }
 
-bool nvmet_check_auth_status(struct nvmet_req *req)
+bool nvmet_sq_auth_status(struct nvmet_sq *sq)
  {
-	struct nvmet_sq *sq = req->sq;
-
  	if (!sq->ctrl->host_key || sq->authenticated)
  		return true;
 
@@ -272,6 +270,11 @@ bool nvmet_check_auth_status(struct nvmet_req *req)
  	return sq->qid && nvmet_queue_tls_keyid(sq);
  }
 
+bool nvmet_check_auth_status(struct nvmet_req *req)
+{
+	return nvmet_sq_auth_status(req->sq);
+}
+
  int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
  			 unsigned int shash_len)
  {
drivers/nvme/target/core.c +16 -0
index d01f8abed66c..eea67fba6f92 100644
@@ -1828,6 +1828,22 @@ void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl)
  }
  EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);
 
+/*
+ * A fatal error on one queue.  The Connect command of an I/O queue needs no
+ * secret, so until the queue has authenticated it may be anybody's: do not
+ * let it take down the controller of the host whose NQN it has named.
+ */
+void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+{
+	struct nvmet_ctrl *ctrl = sq->ctrl;
+
+	if (sq->qid && ctrl->ops->disconnect_queue && !nvmet_sq_auth_status(sq))
+		ctrl->ops->disconnect_queue(sq);
+	else
+		nvmet_ctrl_fatal_error(ctrl);
+}
+EXPORT_SYMBOL_GPL(nvmet_sq_fatal_error);
+
  ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
  		char *traddr, size_t traddr_len)
  {
drivers/nvme/target/fabrics-cmd-auth.c +16 -4
index 8c48140eb542..cb6be285b3c4 100644
@@ -33,6 +33,14 @@ static void nvmet_auth_expired_work(struct work_struct *work)
  	mutex_unlock(&sq->auth_lock);
  }
 
+static void nvmet_auth_fatal_error(struct nvmet_sq *sq, bool passed)
+{
+	if (passed)
+		nvmet_ctrl_fatal_error(sq->ctrl);
+	else
+		nvmet_sq_fatal_error(sq);
+}
+
  void nvmet_auth_sq_init(struct nvmet_sq *sq)
  {
  	/* Initialize in-band authentication */
@@ -253,11 +261,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  	void *d;
  	u32 tl;
  	u16 status = 0;
+	bool passed;
  	u8 dhchap_status;
 
  	mutex_lock(&req->sq->auth_lock);
  	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
  	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
  		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
  		req->error_loc =
@@ -430,7 +440,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
  	/* Final states, clear up variables */
  	nvmet_auth_sq_free(req->sq);
  	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
 
  complete:
  	mutex_unlock(&ctrl->auth_lock);
@@ -551,10 +561,12 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
  	void *d;
  	u32 al;
  	u16 status = 0;
+	bool passed;
 
  	mutex_lock(&req->sq->auth_lock);
  	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
  	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
  		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
  		req->error_loc =
@@ -641,7 +653,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
  		nvmet_auth_sq_free(req->sq);
  	else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
  		nvmet_auth_sq_free(req->sq);
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
  	}
  	mutex_unlock(&ctrl->auth_lock);
  	mutex_unlock(&req->sq->auth_lock);
drivers/nvme/target/nvmet.h +7 -0
index cc7fface77de..e354a71514ae 100644
@@ -441,6 +441,7 @@ struct nvmet_fabrics_ops {
  	int (*add_port)(struct nvmet_port *port);
  	void (*remove_port)(struct nvmet_port *port);
  	void (*delete_ctrl)(struct nvmet_ctrl *ctrl);
+	void (*disconnect_queue)(struct nvmet_sq *sq);
  	void (*disc_traddr)(struct nvmet_req *req,
  			struct nvmet_port *port, char *traddr);
  	ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
@@ -622,6 +623,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);
  int nvmet_sq_init(struct nvmet_sq *sq, struct nvmet_cq *cq);
 
  void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl);
+void nvmet_sq_fatal_error(struct nvmet_sq *sq);
 
  void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new);
 
@@ -989,6 +991,7 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
  void nvmet_auth_sq_free(struct nvmet_sq *sq);
  void nvmet_auth_sq_destroy(struct nvmet_sq *sq);
  int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id);
+bool nvmet_sq_auth_status(struct nvmet_sq *sq);
  bool nvmet_check_auth_status(struct nvmet_req *req);
  int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
  			 unsigned int hash_len);
@@ -1015,6 +1018,10 @@ static inline void nvmet_auth_sq_init(struct nvmet_sq *sq)
  static inline void nvmet_destroy_auth(struct nvmet_ctrl *ctrl) {};
  static inline void nvmet_auth_sq_free(struct nvmet_sq *sq) {};
  static inline void nvmet_auth_sq_destroy(struct nvmet_sq *sq) {};
+static inline bool nvmet_sq_auth_status(struct nvmet_sq *sq)
+{
+	return true;
+}
  static inline bool nvmet_check_auth_status(struct nvmet_req *req)
  {
  	return true;
drivers/nvme/target/rdma.c +8 -1
index ffa1e172c259..c4b848f84465 100644
@@ -687,7 +687,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
  static void nvmet_rdma_error_comp(struct nvmet_rdma_queue *queue)
  {
  	if (queue->nvme_sq.ctrl) {
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
  	} else {
  		/*
  		 * we didn't setup the controller yet in case
@@ -1848,6 +1848,12 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,
  	return ret;
  }
 
+static void nvmet_rdma_disconnect_queue(struct nvmet_sq *sq)
+{
+	nvmet_rdma_queue_disconnect(container_of(sq, struct nvmet_rdma_queue,
+						 nvme_sq));
+}
+
  static void nvmet_rdma_delete_ctrl(struct nvmet_ctrl *ctrl)
  {
  	struct nvmet_rdma_queue *queue, *n;
@@ -2078,6 +2084,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
  	.remove_port		= nvmet_rdma_remove_port,
  	.queue_response		= nvmet_rdma_queue_response,
  	.delete_ctrl		= nvmet_rdma_delete_ctrl,
+	.disconnect_queue	= nvmet_rdma_disconnect_queue,
  	.disc_traddr		= nvmet_rdma_disc_port_addr,
  	.host_traddr		= nvmet_rdma_host_port_addr,
  	.get_mdts		= nvmet_rdma_get_mdts,
drivers/nvme/target/tcp.c +9 -1
index a2a5f39d2bb7..c8f0c52a1b95 100644
@@ -412,7 +412,7 @@ static void nvmet_tcp_socket_error(struct nvmet_tcp_queue *queue, int status)
  	if (status == -EPIPE || status == -ECONNRESET || !queue->nvme_sq.ctrl)
  		kernel_sock_shutdown(queue->sock, SHUT_RDWR);
  	else
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
  }
 
  static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
@@ -2223,6 +2223,14 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
  	kfree(port);
  }
 
+static void nvmet_tcp_disconnect_queue(struct nvmet_sq *sq)
+{
+	struct nvmet_tcp_queue *queue =
+		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
+
+	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
+}
+
  static void nvmet_tcp_delete_ctrl(struct nvmet_ctrl *ctrl)
  {
  	struct nvmet_tcp_queue *queue;

drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops = { · line 2314

Quoted 7 lines
drivers/nvme/target/tcp.c +1 -0
@@ -2300,6 +2308,7 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = {
  	.remove_port		= nvmet_tcp_remove_port,
  	.queue_response		= nvmet_tcp_queue_response,
  	.delete_ctrl		= nvmet_tcp_delete_ctrl,
+	.disconnect_queue	= nvmet_tcp_disconnect_queue,
  	.install_queue		= nvmet_tcp_install_queue,
  	.disc_traddr		= nvmet_tcp_disc_port_addr,
  	.host_traddr		= nvmet_tcp_host_port_addr,

No reason why we shouldn't to this for FC.
I'll update the patch.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#148 lore raw

PATCH 70/98 nvmet-tcp: don't wait for the host in the accept work

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

On a port that offers TLS but doesn't require it, nvmet_tcp_alloc_queue() peeks at the first bytes of a new connection to tell an ICReq from a TLS handshake. That receive blocks until the host sends something or closes, and it runs in the accept work, of which there is one per port. A connection that stays silent blocks every other connect on that port for as long as it's held open, no secret needed. And nvmet_tcp_remove_port() waits for the accept work with nvmet_config_sem held for writing, so an admin taking the port down hangs there and no port of the target answers a Connect anymore.

Install the socket callbacks first and peek without blocking from the io work, which data_ready() reruns. If it's a handshake, restore the default callbacks for the upcall as before, and make the upcall from one more run of the io work so no run is left that could steal data from the handshake. The state change happens under state_lock and only if the queue isn't already going away. The io work doesn't hold the port like the accept work did, so note the keyring serial, which is all the upcall needs from the port, at allocation.

A partially arrived header used to be taken for "no TLS", now the rest is waited for. A silent connection still holds its queue, as on a port without TLS.

Fixes 70525e5d82f6 ("nvmet-tcp: peek icreq before starting TLS")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 67 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 59 insertions(+), 8 deletions(-)
drivers/nvme/target/tcp.c +59 -8
index c8f0c52a1b95..3d24efe3b7c3 100644
@@ -140,6 +140,7 @@ struct nvmet_tcp_cmd {
 
 enum nvmet_tcp_queue_state {
 	NVMET_TCP_Q_CONNECTING,
+	NVMET_TCP_Q_TLS_PEEK,
 	NVMET_TCP_Q_TLS_HANDSHAKE,
 	NVMET_TCP_Q_LIVE,
 	NVMET_TCP_Q_DISCONNECTING,
@@ -176,6 +177,7 @@ struct nvmet_tcp_queue {
 
 	/* TLS state */
 	key_serial_t		tls_pskid;
+	key_serial_t		tls_keyring;
 	struct delayed_work	tls_handshake_tmo_work;
 
 	unsigned long           poll_end;
@@ -1496,6 +1498,8 @@ static bool nvmet_tcp_check_queue_deadline(struct nvmet_tcp_queue *queue,
 	return !time_after(jiffies, queue->poll_end);
 }
 
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue);
+
 static void nvmet_tcp_io_work(struct work_struct *w)
 {
 	struct nvmet_tcp_queue *queue =
@@ -1503,6 +1507,11 @@ static void nvmet_tcp_io_work(struct work_struct *w)
 	bool pending;
 	int ret, ops = 0;
 
+	if ((queue->state == NVMET_TCP_Q_TLS_PEEK ||
+	     queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) &&
+	    !nvmet_tcp_tls_peek(queue))
+		return;
+
 	do {
 		pending = false;
 
@@ -1828,12 +1837,9 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 	struct msghdr msg = {
 		.msg_control = cbuf,
 		.msg_controllen = sizeof(cbuf),
-		.msg_flags = MSG_PEEK,
+		.msg_flags = MSG_PEEK | MSG_DONTWAIT,
 	};
 
-	if (nvmet_port_secure_channel_required(queue->port->nport))
-		return 0;
-
 	len = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(len < 0)) {
@@ -1955,7 +1961,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	args.ta_sock = queue->sock;
 	args.ta_done = nvmet_tcp_tls_handshake_done;
 	args.ta_data = queue;
-	args.ta_keyring = key_serial(queue->port->nport->keyring);
+	args.ta_keyring = queue->tls_keyring;
 	args.ta_timeout_ms = tls_handshake_timeout * 1000;
 
 	ret = tls_server_hello_psk(&args, GFP_KERNEL);
@@ -1968,8 +1974,51 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	}
 	return ret;
 }
+
+/*
+ * TLS is offered, but not required: the first bytes tell what the host wants.
+ * Returns true if the queue goes on without TLS.
+ */
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue)
+{
+	int ret;
+
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		if (nvmet_tcp_tls_handshake(queue))
+			nvmet_tcp_schedule_release_queue(queue);
+		return false;
+	}
+
+	/* ->data_ready() brings us back */
+	ret = nvmet_tcp_try_peek_pdu(queue);
+	if (ret == -EAGAIN)
+		return false;
+
+	/* The upcall wants the default callbacks */
+	if (!ret)
+		nvmet_tcp_restore_socket_callbacks(queue);
+
+	spin_lock_bh(&queue->state_lock);
+	if (queue->state != NVMET_TCP_Q_TLS_PEEK) {
+		spin_unlock_bh(&queue->state_lock);
+		return false;
+	}
+	queue->state = ret ? NVMET_TCP_Q_CONNECTING : NVMET_TCP_Q_TLS_HANDSHAKE;
+	spin_unlock_bh(&queue->state_lock);
+	if (ret)
+		return true;
+
+	/*
+	 * ->data_ready() may have queued the io work meanwhile, nothing does
+	 * from now on. The handshake must not start before that run is over,
+	 * so leave it to that run.
+	 */
+	queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
+	return false;
+}
 #else
 static void nvmet_tcp_tls_handshake_timeout(struct work_struct *w) {}
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue) { return true; }
 #endif
 
 static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
@@ -2039,14 +2088,16 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 		sk->sk_user_data = NULL;
 		sk->sk_data_ready = port->data_ready;
 		write_unlock_bh(&sk->sk_callback_lock);
-		if (!nvmet_tcp_try_peek_pdu(queue)) {
+		/* The port may be gone when the upcall is made */
+		queue->tls_keyring = key_serial(port->nport->keyring);
+		if (nvmet_port_secure_channel_required(port->nport)) {
 			if (!nvmet_tcp_tls_handshake(queue))
 				return;
 			/* TLS handshake failed, terminate the connection */
 			goto out_destroy_sq;
 		}
-		/* Not a TLS connection, continue with normal processing */
-		queue->state = NVMET_TCP_Q_CONNECTING;
+		/* Don't wait for the host here: others want to connect, too */
+		queue->state = NVMET_TCP_Q_TLS_PEEK;
 	}
 #endif
 
Signature · 1 line
2.56.0
author maintainer
#149 lore raw

PATCH 71/98 nvmet: check the host NQN against the TLS PSK identity

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The identity of a TLS PSK names the host and subsystem the key was generated for. The target remembers which key a handshake ended with, but never compares it with the host NQN in the Connect that follows. So a host holding any PSK in the port's keyring can connect as another host, sees what that host may see, and isn't asked for DH-HMAC-CHAP since TLS queues skip it. It can add an I/O queue to that host's TLS controller the same way.

Refuse a Connect whose host NQN isn't the one in the PSK identity of the queue. The subsystem NQN isn't compared, a discovery subsystem goes by two names.

Fixes 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/fabrics-cmd.c | 33 +++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
drivers/nvme/target/fabrics-cmd.c +33 -0
index ddb732f62981..089e3082c388 100644
@@ -264,6 +264,30 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq)
 		(needs_auth ? NVME_CONNECT_AUTHREQ_ATR : 0);
 }
 
+/*
+ * The identity of a TLS PSK names the host that the key was made for:
+ * "NVMe<version><type><hash> <host NQN> <subsystem NQN> ...".
+ * Whoever has proven to hold the key is that host, and no other one.
+ */
+static u16 nvmet_check_tls_host(struct nvmet_sq *sq, const char *hostnqn)
+{
+#ifdef CONFIG_NVME_TARGET_TCP_TLS
+	size_t len = strlen(hostnqn);
+	const char *id;
+
+	if (!sq->tls_key)
+		return 0;
+
+	id = strchr(sq->tls_key->description, ' ');
+	if (!id || strncmp(id + 1, hostnqn, len) || id[len + 1] != ' ') {
+		pr_warn("TLS PSK %08x does not belong to host %s\n",
+			key_serial(sq->tls_key), hostnqn);
+		return NVME_SC_CONNECT_INVALID_HOST | NVME_STATUS_DNR;
+	}
+#endif
+	return 0;
+}
+
 static void nvmet_execute_admin_connect(struct nvmet_req *req)
 {
 	struct nvmf_connect_command *c = &req->cmd->connect;
@@ -309,6 +333,10 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 
+	args.status = nvmet_check_tls_host(req->sq, d->hostnqn);
+	if (args.status)
+		goto out;
+
 	args.subsysnqn = d->subsysnqn;
 	args.hostnqn = d->hostnqn;
 	args.hostid = &d->hostid;
@@ -363,6 +391,11 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
+
+	status = nvmet_check_tls_host(req->sq, d->hostnqn);
+	if (status)
+		goto out;
+
 	ctrl = nvmet_ctrl_find_get(d->subsysnqn, d->hostnqn,
 				   le16_to_cpu(d->cntlid), req);
 	if (!ctrl) {
Signature · 1 line
2.56.0
author maintainer
#150 lore raw

PATCH 72/98 nvmet: fix NULL dereference of PCI endpoint targets

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_pci_epf_create_ctrl() calls nvmet_alloc_ctrl() without a queue, the admin queue comes later from the host. nvmet_alloc_ctrl() passes args->sq to nvmet_queue_tls_keyid(), which with CONFIG_NVME_TARGET_TCP_TLS dereferences it, so every PCI endpoint controller creation crashes on such a kernel. The debugfs tls_key file passes ctrl->sqs[0] to the same function, which is NULL until the admin queue exists and again once a fabrics host has closed it while keeping an I/O queue.

A queue that isn't there has no TLS key.

Fixes fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/nvmet.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/nvmet.h +2 -1
index e354a71514ae..5c2021b89ec4 100644
@@ -964,7 +964,8 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
 #ifdef CONFIG_NVME_TARGET_TCP_TLS
 static inline key_serial_t nvmet_queue_tls_keyid(struct nvmet_sq *sq)
 {
-	return sq->tls_key ? key_serial(sq->tls_key) : 0;
+	/* A PCI endpoint controller is created without a queue */
+	return sq && sq->tls_key ? key_serial(sq->tls_key) : 0;
 }
 static inline void nvmet_sq_put_tls_key(struct nvmet_sq *sq)
 {
Signature · 1 line
2.56.0
author maintainer
#151 lore raw

PATCH 73/98 nvmet-tcp: stop listening when a port is removed

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_accept_work() accepts until there's nothing left. A connection that is already dead when accepted is torn down right there, which takes a few ms, and more can arrive meanwhile. nvmet_tcp_remove_port() waits for the work with cancel_work_sync(), so someone opening and closing connections in a loop keeps the removal from finishing. It holds nvmet_config_sem for writing, so nothing can be configured and no port answers a Connect until the flood stops. No secret needed, just an admin removing the port at the wrong time.

Shut the listening socket down first. Pending connections are dropped, as they were on release, and the next kernel_accept() fails and ends the work.

Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/target/tcp.c +2 -0
index 3d24efe3b7c3..0b70d3647e28 100644
@@ -2263,6 +2263,8 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 	port->sock->sk->sk_data_ready = port->data_ready;
 	port->sock->sk->sk_user_data = NULL;
 	write_unlock_bh(&port->sock->sk->sk_callback_lock);
+	/* The accept work goes on for as long as there is something to accept */
+	kernel_sock_shutdown(port->sock, SHUT_RDWR);
 	cancel_work_sync(&port->accept_work);
 	/*
 	 * Destroy the remaining queues, which are not belong to any
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 73/98
#152 lore raw

Re: [PATCH 73/98] nvmet-tcp: stop listening when a port is removed

Quoted 30 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_accept_work() accepts until there's nothing left. A connection that is already dead when accepted is torn down right there, which takes a few ms, and more can arrive meanwhile. nvmet_tcp_remove_port() waits for the work with cancel_work_sync(), so someone opening and closing connections in a loop keeps the removal from finishing. It holds nvmet_config_sem for writing, so nothing can be configured and no port answers a Connect until the flood stops. No secret needed, just an admin removing the port at the wrong time.

Shut the listening socket down first. Pending connections are dropped, as they were on release, and the next kernel_accept() fails and ends the work.

Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/tcp.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/target/tcp.c +0 -0
index 3d24efe3b7c3..0b70d3647e28 100644

drivers/nvme/target/tcp.c static void nvmet_tcp_remove_port(struct nvmet_port *nport) · line 2270

Quoted 8 lines
drivers/nvme/target/tcp.c +2 -0
@@ -2263,6 +2263,8 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
  	port->sock->sk->sk_data_ready = port->data_ready;
  	port->sock->sk->sk_user_data = NULL;
  	write_unlock_bh(&port->sock->sk->sk_callback_lock);
+	/* The accept work goes on for as long as there is something to accept */
+	kernel_sock_shutdown(port->sock, SHUT_RDWR);
  	cancel_work_sync(&port->accept_work);
  	/*
  	 * Destroy the remaining queues, which are not belong to any

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#153 lore raw

PATCH 74/98 nvme-tcp: keep the page frag cache with the controller

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvme_tcp_init_request() allocates the PDU from the queue's page frag cache under pf_cache_lock, and it's called not just for the tag set but for the flush request of every hctx of a new namespace. The cache and the lock are torn down and reinitialized with the connection though, and a namespace scan isn't stopped by error recovery, so it allocates from a cache that's being drained or set up:

  WARNING: kernel/locking/mutex.c:625 at __mutex_lock+0xa25/0x1a30
   nvme_tcp_init_request+0x8c/0x1a0
   blk_mq_alloc_and_init_hctx+0x28e/0x6c0

Without lock debugging a waiter can spin forever on a mutex that's being reinitialized, an allocation beside the drain can return a fragment of a freed page, and what's allocated after the drain leaks if the controller goes away. The target, or anyone who can break the connection, picks the moment.

The PDUs outlive the connection anyway. Initialize the lock with the controller and drain the caches when it's freed.

Fixes a5053c92b3db ("nvme-tcp: fix memory leak when freeing a queue")
Fixes 36ac05f7cfd5 ("nvme-tcp: fix usage of page_frag_cache")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
drivers/nvme/host/tcp.c +13 -7
index c9531e653a2c..4aa666dd0266 100644
@@ -162,6 +162,7 @@ struct nvme_tcp_queue_setup_ctx {
 struct nvme_tcp_ctrl {
 	/* read only in the hot path */
 	struct nvme_tcp_queue	*queues;
+	int			nr_queues;	/* allocated, see queue_count */
 	struct blk_mq_tag_set	tag_set;
 
 	/* other member variables */
@@ -1668,8 +1669,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
 		return;
 
-	page_frag_cache_drain(&queue->pf_cache);
-
 	/**
 	 * Prevent memory reclaim from triggering block I/O during socket
 	 * teardown. The socket release path fput -> tcp_close ->
@@ -1692,7 +1691,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->pf_cache_lock);
 }
 
 static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
@@ -2021,7 +2019,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	INIT_LIST_HEAD(&queue->send_list);
 	mutex_init(&queue->send_mutex);
 	INIT_WORK(&queue->io_work, nvme_tcp_io_work);
-	mutex_init(&queue->pf_cache_lock);
 
 	if (qid > 0)
 		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -2153,7 +2150,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
 
@@ -2840,6 +2836,10 @@ static void nvme_tcp_stop_ctrl(struct nvme_ctrl *ctrl)
 static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
 {
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+	int i;
+
+	for (i = 0; i < ctrl->nr_queues; i++)
+		page_frag_cache_drain(&ctrl->queues[i].pf_cache);
 
 	if (list_empty_careful(&ctrl->list))
 		goto free_ctrl;
@@ -3265,9 +3265,15 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		ret = -ENOMEM;
 		goto out_free_ctrl;
 	}
-	/* A poller may try the lock of a queue that has been freed */
-	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+	ctrl->nr_queues = ctrl->ctrl.queue_count;
+	/*
+	 * A poller may try the lock of a queue that has been freed, and a
+	 * request of a new namespace may be set up for one.
+	 */
+	for (i = 0; i < ctrl->nr_queues; i++) {
 		mutex_init(&ctrl->queues[i].queue_lock);
+		mutex_init(&ctrl->queues[i].pf_cache_lock);
+	}
 
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
Signature · 1 line
2.56.0
author maintainer
#154 lore raw

PATCH 75/98 nvmet-tcp: hold the queue while the TLS timer is armed

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_tls_handshake() hands the queue to tls_server_hello_psk() and arms the timeout work afterwards. The reference it takes belongs to the handshake and is dropped by nvmet_tcp_tls_handshake_done(). If the handshake already ended by then, the callback found no timer to cancel, and if it failed because the host closed the connection, the queue may be gone as well. The timer is then armed in freed memory and fires there. Anyone who can reach a TLS port can start a handshake and close the connection, it just takes a quick tlshd.

Hold a second reference until the timer is armed, and arm it under state_lock only if the handshake is still in progress. The callback changes the state under that lock before it cancels the timer.

Fixes 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +10 -2
index 0b70d3647e28..908614fa82b0 100644
@@ -1955,6 +1955,8 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 		return -EINVAL;
 	}
 
+	/* One for the handshake, one to keep the queue until we are through */
+	kref_get(&queue->kref);
 	kref_get(&queue->kref);
 	pr_debug("queue %d: TLS ServerHello\n", queue->idx);
 	memset(&args, 0, sizeof(args));
@@ -1969,9 +1971,15 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 		kref_put(&queue->kref, nvmet_tcp_release_queue);
 		pr_err("failed to start TLS, err=%d\n", ret);
 	} else {
-		queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work,
-				   tls_handshake_timeout * HZ);
+		/* The handshake may be over, then nobody cancels the timer */
+		spin_lock_bh(&queue->state_lock);
+		if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE)
+			queue_delayed_work(nvmet_wq,
+					   &queue->tls_handshake_tmo_work,
+					   tls_handshake_timeout * HZ);
+		spin_unlock_bh(&queue->state_lock);
 	}
+	kref_put(&queue->kref, nvmet_tcp_release_queue);
 	return ret;
 }
 
Signature · 1 line
2.56.0
author maintainer
#155 lore raw

PATCH 76/98 nvmet: fix NULL dereference in debugfs host_traddr

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

The host_traddr callbacks of tcp, rdma and fc take the host address from the transport queue around ctrl->sqs[0], which nvmet_sq_destroy() clears when the admin queue goes away while the controller lives on with its I/O queues. A host that closes the admin connection and keeps an I/O queue leaves such a controller behind, and reading its host_traddr in debugfs dereferences a near-NULL pointer:

  BUG: kernel NULL pointer dereference, address: 00000000000002e0
  RIP: 0010:nvmet_tcp_host_port_addr+0x19/0x50 [nvmet_tcp]

Don't ask the transport if there's no admin queue. Nothing holds the queue for debugfs, so a reader that's past the check when the queue goes away is still on its own.

Fixes 7e5c3de3f2f2 ("nvmet: add 'host_traddr' callback for debugfs")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/core.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/nvme/target/core.c +3 -0
index eea67fba6f92..142df0db5208 100644
@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
+	/* They all take the address from the admin queue */
+	if (!ctrl->sqs[0])
+		return -ENOTCONN;
 	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
 }
 
Signature · 1 line
2.56.0
#156 lore raw

Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr

drivers/nvme/target/core.c ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, · line 1856

Quoted 9 lines · Jens Axboe
drivers/nvme/target/core.c +3 -0
@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
+	/* They all take the address from the admin queue */
+	if (!ctrl->sqs[0])
+		return -ENOTCONN;
 	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
 }

As the commit message notes, nothing synchronizes this check with nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(), and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and dereference the container transport queue, leaving both a TOCTOU NULL dereference and a use-after-free.

Instead of reaching into ctrl->sqs[0] from debugfs, why not format and save the host transport address once in a char host_traddr[NVMF_TRADDR_SIZE] buffer on struct nvmet_ctrl when the admin queue connects, and have nvmet_ctrl_host_traddr() return that string? That avoids touching the admin queue after setup altogether.

#157 lore raw

Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr

In reply to Jens Axboe

Review ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, · line 1856

Quoted 9 lines · Jens Axboe
(quoted diff) +3 -0
@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
+	/* They all take the address from the admin queue */
+	if (!ctrl->sqs[0])
+		return -ENOTCONN;
 	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
 }

As the commit message notes, nothing synchronizes this check with nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(), and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and dereference the container transport queue, leaving a TOCTOU NULL dereference and use-after-free.

Since args->sq is already passed to nvmet_alloc_ctrl() when the admin queue creates the controller, we can pass struct nvmet_sq * directly to ops->host_traddr, format the address once into ctrl->host_traddr before nvmet_debugfs_ctrl_setup(ctrl), and have nvmet_ctrl_host_traddr() copy out ctrl->host_traddr. That avoids touching ctrl->sqs[0] or the admin transport queue after controller allocation altogether, and also keeps debugfs host_traddr working even if the admin connection has closed.

Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

drivers/nvme/target/core.c +7 -1
index eea67fba6f92..b837c591af36 100644
@@ -1729,6 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 		goto init_pr_fail;
 	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
 	nvmet_setup_p2p_ns_map(ctrl, args->p2p_client);
+	if (ctrl->ops->host_traddr && args->sq &&
+	    ctrl->ops->host_traddr(args->sq, ctrl->host_traddr,
+				   sizeof(ctrl->host_traddr)) < 0)
+		ctrl->host_traddr[0] = '\0';
 	nvmet_debugfs_ctrl_setup(ctrl);
 	mutex_unlock(&subsys->lock);
 	up_read(&nvmet_config_sem);
@@ -1849,7 +1853,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
-	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
+	if (!ctrl->host_traddr[0])
+		return -EINVAL;
+	return strscpy(traddr, ctrl->host_traddr, traddr_len);
 }
 
 static struct nvmet_subsys *nvmet_find_get_subsys(struct nvmet_port *port,
drivers/nvme/target/fc.c +1 -2
index eecdffefa24f..1e1583c8b7e7 100644
@@ -2960,10 +2960,9 @@ nvmet_fc_discovery_chg(struct nvmet_port *port)
 }
 
 static ssize_t
-nvmet_fc_host_traddr(struct nvmet_ctrl *ctrl,
+nvmet_fc_host_traddr(struct nvmet_sq *sq,
 		char *traddr, size_t traddr_size)
 {
-	struct nvmet_sq *sq = ctrl->sqs[0];
 	struct nvmet_fc_tgt_queue *queue =
 		container_of(sq, struct nvmet_fc_tgt_queue, nvme_sq);
 	struct nvmet_fc_tgtport *tgtport = queue->assoc ? queue->assoc->tgtport : NULL;
drivers/nvme/target/nvmet.h +2 -1
index 5c2021b89ec4..e05cbf463751 100644
@@ -297,6 +297,7 @@ struct nvmet_ctrl {
 	u32			nr_changed_ns;
 
 	char			hostnqn[NVMF_NQN_FIELD_LEN];
+	char			host_traddr[NVMF_TRADDR_SIZE];
 
 	struct device		*p2p_client;
 	struct radix_tree_root	p2p_ns_map;
@@ -444,7 +445,7 @@ struct nvmet_fabrics_ops {
 	void (*disconnect_queue)(struct nvmet_sq *sq);
 	void (*disc_traddr)(struct nvmet_req *req,
 			struct nvmet_port *port, char *traddr);
-	ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
+	ssize_t (*host_traddr)(struct nvmet_sq *sq,
 			char *traddr, size_t traddr_len);
 	u16 (*install_queue)(struct nvmet_sq *nvme_sq);
 	void (*discovery_chg)(struct nvmet_port *port);
drivers/nvme/target/rdma.c +1 -2
index c4b848f84465..5e24cdb3a71f 100644
@@ -2050,10 +2050,9 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
 	}
 }
 
-static ssize_t nvmet_rdma_host_port_addr(struct nvmet_ctrl *ctrl,
+static ssize_t nvmet_rdma_host_port_addr(struct nvmet_sq *nvme_sq,
 		char *traddr, size_t traddr_len)
 {
-	struct nvmet_sq *nvme_sq = ctrl->sqs[0];
 	struct nvmet_rdma_queue *queue =
 		container_of(nvme_sq, struct nvmet_rdma_queue, nvme_sq);
 
drivers/nvme/target/tcp.c +1 -2
index 908614fa82b0..57b00f3ae988 100644
@@ -2348,10 +2348,9 @@ static void nvmet_tcp_disc_port_addr(struct nvmet_req *req,
 	}
 }
 
-static ssize_t nvmet_tcp_host_port_addr(struct nvmet_ctrl *ctrl,
+static ssize_t nvmet_tcp_host_port_addr(struct nvmet_sq *sq,
 			char *traddr, size_t traddr_len)
 {
-	struct nvmet_sq *sq = ctrl->sqs[0];
 	struct nvmet_tcp_queue *queue =
 		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
 
author maintainer
#158 lore raw

PATCH 77/98 nvmet-tcp: let the accept work come to an end

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_accept_work() loops until there's nothing left to accept, and anyone opening and closing connections fast enough keeps it running indefinitely, since tearing down a dead connection takes long enough for more to arrive. Whoever flushes nvmet_wq waits for it: removing any other port of the target does in nvmet_port_release(), and so does unloading the rdma and fc modules.

Accept a bounded number of connections per run and requeue the work. A flush doesn't wait for work queued behind it.

Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +6 -2
index 908614fa82b0..a0adcc7f0740 100644
@@ -85,6 +85,7 @@ MODULE_PARM_DESC(tls_handshake_timeout,
 #define NVMET_TCP_RECV_BUDGET		8
 #define NVMET_TCP_SEND_BUDGET		8
 #define NVMET_TCP_IO_WORK_BUDGET	64
+#define NVMET_TCP_ACCEPT_BUDGET		16
 
 enum nvmet_tcp_send_state {
 	NVMET_TCP_SEND_DATA_PDU,
@@ -2145,9 +2146,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 	struct nvmet_tcp_port *port =
 		container_of(w, struct nvmet_tcp_port, accept_work);
 	struct socket *newsock;
-	int ret;
+	int ret, i;
 
-	while (true) {
+	for (i = 0; i < NVMET_TCP_ACCEPT_BUDGET; i++) {
 		ret = kernel_accept(port->sock, &newsock, O_NONBLOCK);
 		if (ret < 0) {
 			if (ret != -EAGAIN)
@@ -2156,6 +2157,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 		}
 		nvmet_tcp_alloc_queue(port, newsock);
 	}
+
+	/* There may be more. Come to an end for those who flush nvmet_wq. */
+	queue_work(nvmet_wq, &port->accept_work);
 }
 
 static void nvmet_tcp_listen_data_ready(struct sock *sk)
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 77/98
#159 lore raw

Re: [PATCH 77/98] nvmet-tcp: let the accept work come to an end

Quoted 18 lines · Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_accept_work() loops until there's nothing left to accept, and anyone opening and closing connections fast enough keeps it running indefinitely, since tearing down a dead connection takes long enough for more to arrive. Whoever flushes nvmet_wq waits for it: removing any other port of the target does in nvmet_port_release(), and so does unloading the rdma and fc modules.

Accept a bounded number of connections per run and requeue the work. A flush doesn't wait for work queued behind it.

Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 4 lines
---
  drivers/nvme/target/tcp.c | 8 ++++++--
  1 file changed, 6 insertions(+), 2 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#160 lore raw

PATCH 78/98 nvmet-tcp: don't read a removed port at Connect time

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

nvmet_tcp_alloc_cmd() takes the nvmet port of a command from queue->port, and the commands of a queue are allocated from the io work when the Connect arrives. nvmet_tcp_remove_port() shuts down the queues that have no controller yet and frees the port without waiting for them, they're released later from a work. An io work handling a Connect at that moment reads the freed port for every command of the queue and hands the result to the core.

Note the nvmet port in the queue at allocation. It outlives the transport port.

Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/tcp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/tcp.c +4 -1
index a0adcc7f0740..4cc65decb02e 100644
@@ -151,6 +151,7 @@ enum nvmet_tcp_queue_state {
 struct nvmet_tcp_queue {
 	struct socket		*sock;
 	struct nvmet_tcp_port	*port;
+	struct nvmet_port	*nport;
 	struct work_struct	io_work;
 	struct nvmet_cq		nvme_cq;
 	struct nvmet_sq		nvme_sq;
@@ -1544,7 +1545,7 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
 	u8 hdgst = nvmet_tcp_hdgst_len(queue);
 
 	c->queue = queue;
-	c->req.port = queue->port->nport;
+	c->req.port = queue->nport;
 
 	c->cmd_pdu = page_frag_alloc(&queue->pf_cache,
 			sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
@@ -2048,6 +2049,8 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	kref_init(&queue->kref);
 	queue->sock = newsock;
 	queue->port = port;
+	/* The port may be gone when the commands are allocated */
+	queue->nport = port->nport;
 	queue->nr_cmds = 0;
 	spin_lock_init(&queue->state_lock);
 	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
Signature · 1 line
2.56.0
author maintainer
#161 lore raw

PATCH 79/98 nvme-tcp: start error recovery when a send fails

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

When the target closes a connection, nvme_tcp_state_change() calls nvme_tcp_error_recovery(), which does nothing unless the controller is live. While connecting that's fine for the queue being set up, its Connect fails and the attempt with it. But if an already connected queue is closed at the end of the attempt, nothing fails, the notification is lost and the controller goes live on a dead socket. Every request on that queue then fails with "failed to send request -32" and is retried, forever with native multipathing, and if it's the admin queue the first keep-alive fails too. Nothing but a manual reset recovers it. The Linux target closes its connections at exactly that moment after a fatal error.

Start error recovery when a request can't be sent, as nvme-rdma does.

Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
 drivers/nvme/host/tcp.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +2 -0
index 4aa666dd0266..52c7f6dd34d0 100644
@@ -1461,6 +1461,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
 			"failed to send request %d\n", ret);
 		nvme_tcp_fail_request(queue->request);
 		nvme_tcp_done_send_req(queue);
+		/* The close may have come when nobody could act on it */
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
 	}
 out:
 	memalloc_noreclaim_restore(noreclaim_flag);
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 79/98
#162 lore raw

Re: [PATCH 79/98] nvme-tcp: start error recovery when a send fails

Quoted 31 lines · Jens Axboe

From: Josef Bacik <[email protected]>

When the target closes a connection, nvme_tcp_state_change() calls nvme_tcp_error_recovery(), which does nothing unless the controller is live. While connecting that's fine for the queue being set up, its Connect fails and the attempt with it. But if an already connected queue is closed at the end of the attempt, nothing fails, the notification is lost and the controller goes live on a dead socket. Every request on that queue then fails with "failed to send request -32" and is retried, forever with native multipathing, and if it's the admin queue the first keep-alive fails too. Nothing but a manual reset recovers it. The Linux target closes its connections at exactly that moment after a fatal error.

Start error recovery when a request can't be sent, as nvme-rdma does.

Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
---
  drivers/nvme/host/tcp.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +0 -0
index 4aa666dd0266..52c7f6dd34d0 100644

drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1468

Quoted 8 lines
drivers/nvme/host/tcp.c +2 -0
@@ -1461,6 +1461,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
  			"failed to send request %d\n", ret);
  		nvme_tcp_fail_request(queue->request);
  		nvme_tcp_done_send_req(queue);
+		/* The close may have come when nobody could act on it */
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
  	}
  out:
  	memalloc_noreclaim_restore(noreclaim_flag);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#163 lore raw

PATCH 80/98 nvme-loop: don't BUG() if a scan runs beside a reset

In reply to Jens Axboe

From: Josef Bacik <[email protected]>

A reset sets queue_count to 1 while it destroys the I/O queues and counts it back up when the new ones are made, and it doesn't stop the scan work. A new namespace found in that window trips the check in nvme_loop_init_hctx() for every hctx but the first:

kernel BUG at drivers/nvme/target/loop.c:224!

   nvme_loop_init_hctx+0x7d/0x80
   __blk_mq_realloc_hw_ctxs+0x3af/0xa80
   nvme_alloc_ns+0x1a2/0x22f0

The check protects nothing. ctrl->queues has nr_io_queues + 1 entries for the life of the controller, the tag set never has more hardware queues than that, and nvme_loop_queue_rq() rejects requests until the controller is live again.

Remove it.

Fixes 3a85a5de29ea ("nvme-loop: add a NVMe loopback host driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/loop.c | 2 --
 1 file changed, 2 deletions(-)
drivers/nvme/target/loop.c +0 -2
index 93ba6357a850..80b67608eb15 100644
@@ -221,8 +221,6 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(data);
 	struct nvme_loop_queue *queue = &ctrl->queues[hctx_idx + 1];
 
-	BUG_ON(hctx_idx >= ctrl->ctrl.queue_count);
-
 	/*
 	 * flush_end_io() can be called recursively for us, so use our own
 	 * lock class key for avoiding lockdep possible recursive locking,
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 80/98
#164 lore raw

Re: [PATCH 80/98] nvme-loop: don't BUG() if a scan runs beside a reset

Quoted 34 lines · Jens Axboe

From: Josef Bacik <[email protected]>

A reset sets queue_count to 1 while it destroys the I/O queues and counts it back up when the new ones are made, and it doesn't stop the scan work. A new namespace found in that window trips the check in nvme_loop_init_hctx() for every hctx but the first:

   kernel BUG at drivers/nvme/target/loop.c:224!
    nvme_loop_init_hctx+0x7d/0x80
    __blk_mq_realloc_hw_ctxs+0x3af/0xa80
    nvme_alloc_ns+0x1a2/0x22f0

The check protects nothing. ctrl->queues has nr_io_queues + 1 entries for the life of the controller, the tag set never has more hardware queues than that, and nvme_loop_queue_rq() rejects requests until the controller is live again.

Remove it.

Fixes 3a85a5de29ea ("nvme-loop: add a NVMe loopback host driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/loop.c | 2 --
  1 file changed, 2 deletions(-)
drivers/nvme/target/loop.c +0 -0
index 93ba6357a850..80b67608eb15 100644

drivers/nvme/target/loop.c static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, · line 226

Quoted 8 lines
drivers/nvme/target/loop.c +0 -2
@@ -221,8 +221,6 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
  	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(data);
  	struct nvme_loop_queue *queue = &ctrl->queues[hctx_idx + 1];
 
-	BUG_ON(hctx_idx >= ctrl->ctrl.queue_count);
-
  	/*
  	 * flush_end_io() can be called recursively for us, so use our own
  	 * lock class key for avoiding lockdep possible recursive locking,

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#165 lore raw

PATCH 81/98 nvmet: fix pc_ref use-after-free in preempt and abort

In reply to Jens Axboe

From: Chris Mason <[email protected]>

With reservations enabled on a namespace, the target keeps a pc_ref per controller, a percpu_ref counting that controller's I/O on the namespace. "Preempt and abort" kills the pc_refs of the preempted controllers and waits for the kill to be confirmed in nvmet_pr_set_ctrl_to_abort(), then waits for the count to drop to zero and resurrects them in nvmet_pr_do_abort(). Neither holds a reference to the pc_ref, only pr_sem. nvmet_ctrl_destroy_pr() frees the pc_refs under subsys->lock when the controller goes away, so a host that registers a key, disconnects, and sends preempt and abort with that key over a second connection frees a pc_ref that the abort is sleeping on:

preempt and abort on C1 last put of C2
nvmet_pr_set_ctrl_to_abort()
percpu_ref_kill_and_confirm() wait_for_completion()
nvmet_ctrl_destroy_pr()
percpu_ref_exit() kfree(pc_ref)
percpu_ref_switch_to_atomic_rcu() /* use after free */

Put the pc_refs that an abort has killed on a per-namespace list. nvmet_ctrl_destroy_pr() only flags a pc_ref on that list, and nvmet_pr_do_abort() frees flagged ones once it's done with them. The list and the flag are written under the xarray lock.

Fixes 5a47c2080a73 ("nvmet: support reservation feature")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/nvmet.h |  5 ++
 drivers/nvme/target/pr.c    | 92 ++++++++++++++++++++++++++-----------
 2 files changed, 69 insertions(+), 28 deletions(-)
drivers/nvme/target/nvmet.h +5 -0
index 5c2021b89ec4..563b966b8f75 100644
@@ -88,6 +88,8 @@ struct nvmet_pr {
 	 */
 	struct semaphore	pr_sem;
 	struct list_head	registrant_list;
+	/* The pc_refs that a running "preempt and abort" has killed. */
+	struct list_head	abort_list;
 };
 
 struct nvmet_pr_per_ctrl_ref {
@@ -95,6 +97,9 @@ struct nvmet_pr_per_ctrl_ref {
 	struct completion	free_done;
 	struct completion	confirm_done;
 	uuid_t			hostid;
+	/* Both are written under the lock of ns->pr_per_ctrl_refs. */
+	struct list_head	abort_entry;
+	bool			ctrl_gone;
 };
 
 struct nvmet_ns {
drivers/nvme/target/pr.c +64 -28

152 lines not on this page · raw message

Signature · 1 line
2.56.0
author maintainer
#166 lore raw

PATCH 82/98 nvmet: move preempt and abort work off nvmet-wq

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_pr_do_abort() runs on nvmet-wq, holds pr_sem while it waits for the preempted controllers' I/O, and releases it at the end. nvmet-wq also runs nvmet_tcp_release_queue_work(), which cancel_work_sync()s the io_work of a closed connection, and that io_work may be sleeping on pr_sem in a reservation command. So a host that keeps a write in flight on a preempted controller, sends a second preempt and abort, and then closes enough connections with a reservation command in flight fills nvmet-wq with release works that wait for pr_sem. When the first abort finishes, the second one is queued on a full workqueue behind them and never starts, pr_sem is never released, and the accept work doesn't run either, so nobody can connect.

Queue the abort work on its own workqueue so it starts regardless. The release works can still fill nvmet-wq while pr_sem is held, and a file-backed namespace's flush, DSM and write zeroes need nvmet-wq themselves, so this is a mitigation.

Fixes 5a47c2080a73 ("nvmet: support reservation feature")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/core.c  | 11 ++++++++++-
 drivers/nvme/target/nvmet.h |  1 +
 drivers/nvme/target/pr.c    |  6 +++++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/nvme/target/core.c +10 -1
index 142df0db5208..f96b21257169 100644
@@ -29,6 +29,7 @@ struct workqueue_struct *nvmet_wq;
 EXPORT_SYMBOL_GPL(nvmet_wq);
 struct workqueue_struct *nvmet_aen_wq;
 EXPORT_SYMBOL_GPL(nvmet_aen_wq);
+struct workqueue_struct *nvmet_pr_wq;
 
 /*
  * This read/write semaphore is used to synchronize access to configuration
@@ -2034,9 +2035,14 @@ static int __init nvmet_init(void)
 	if (!nvmet_aen_wq)
 		goto out_free_nvmet_work_queue;
 
+	nvmet_pr_wq = alloc_workqueue("nvmet-pr-wq",
+				      WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	if (!nvmet_pr_wq)
+		goto out_free_nvmet_aen_work_queue;
+
 	error = nvmet_init_debugfs();
 	if (error)
-		goto out_free_nvmet_aen_work_queue;
+		goto out_free_nvmet_pr_work_queue;
 
 	error = nvmet_init_discovery();
 	if (error)
@@ -2052,6 +2058,8 @@ static int __init nvmet_init(void)
 	nvmet_exit_discovery();
 out_exit_debugfs:
 	nvmet_exit_debugfs();
+out_free_nvmet_pr_work_queue:
+	destroy_workqueue(nvmet_pr_wq);
 out_free_nvmet_aen_work_queue:
 	destroy_workqueue(nvmet_aen_wq);
 out_free_nvmet_work_queue:
@@ -2071,6 +2079,7 @@ static void __exit nvmet_exit(void)
 	nvmet_exit_discovery();
 	nvmet_exit_debugfs();
 	ida_destroy(&cntlid_ida);
+	destroy_workqueue(nvmet_pr_wq);
 	destroy_workqueue(nvmet_aen_wq);
 	destroy_workqueue(nvmet_wq);
 	destroy_workqueue(buffered_io_wq);
drivers/nvme/target/nvmet.h +1 -0
index 563b966b8f75..9e8fbb2d1fc1 100644
@@ -532,6 +532,7 @@ extern struct workqueue_struct *buffered_io_wq;
 extern struct workqueue_struct *zbd_wq;
 extern struct workqueue_struct *nvmet_wq;
 extern struct workqueue_struct *nvmet_aen_wq;
+extern struct workqueue_struct *nvmet_pr_wq;
 
 static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
 {
drivers/nvme/target/pr.c +5 -1
index 1c0d6cac1a6b..1a03f74c2e14 100644
@@ -756,7 +756,11 @@ static void nvmet_execute_pr_acquire(struct nvmet_req *req)
 	if (!status && acquire_act == NVME_PR_ACQUIRE_ACT_PREEMPT_AND_ABORT) {
 		kfree(d);
 		INIT_WORK(&req->r.abort_work, nvmet_pr_do_abort);
-		queue_work(nvmet_wq, &req->r.abort_work);
+		/*
+		 * Not nvmet_wq: works on it can wait for pr_sem, which this
+		 * work releases.
+		 */
+		queue_work(nvmet_pr_wq, &req->r.abort_work);
 		return;
 	}
 
Signature · 1 line
2.56.0
author maintainer
#167 lore raw

PATCH 83/98 nvmet-rdma: treat a failed connect as a disconnect

In reply to Jens Axboe

From: Chris Mason <[email protected]>

Once a connect request has been accepted, a queue is only freed by its release_work, so that must be queued once. The disconnect paths take the queue off nvmet_rdma_queue_list first and queue the work only if it was still on it. nvmet_rdma_queue_connect_fail(), for REJECTED, UNREACHABLE and CONNECT_ERROR, queues it unconditionally and only warns if the queue isn't connecting. But the IB CM delivers a REJ from the peer while the connection is established and even after we've sent a DREQ, so a peer that causes a disconnect with a short capsule and then sends a REJ while the release work is running queues it a second time and the work runs on a freed queue. No credentials needed, just RoCE or IB reachability, reproduced over rxe with a UDP socket.

Route connect failures through nvmet_rdma_queue_disconnect(), so only the path that takes the queue off the list queues the work. A REJ on an established connection still frees the queue, the RDMA CM reports no disconnect after one.

Fixes 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
drivers/nvme/target/rdma.c +7 -8
index c4b848f84465..13243e3edca5 100644
@@ -1738,15 +1738,14 @@ static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
 		struct nvmet_rdma_queue *queue)
 {
-	WARN_ON_ONCE(queue->state != NVMET_RDMA_Q_CONNECTING);
-
-	mutex_lock(&nvmet_rdma_queue_mutex);
-	if (!list_empty(&queue->queue_list))
-		list_del_init(&queue->queue_list);
-	mutex_unlock(&nvmet_rdma_queue_mutex);
-
 	pr_err("failed to connect queue %d\n", queue->idx);
-	queue_work(nvmet_wq, &queue->release_work);
+
+	/*
+	 * The peer decides when these events arrive: a REJ is also delivered
+	 * for a connection that is established, or that we are disconnecting.
+	 * Whoever takes the queue off the list queues release_work, once.
+	 */
+	nvmet_rdma_queue_disconnect(queue);
 }
 
 /**
Signature · 1 line
2.56.0
author maintainer
#168 lore raw

PATCH 84/98 nvmet-rdma: disconnect only established queues

In reply to Jens Axboe

From: Chris Mason <[email protected]>

Over iWARP, rdma_disconnect() sleeps in iw_cm_disconnect() until the ESTABLISHED event of the connection has been handled, and iw_cm handles all events of all connections one at a time on iw_cm_wq. nvmet_rdma_destroy_port_queues(), nvmet_rdma_remove_one() and nvmet_rdma_delete_ctrl() walk the queue list with nvmet_rdma_queue_mutex held and call rdma_disconnect() for each, while nvmet_rdma_cm_handler() takes the same mutex. If the walk sleeps on a queue whose ESTABLISHED is queued behind another queue's event, both are stuck and iWARP connection handling is dead until reboot. A peer sets that up by connecting in a loop, or over siw by asking for MPA peer-to-peer mode and then staying silent, which delays ESTABLISHED indefinitely.

Only call rdma_disconnect() for a queue in NVMET_RDMA_Q_LIVE, which the handler sets when it gets ESTABLISHED. A queue that never got there is left to the release work: ib_drain_qp() moves the QP to the error state, which ends the connection, and rdma_destroy_id() tells the peer. Over IB and RoCE the skipped call did nothing more than that.

Fixes f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Fixes fcf73a804c7d ("nvmet-rdma: fix use-after-free when a port is removed")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +13 -2
index 13243e3edca5..c5970bd2dc29 100644
@@ -1687,7 +1687,7 @@ static void nvmet_rdma_queue_established(struct nvmet_rdma_queue *queue)
 
 static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 {
-	bool disconnect = false;
+	bool disconnect = false, live = false;
 	unsigned long flags;
 
 	pr_debug("cm_id= %p queue->state= %d\n", queue->cm_id, queue->state);
@@ -1706,6 +1706,7 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 		}
 		fallthrough;
 	case NVMET_RDMA_Q_LIVE:
+		live = queue->state == NVMET_RDMA_Q_LIVE;
 		queue->state = NVMET_RDMA_Q_DISCONNECTING;
 		disconnect = true;
 		break;
@@ -1715,7 +1716,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 	spin_unlock_irqrestore(&queue->state_lock, flags);
 
 	if (disconnect) {
-		rdma_disconnect(queue->cm_id);
+		/*
+		 * Over iWARP rdma_disconnect() sleeps until the CM has handled
+		 * the ESTABLISHED event of this connection. The CM handles the
+		 * events of all connections one at a time, and our handler
+		 * takes nvmet_rdma_queue_mutex, which callers of this function
+		 * hold. So a queue that has not seen the event is left to the
+		 * release work: ib_drain_qp() moves the queue pair to the error
+		 * state, which ends the connection.
+		 */
+		if (live)
+			rdma_disconnect(queue->cm_id);
 		queue_work(nvmet_wq, &queue->release_work);
 	}
 }
Signature · 1 line
2.56.0
author maintainer
#169 lore raw

PATCH 85/98 nvmet-rdma: take a device off the list last

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_rdma_remove_one() disconnects the queues of a device and flushes nvmet-wq, but since commit a3dd7d0022c3 it returns right away if the device isn't on device_list. nvmet_rdma_free_dev(), run by the release work of the last queue, takes the device off that list first and frees the SRQs and the PD afterwards. A device removal in between finds nothing on the list, returns, and the RDMA core frees the device under ib_dealloc_pd().

Free the SRQs and the PD first and unlist the device last, so the remove callback finds it and its flush waits for the release work.

Fixes a3dd7d0022c3 ("nvmet-rdma: Don't flush system_wq by default during remove_one")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/nvme/target/rdma.c +7 -3
index c5970bd2dc29..a85e6fa71e0b 100644
@@ -1190,13 +1190,17 @@ static void nvmet_rdma_free_dev(struct kref *ref)
 	struct nvmet_rdma_device *ndev =
 		container_of(ref, struct nvmet_rdma_device, ref);
 
+	nvmet_rdma_destroy_srqs(ndev);
+	ib_dealloc_pd(ndev->pd);
+
+	/*
+	 * Not earlier: nvmet_rdma_remove_one() waits for the release work that
+	 * got here only if it finds the device on the list.
+	 */
 	mutex_lock(&device_list_mutex);
 	list_del(&ndev->entry);
 	mutex_unlock(&device_list_mutex);
 
-	nvmet_rdma_destroy_srqs(ndev);
-	ib_dealloc_pd(ndev->pd);
-
 	kfree(ndev);
 }
 
Signature · 1 line
2.56.0
author maintainer
#170 lore raw

PATCH 86/98 nvmet-rdma: queue the release work under the mutex

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_rdma_remove_one() takes the queue mutex, unlists each queue of the device and queues its release work, then flushes nvmet-wq. So a queue that still holds objects on the device must either be on the list or have its release work queued. nvmet_rdma_queue_disconnect() breaks that: it unlists the queue under the mutex, drops it, and only then calls __nvmet_rdma_queue_disconnect() to queue the work. A device removal in that window finds nothing to wait for, and the release work then destroys the QP and CQ share on a freed device.

Queue the work before dropping the mutex, as nvmet_rdma_remove_one() itself does. That's safe now that rdma_disconnect() no longer sleeps on an iWARP event that needs the same mutex.

Fixes f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/nvme/target/rdma.c +6 -6
index a85e6fa71e0b..1f71ef2961ba 100644
@@ -1737,17 +1737,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 
 static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 {
-	bool disconnect = false;
-
+	/*
+	 * The release work is queued with the mutex held, so that
+	 * nvmet_rdma_remove_one() finds the queue on the list or its work on
+	 * the workqueue.
+	 */
 	mutex_lock(&nvmet_rdma_queue_mutex);
 	if (!list_empty(&queue->queue_list)) {
 		list_del_init(&queue->queue_list);
-		disconnect = true;
+		__nvmet_rdma_queue_disconnect(queue);
 	}
 	mutex_unlock(&nvmet_rdma_queue_mutex);
-
-	if (disconnect)
-		__nvmet_rdma_queue_disconnect(queue);
 }
 
 static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
Signature · 1 line
2.56.0
author maintainer
#171 lore raw

PATCH 87/98 nvmet-rdma: disconnect a queue on device removal

In reply to Jens Axboe

From: Chris Mason <[email protected]>

cma_remove_one() sends DEVICE_REMOVAL to the owner of every cm_id on the device and then waits, without timeout, until they're all destroyed. nvmet_rdma_device_removal() does nothing for a queue cm_id and leaves the queues to nvmet_rdma_remove_one(), which only works if that runs first. As a module it does, nvmet-rdma registers after rdma_cm. Built in, both register at the same initcall level and drivers/Makefile lists nvme/ ahead of infiniband/, so cma_remove_one() runs first and waits for cm_ids that only the later callback would destroy. Since the RDMA CM drops all later events for a cm_id once it has sent DEVICE_REMOVAL, a peer closing its connection doesn't help either. One connection from anyone who can reach the port hangs "rdma link delete" until something else ends the queue.

Call nvmet_rdma_queue_disconnect() for the queue. It unlists the queue and queues the release work, which destroys the cm_id. Don't wait for it, rdma_destroy_id() waits for the handler to return, which was the deadlock before commit f1d4ef7d8883.

Fixes f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/nvme/target/rdma.c +8 -9
index 1f71ef2961ba..baaec9a36d8d 100644
@@ -1773,10 +1773,13 @@ static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
  * queue cm_id and/or a device bound listener cm_id (where in this
  * case queue will be null).
  *
- * We registered an ib_client to handle device removal for queues,
- * so we only need to handle the listening port cm_ids. In this case
- * we nullify the priv to prevent double cm_id destruction and destroying
- * the cm_id implicitly by returning a non-zero rc to the callout.
+ * The RDMA CM waits until each cm_id of the device is destroyed, so a queue
+ * has to go now. Our ib_client also removes the queues, but it cannot be
+ * relied on here: built in, it is called after the RDMA CM's client.
+ *
+ * For a listening port cm_id we nullify the priv to prevent double cm_id
+ * destruction and destroying the cm_id implicitly by returning a non-zero
+ * rc to the callout.
  */
 static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
 		struct nvmet_rdma_queue *queue)
@@ -1784,11 +1787,7 @@ static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
 	struct nvmet_rdma_port *port;
 
 	if (queue) {
-		/*
-		 * This is a queue cm_id. we have registered
-		 * an ib_client to handle queues removal
-		 * so don't interfere and just return.
-		 */
+		nvmet_rdma_queue_disconnect(queue);
 		return 0;
 	}
 
Signature · 1 line
2.56.0
author maintainer
#172 lore raw

PATCH 88/98 nvmet-rdma: flush nvmet-wq on module exit

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_rdma_exit() relies on ib_unregister_client() to flush nvmet-wq through nvmet_rdma_remove_one(), but since commit a3dd7d0022c3 that returns at once for a device no longer on device_list. The release work of the last queue takes the device off the list in nvmet_rdma_free_dev() and still has to drop the mutex, free the struct and return through nvmet_rdma_release_queue_work(), so an rmmod in that window frees the module code it's running.

Flush nvmet-wq after ib_unregister_client(). No release work can be queued after that point.

Fixes a3dd7d0022c3 ("nvmet-rdma: Don't flush system_wq by default during remove_one")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/target/rdma.c +6 -0
index baaec9a36d8d..b961d1c4aaa4 100644
@@ -2170,6 +2170,12 @@ static void __exit nvmet_rdma_exit(void)
 {
 	nvmet_unregister_transport(&nvmet_rdma_ops);
 	ib_unregister_client(&nvmet_rdma_ib_client);
+	/*
+	 * nvmet_rdma_remove_one() does not flush for a device that has left
+	 * device_list, and the release work that took it off still has to
+	 * return from nvmet_rdma_free_dev().
+	 */
+	flush_workqueue(nvmet_wq);
 	WARN_ON_ONCE(!list_empty(&nvmet_rdma_queue_list));
 	ida_destroy(&nvmet_rdma_queue_ida);
 }
Signature · 1 line
2.56.0
author maintainer
#173 lore raw

PATCH 89/98 nvmet-rdma: reject queues larger than max_cqe

In reply to Jens Axboe

From: Chris Mason <[email protected]>

The host picks hrqsize and hsqsize in its RDMA CM connect request, and nvmet_rdma_create_queue_ib() asks ib_cq_pool_get() for
recv + 2 * send + 1 CQ entries, up to 196607 for an I/O queue. The pool limits each CQ it allocates to the device's max_cqe, so if the request exceeds that no CQ ever fits and ib_alloc_cqs() keeps adding CQs until an allocation fails. Those CQs stay until the device is removed. rxe has a max_cqe of 32767, and one connect request from an unprivileged process OOMs a 1 GiB target in under a second, with nothing to kill that frees the memory.

Refuse a queue that needs more than max_cqe entries before touching the pool. The connect is rejected with NVME_RDMA_CM_NO_RSC like any other queue creation failure.

Fixes ca0f1a8055be ("nvmet-rdma: use new shared CQ mechanism")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +11 -2
index b961d1c4aaa4..26ea4e1d62b3 100644
@@ -1286,8 +1286,17 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
 	 */
 	nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size;
 
-	queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
-				   queue->comp_vector, IB_POLL_WORKQUEUE);
+	/*
+	 * The host picks both sizes. Never ask ib_cq_pool_get() for more
+	 * entries than one CQ of the device can have: no CQ would ever fit,
+	 * and it would allocate new CQs until an allocation fails.
+	 */
+	if (nr_cqe + 1 > ndev->device->attrs.max_cqe)
+		queue->cq = ERR_PTR(-EINVAL);
+	else
+		queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
+					   queue->comp_vector,
+					   IB_POLL_WORKQUEUE);
 	if (IS_ERR(queue->cq)) {
 		ret = PTR_ERR(queue->cq);
 		pr_err("failed to create CQ cqe= %d ret= %d\n",
Signature · 1 line
2.56.0
author maintainer
#174 lore raw

PATCH 90/98 nvmet-rdma: disable interrupts under rsp_wr_wait_lock

In reply to Jens Axboe

From: Chris Mason <[email protected]>

rsp_wr_wait_lock protects the list of commands waiting for room in the send queue, and it's taken with plain spin_lock(). But nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() directly if ib_post_send() fails, and that processes the wait list. The call chain starts in nvmet_req_complete(), which for a block device namespace runs from the bio end_io callback, which can be in hard interrupt context. An interrupt that completes a command on a CPU holding the lock for the same queue spins forever. Lockdep over siw with a null_blk namespace:

inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage

Use spin_lock_irqsave() for both takers. The processing loop still drops the lock, and restores the caller's interrupt state, around nvmet_rdma_execute_command().

Fixes 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/nvme/target/rdma.c +9 -6
index 26ea4e1d62b3..7c97a47e305a 100644
@@ -513,7 +513,9 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 
 static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 {
-	spin_lock(&queue->rsp_wr_wait_lock);
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
 		bool ret;
@@ -522,16 +524,16 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
 
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
 		}
 	}
-	spin_unlock(&queue->rsp_wr_wait_lock);
+	spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 }
 
 static u16 nvmet_rdma_check_pi_status(struct ib_mr *sig_mr)
@@ -976,6 +978,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		struct nvmet_rdma_rsp *cmd)
 {
+	unsigned long flags;
 	u16 status;
 
 	ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +996,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		goto out_err;
 
 	if (unlikely(!nvmet_rdma_execute_command(cmd))) {
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 	}
 
 	return;
Signature · 1 line
2.56.0
#175 lore raw

Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock

drivers/nvme/target/rdma.c static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue) · line 529

Quoted 5 lines · Jens Axboe
drivers/nvme/target/rdma.c +2 -2
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);

Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself, but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io callback in hardirq or softirq context (via nvmet_req_complete() -> nvmet_rdma_queue_response() -> failed ib_post_send() -> nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) -> rsp->req.execute(&rsp->req).

The nvmet ->execute() handlers assume process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls submit_bio() -> submit_bio_noacct(), which hits might_sleep() and manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations and take mutexes / rwsems.

Shouldn't rsp_wr_wait_list processing be deferred to a work_struct when !in_task() (or whenever nvmet_rdma_release_rsp() is called from atomic context)?

#176 lore raw

Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock

In reply to Jens Axboe

Review

Quoted 5 lines · Jens Axboe
(quoted diff) +2 -2
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);

Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself, but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io callback in hardirq or softirq context (via nvmet_req_complete() -> nvmet_rdma_queue_response() -> failed ib_post_send() -> nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) -> rsp->req.execute(&rsp->req).

The nvmet ->execute() handlers require process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls submit_bio() -> submit_bio_noacct(), which hits might_sleep() and manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations and take mutexes / rwsems.

If nvmet_rdma_process_wr_wait_list() checks !in_task() under rsp_wr_wait_lock and schedules a per-queue wr_wait_work on system_dfl_wq when rsp_wr_wait_list is non-empty (cancelled after ib_drain_qp() in nvmet_rdma_destroy_queue_ib()), normal task/CQ-workqueue callers pay no extra cost while hardirq/softirq completions safely defer command execution to process context.

Note: patches 91/98 and 92/98 compose cleanly on top by placing the if (done && done->wait_released) check right above if (!in_task()) and passing NULL from nvmet_rdma_wr_wait_work().

Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---

drivers/nvme/target/rdma.c +26 -6
index 26ea4e1d62b3..60ea488db8f4 100644
@@ -104,6 +104,7 @@ struct nvmet_rdma_queue {
 	struct nvmet_rdma_cmd	*cmds;
 
 	struct work_struct	release_work;
+	struct work_struct	wr_wait_work;
 	struct list_head	rsp_wait_list;
 	struct list_head	rsp_wr_wait_list;
 	spinlock_t		rsp_wr_wait_lock;
@@ -513,7 +514,15 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 
 static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 {
-	spin_lock(&queue->rsp_wr_wait_lock);
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
+	if (!in_task()) {
+		if (!list_empty(&queue->rsp_wr_wait_list))
+			schedule_work(&queue->wr_wait_work);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		return;
+	}
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
 		bool ret;
@@ -522,16 +531,24 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
 
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
 		}
 	}
-	spin_unlock(&queue->rsp_wr_wait_lock);
+	spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+}
+
+static void nvmet_rdma_wr_wait_work(struct work_struct *w)
+{
+	struct nvmet_rdma_queue *queue =
+		container_of(w, struct nvmet_rdma_queue, wr_wait_work);
+
+	nvmet_rdma_process_wr_wait_list(queue);
 }
 
 static u16 nvmet_rdma_check_pi_status(struct ib_mr *sig_mr)
@@ -976,6 +993,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		struct nvmet_rdma_rsp *cmd)
 {
+	unsigned long flags;
 	u16 status;
 
 	ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +1011,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		goto out_err;
 
 	if (unlikely(!nvmet_rdma_execute_command(cmd))) {
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 	}
 
 	return;
@@ -1374,6 +1392,7 @@ static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr,
 static void nvmet_rdma_destroy_queue_ib(struct nvmet_rdma_queue *queue)
 {
 	ib_drain_qp(queue->qp);
+	cancel_work_sync(&queue->wr_wait_work);
 
 	/*
 	 * Reclaim resources of a response that is still in-flight when the
@@ -1494,6 +1513,7 @@ nvmet_rdma_alloc_queue(struct nvmet_rdma_device *ndev,
 	 * inside a CM callback would trigger a deadlock. (great API design..)
 	 */
 	INIT_WORK(&queue->release_work, nvmet_rdma_release_queue_work);
+	INIT_WORK(&queue->wr_wait_work, nvmet_rdma_wr_wait_work);
 	queue->dev = ndev;
 	queue->cm_id = cm_id;
 	queue->port = port->nport;
author maintainer
#177 lore raw

PATCH 91/98 nvmet-rdma: don't run the wait list recursively

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_rdma_release_rsp() returns the send queue slots of a response and then runs the wait list, passing each waiting command to nvmet_rdma_execute_command(). Many commands complete inside their execute callback, and if the response can't be posted, nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() right there, which starts another run of the wait list inside the first:

nvmet_rdma_process_wr_wait_list()
nvmet_rdma_execute_command()
nvmet_execute_admin_connect()
nvmet_req_complete()
nvmet_rdma_queue_response()
ib_post_send() fails
nvmet_rdma_release_rsp()
nvmet_rdma_process_wr_wait_list()

Each waiting command adds a level while posts keep failing, and the host decides how many commands wait: it sizes the receive queue, can fill the send queue with commands that nvmet_req_init() rejects without taking a slot, and over siw can make posts fail by holding back an RDMA READ. 400 waiting Connects overflow the stack of the CQ worker.

Before the loop executes a command, point the response's new wait_released at a flag on the loop's stack. A release that finds it set means the loop hasn't returned from that command yet: it returns the slots and releases the response, but sets the flag and leaves the list to the loop instead of running it. The flag tells the loop not to touch the response again.

Fixes 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)
drivers/nvme/target/rdma.c +25 -3
index 7c97a47e305a..b8098ec53d7b 100644
@@ -77,6 +77,8 @@ struct nvmet_rdma_rsp {
 	u32			invalidate_rkey;
 
 	struct list_head	wait_list;
+	/* set under rsp_wr_wait_lock while the wait list's loop executes us */
+	bool			*wait_released;
 	int			tag;
 };
 
@@ -511,23 +513,42 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 	return ret;
 }
 
-static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
+static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
+					    struct nvmet_rdma_rsp *done)
 {
 	unsigned long flags;
 
 	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
+	if (done->wait_released) {
+		/*
+		 * The loop below has not come back from this command yet. It is
+		 * further up this stack if the command completed in place and
+		 * its response could not be posted. Otherwise it runs in
+		 * another context. It goes on with the list, so leave the list
+		 * to it.
+		 */
+		*done->wait_released = true;
+		WRITE_ONCE(done->wait_released, NULL);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		return;
+	}
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
+		bool released = false;
 		bool ret;
 
 		rsp = list_entry(queue->rsp_wr_wait_list.next,
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
+		rsp->wait_released = &released;
 
 		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
 		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
+		/* A released response can be freed, or in use again. */
+		if (!released)
+			WRITE_ONCE(rsp->wait_released, NULL);
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
@@ -680,8 +701,9 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 
 	nvmet_rdma_free_rsp_resources(rsp);
 
-	if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
-		nvmet_rdma_process_wr_wait_list(queue);
+	if (unlikely(READ_ONCE(rsp->wait_released) ||
+		     !list_empty_careful(&queue->rsp_wr_wait_list)))
+		nvmet_rdma_process_wr_wait_list(queue, rsp);
 
 	nvmet_rdma_put_rsp(rsp);
 }
Signature · 1 line
2.56.0
author maintainer
#178 lore raw

PATCH 92/98 nvmet-rdma: fix a lost run of the wait list

In reply to Jens Axboe

From: Chris Mason <[email protected]>

A command is checked for free send queue slots before it's on the wait list, in nvmet_rdma_handle_command() and when the processing loop takes it off to retry, and nvmet_rdma_release_rsp() looks at the list without a lock after giving slots back. The CQ worker and the CM thread (via nvmet_rdma_queue_established()) can handle the commands of one queue concurrently, so a release can see an empty list while the other side is between the failed slot check and the list add:

handle a command release a response
too few free slots
give the slots back wait list empty: return
add the command to the list /* nobody runs the list */

If that was the last response in flight the command stays on the list forever. It holds sq->ref, so nvmet_sq_destroy() hangs at disconnect and the queue's memory is never freed. A host hits it by sending more Connects at once than the send queue has slots.

Look at the slots again after a command is on the list: the loop retries after a put-back with a full barrier in between, nvmet_rdma_release_rsp() has a matching barrier between returning the slots and looking at the list, and nvmet_rdma_handle_command() runs the list once after adding its command.

Fixes 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +17 -1
index b8098ec53d7b..7cbf1caf3208 100644
@@ -519,7 +519,7 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
 	unsigned long flags;
 
 	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
-	if (done->wait_released) {
+	if (done && done->wait_released) {
 		/*
 		 * The loop below has not come back from this command yet. It is
 		 * further up this stack if the command completed in place and
@@ -551,6 +551,15 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
 			WRITE_ONCE(rsp->wait_released, NULL);
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
+			/*
+			 * A release that came while the command was off the
+			 * list can have seen an empty list, so look at the
+			 * room again.
+			 * Pairs with the barrier in nvmet_rdma_release_rsp().
+			 */
+			smp_mb();
+			if (atomic_read(&queue->sq_wr_avail) > rsp->n_rdma)
+				continue;
 			break;
 		}
 	}
@@ -698,6 +707,11 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 	struct nvmet_rdma_queue *queue = rsp->queue;
 
 	atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
+	/*
+	 * Give the room back ahead of the look at the wait list below.
+	 * Pairs with the smp_mb() in nvmet_rdma_process_wr_wait_list().
+	 */
+	smp_mb__after_atomic();
 
 	nvmet_rdma_free_rsp_resources(rsp);
 
@@ -1021,6 +1035,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
 		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		/* The last response may have been released in between. */
+		nvmet_rdma_process_wr_wait_list(queue, NULL);
 	}
 
 	return;
Signature · 1 line
2.56.0
author maintainer
#179 lore raw

PATCH 93/98 nvmet-fc: only send read data for successful commands

In reply to Jens Axboe

From: Andrea Parri <[email protected]>

__nvmet_fc_fcp_nvme_cmd_done() pushes the whole data buffer of a read to the host even when the command failed. The buffer comes from sgl_alloc() without __GFP_ZERO, and a command rejected by nvmet_check_transfer_len() never touches it, so the host receives stale target memory, as much as it asked for.

The tcp and rdma targets only send data when cqe->status is zero. Do the same for fc. The ERSP then reports fod->offset, which is 0, as the transferred length, matching what the host checks.

Fixes c53432030d86 ("nvme-fabrics: Add target support for FC transport")
Cc Justin Tee <[email protected]>
Cc Naresh Gottumukkala <[email protected]>
Cc Paul Ely <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/fc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/fc.c +4 -4
index eecdffefa24f..b95893adae36 100644
@@ -2487,11 +2487,11 @@ __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport,
 	} else {
 
 		/*
-		 * try to push the data even if the SQE status is non-zero.
-		 * There may be a status where data still was intended to
-		 * be moved
+		 * A failed command may not have filled its read buffer, so
+		 * only push data for success, as the tcp and rdma targets do.
 		 */
-		if ((fod->io_dir == NVMET_FCP_READ) && (fod->data_sg_cnt)) {
+		if (fod->io_dir == NVMET_FCP_READ && fod->data_sg_cnt &&
+		    !cqe->status) {
 			/* push the data over before sending rsp */
 			nvmet_fc_transfer_fcp_data(tgtport, fod,
 						NVMET_FCOP_READDATA);
Signature · 1 line
2.56.0
maintainer Reviewed-by on v1 93/98
#180 lore raw

Re: [PATCH 93/98] nvmet-fc: only send read data for successful commands

Quoted 37 lines · Jens Axboe

From: Andrea Parri <[email protected]>

__nvmet_fc_fcp_nvme_cmd_done() pushes the whole data buffer of a read to the host even when the command failed. The buffer comes from sgl_alloc() without __GFP_ZERO, and a command rejected by nvmet_check_transfer_len() never touches it, so the host receives stale target memory, as much as it asked for.

The tcp and rdma targets only send data when cqe->status is zero. Do the same for fc. The ERSP then reports fod->offset, which is 0, as the transferred length, matching what the host checks.

Fixes c53432030d86 ("nvme-fabrics: Add target support for FC transport")
Cc Justin Tee <[email protected]>
Cc Naresh Gottumukkala <[email protected]>
Cc Paul Ely <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
  drivers/nvme/target/fc.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/fc.c +0 -2
index eecdffefa24f..b95893adae36 100644
@@ -2487,11 +2487,11 @@ __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport,
  	} else {
 
  		/*
-		 * try to push the data even if the SQE status is non-zero.
-		 * There may be a status where data still was intended to

drivers/nvme/target/fc.c __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport, · line 2497

Quoted 10 lines
drivers/nvme/target/fc.c +4 -2
-		 * be moved
+		 * A failed command may not have filled its read buffer, so
+		 * only push data for success, as the tcp and rdma targets do.
  		 */
-		if ((fod->io_dir == NVMET_FCP_READ) && (fod->data_sg_cnt)) {
+		if (fod->io_dir == NVMET_FCP_READ && fod->data_sg_cnt &&
+		    !cqe->status) {
  			/* push the data over before sending rsp */
  			nvmet_fc_transfer_fcp_data(tgtport, fod,
  						NVMET_FCOP_READDATA);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#181 lore raw

PATCH 94/98 nvmet: wait only for executing I/O in preempt and abort

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_parse_io_cmd() takes the pc_ref for an I/O command as soon as it's parsed, and "preempt and abort" waits for all pc_refs of the preempted controllers before it releases pr_sem. But a write that still needs its data can't complete without the host sending it and without the transport context receiving it, and that same context is where Register, Acquire and Release sleep on pr_sem. So a host that sends a write and holds the data back, then preempt and abort with its own key, then Register on the same queue, deadlocks io_work, the abort work and the write with each other. Two well-behaved hosts get there by accident when one fences the other while it re-registers under I/O. The connection can't be torn down, the namespace can't be disabled, and with nvmet_config_sem held across that, nobody can connect.

Take the pc_ref when the transport calls ->execute(), once all the data is there: nvmet_parse_io_cmd() stashes the real execute method and installs nvmet_pr_execute_io(), which gets the reference, rechecks the reservation access since the host may have lost it since parse time, and calls through. Preempt and abort then only waits for commands that have started executing, which need neither the host nor the transport to finish. A write whose data arrives after the abort now fails with Reservation Conflict where it was previously executed, that's the input the hang needs.

Fixes 5a47c2080a73 ("nvmet: support reservation feature")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/core.c  |  2 +-
 drivers/nvme/target/nvmet.h |  4 +++-
 drivers/nvme/target/pr.c    | 42 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 45 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +1 -1
index f96b21257169..b78a4d4eb838 100644
@@ -1171,7 +1171,7 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		if (ret)
 			return ret;
 
-		ret = nvmet_pr_get_ns_pc_ref(req);
+		nvmet_pr_set_execute(req);
 	}
 	return ret;
 }
drivers/nvme/target/nvmet.h +3 -1
index 9e8fbb2d1fc1..29105e2a7022 100644
@@ -517,6 +517,8 @@ struct nvmet_req {
 	struct nvmet_port	*port;
 
 	void (*execute)(struct nvmet_req *req);
+	/* what nvmet_pr_execute_io() calls once it has the pc_ref */
+	void (*pr_execute)(struct nvmet_req *req);
 	const struct nvmet_fabrics_ops *ops;
 
 	struct pci_dev		*p2p_dev;
@@ -1051,7 +1053,7 @@ void nvmet_pr_exit_ns(struct nvmet_ns *ns);
 void nvmet_execute_get_log_page_resv(struct nvmet_req *req);
 u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask);
 u16 nvmet_get_feat_resv_notif_mask(struct nvmet_req *req);
-u16 nvmet_pr_get_ns_pc_ref(struct nvmet_req *req);
+void nvmet_pr_set_execute(struct nvmet_req *req);
 static inline void nvmet_pr_put_ns_pc_ref(struct nvmet_pr_per_ctrl_ref *pc_ref)
 {
 	percpu_ref_put(&pc_ref->ref);
drivers/nvme/target/pr.c +41 -1

54 lines not on this page · raw message

Signature · 1 line
2.56.0
author maintainer
#182 lore raw

PATCH 95/98 nvmet-rdma: fix double puts after a failed RDMA WRITE

In reply to Jens Axboe

From: Chris Mason <[email protected]>

For a read with T10-PI the RDMA WRITE is posted alone and nvmet_rdma_write_data_done() posts the SEND once the metadata check result is known. By then the request has been completed and its references put: __nvmet_req_complete() put ns->ref and the pc_ref, and nvmet_req_complete() sq->ref, right after queuing the response. The error branch of nvmet_rdma_write_data_done() copies the one from nvmet_rdma_read_data_done() and calls nvmet_req_uninit(), which puts all three again. A WRITE fails whenever the connection ends with reads in flight, which is up to the host. Depending on timing the sq->ref count goes to zero early and the queue is freed under the remaining completions, or never reaches zero and nvmet_sq_destroy() sleeps forever, or percpu_ref_put() dereferences NULL after percpu_ref_exit(). ns->ref stays too low for good, so disabling the namespace hangs too. Reproduced with KASAN over rxe with a test patch forcing the PI path.

Drop the nvmet_req_uninit(). The branch still releases the RDMA context and the response, which is all the request owns at that point.

Fixes b09160c3996c ("nvmet-rdma: add metadata/T10-PI support")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/rdma.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +4 -1
index 7cbf1caf3208..994a668f45d6 100644
@@ -839,7 +839,10 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)
 
 	if (unlikely(wc->status != IB_WC_SUCCESS)) {
 		nvmet_rdma_rw_ctx_destroy(rsp);
-		nvmet_req_uninit(&rsp->req);
+		/*
+		 * Unlike in nvmet_rdma_read_data_done(), the request has been
+		 * completed and its references are put: no nvmet_req_uninit().
+		 */
 		nvmet_rdma_release_rsp(rsp);
 		if (wc->status != IB_WC_WR_FLUSH_ERR) {
 			pr_info("RDMA WRITE for CQE failed with status %s (%d).\n",
Signature · 1 line
2.56.0
author maintainer
#183 lore raw

PATCH 96/98 nvmet: take sq->ref in nvmet_sq_fatal_error()

In reply to Jens Axboe

From: Chris Mason <[email protected]>

nvmet_sq_destroy() kills sq->ref, waits for the commands to drop it, then puts the controller and clears sq->ctrl. So sq->ctrl may only be used while holding sq->ref. nvmet_sq_fatal_error() reads it without, and its transport callers hold nothing either: in nvmet-rdma the completion handlers run until the QP is drained, which nvmet_rdma_release_queue_work() does only after nvmet_sq_destroy(), and in nvmet-tcp io_work can run between the two cancels of the release work. A short capsule or a failed completion that races the disconnect then dereferences a NULL sq->ctrl, or locks ctrl->lock and queues fatal_err_work in a freed controller. The host needs nothing but a Connect, reproduced over rxe with a sleep added to widen the window.

Take sq->ref with percpu_ref_tryget_live() around the body. If the ref is already dead the queue is going away and the error is dropped, which is the mitigation: nothing can tell a call that would have finished in time from one that wouldn't. tryget_live after percpu_ref_exit() is fine, it sees __PERCPU_REF_DEAD and touches nothing freed, and nvmet_req_init() already relies on that. nvmet_auth_fatal_error() runs in a command that holds the ref, so it calls the body directly.

Fixes 14c862dbb0a0 ("nvmet-rdma: drain the queue-pair just before freeing it")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 drivers/nvme/target/core.c             | 21 ++++++++++++++++++++-
 drivers/nvme/target/fabrics-cmd-auth.c |  3 ++-
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 23 insertions(+), 2 deletions(-)
drivers/nvme/target/core.c +20 -1
index b78a4d4eb838..19f96f1b7e74 100644
@@ -1833,8 +1833,11 @@ EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);
  * A fatal error on one queue.  The Connect command of an I/O queue needs no
  * secret, so until the queue has authenticated it may be anybody's: do not
  * let it take down the controller of the host whose NQN it has named.
+ *
+ * The caller holds sq->ref, as a command that is being executed does: until it
+ * is put, nvmet_sq_destroy() waits, and does not clear sq->ctrl.
  */
-void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+void __nvmet_sq_fatal_error(struct nvmet_sq *sq)
 {
 	struct nvmet_ctrl *ctrl = sq->ctrl;
 
@@ -1843,6 +1846,22 @@ void nvmet_sq_fatal_error(struct nvmet_sq *sq)
 	else
 		nvmet_ctrl_fatal_error(ctrl);
 }
+
+/*
+ * A transport calls this with no reference of its own, concurrently with
+ * nvmet_sq_destroy(), which puts the controller and clears sq->ctrl.  As in
+ * nvmet_req_init(), the call can come after percpu_ref_exit():
+ * percpu_ref_tryget_live() then sees that the ref is dead and touches nothing
+ * that is freed.
+ */
+void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+{
+	if (!percpu_ref_tryget_live(&sq->ref))
+		return;
+
+	__nvmet_sq_fatal_error(sq);
+	percpu_ref_put(&sq->ref);
+}
 EXPORT_SYMBOL_GPL(nvmet_sq_fatal_error);
 
 ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
drivers/nvme/target/fabrics-cmd-auth.c +2 -1
index cb6be285b3c4..020a989fe263 100644
@@ -33,12 +33,13 @@ static void nvmet_auth_expired_work(struct work_struct *work)
 	mutex_unlock(&sq->auth_lock);
 }
 
+/* Called from a command that is being executed, which holds sq->ref. */
 static void nvmet_auth_fatal_error(struct nvmet_sq *sq, bool passed)
 {
 	if (passed)
 		nvmet_ctrl_fatal_error(sq->ctrl);
 	else
-		nvmet_sq_fatal_error(sq);
+		__nvmet_sq_fatal_error(sq);
 }
 
 void nvmet_auth_sq_init(struct nvmet_sq *sq)
drivers/nvme/target/nvmet.h +1 -0
index 29105e2a7022..3519537705f8 100644
@@ -631,6 +631,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);
 int nvmet_sq_init(struct nvmet_sq *sq, struct nvmet_cq *cq);
 
 void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl);
+void __nvmet_sq_fatal_error(struct nvmet_sq *sq);
 void nvmet_sq_fatal_error(struct nvmet_sq *sq);
 
 void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new);
Signature · 1 line
2.56.0