Various block related fixes
Subsystems (15)
Jens Axboe
maintainercover Various block related fixes
Hi,
Got a giant pile of stuff dumped on me, and went through and did some vetting and, particularly, commit message rewriting to get it to a somewhat comprehensible shape. If you're CC'ed on a patch, please review it. Feel free to queue it up, or reply back why the particular patch is garbage, if it is garbage.
Thanks!
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/98 blk-cgroup: keep the old blkg while reassociating a bio
From: Josef Bacik <[email protected]>
bio_associate_blkg_from_css() drops the bio's old blkg before it looks up the new one through the css. The css is pinned by the blkg, not by the bio, so if the cgroup is gone the bio may hold the last reference and the blkcg can be freed under blkg_tryget_closest(). The rcu_read_lock() around the whole thing used to cover that, until commit 457d3c4f0fdd dropped it.
Put the old blkg after the new one has been looked up.
--- block/blk-cgroup.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
block/blk-cgroup.c +5 -2
@@ -2139,8 +2139,7 @@ static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,void bio_associate_blkg_from_css (struct bio *bio,struct cgroup_subsys_state *css) { -if (bio->bi_blkg) -blkg_put (bio->bi_blkg); +struct blkcg_gq *old_blkg = bio->bi_blkg;if (css && css->parent) { bio->bi_blkg =blkg_tryget_closest (bio, css);
@@ -2148,6 +2147,10 @@ void bio_associate_blkg_from_css(struct bio *bio,blkg_get (bdev_get_queue (bio->bi_bdev)->root_blkg); bio->bi_blkg =bdev_get_queue (bio->bi_bdev)->root_blkg; } + +/* @old_blkg may be the only thing which pins @css */ +if (old_blkg) +blkg_put (old_blkg); }EXPORT_SYMBOL_GPL (bio_associate_blkg_from_css);
Signature · 1 line
2.56.0
Re: [PATCH 01/98] blk-cgroup: keep the old blkg while reassociating a bio
Quoted 7 lines · Jens Axboe
bio_associate_blkg_from_css()drops the bio's old blkg before it looks up the new one through the css. The css is pinned by the blkg, not by the bio, so if the cgroup is gone the bio may hold the last reference and the blkcg can be freed underblkg_tryget_closest(). Thercu_read_lock()around the whole thing used to cover that, until commit457d3c4f0fdddropped it.
Quoted 6 lines
Put the old blkg after the new one has been looked up.
Fixes 457d3c4f0fdd ("blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 2/98 blk-iocost: free struct ioc after a grace period
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
ioc_qos_prfill() and ioc_cost_model_prfill() take ioc->lock from blkcg_print_blkgs(), which since commit 56cc24f59c14 runs under RCU only. That covers the policy data, which is freed after a grace period, but not the ioc it points to: ioc_rqos_exit() frees that right away, and a concurrent read of io.cost.qos locks freed memory.
Free the ioc with kfree_rcu_mightsleep().
--- block/blk-iocost.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-iocost.c +2 -1
@@ -2920,7 +2920,8 @@ static void ioc_rqos_exit(struct rq_qos *rqos)timer_shutdown_sync (&ioc->timer);free_percpu (ioc->pcpu_stat); -kfree (ioc); +/* blkcg_print_blkgs() gets here from a pd with only RCU held */ +kfree_rcu_mightsleep (ioc); }static const struct rq_qos_ops ioc_rqos_ops = {
Signature · 1 line
2.56.0
Re: [PATCH 02/98] blk-iocost: free struct ioc after a grace period
Quoted 6 lines · Jens Axboe
ioc_qos_prfill()andioc_cost_model_prfill()takeioc->lockfromblkcg_print_blkgs(), which since commit56cc24f59c14runs under RCU only. That covers the policy data, which is freed after a grace period, but not the ioc it points to:ioc_rqos_exit()frees that right away, and a concurrent read of io.cost.qos locks freed memory.
Quoted 6 lines
Free the ioc with
kfree_rcu_mightsleep().Fixes 56cc24f59c14 ("blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 3/98 blk-throttle: don't truncate the cgroup v1 iops limits
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
tg_set_conf() parses the limit into a u64 and stores the iops ones through an unsigned int, so anything above UINT_MAX is truncated and a multiple of 2^32 becomes a limit of 0. The next IO of the group then divides by it in tg_within_iops_limit().
Clamp to UINT_MAX, which means no limit, like tg_set_limit() does on the default hierarchy.
--- block/blk-throttle.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-throttle.c +2 -1
@@ -1386,7 +1386,8 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of,if (is_u64) *(u64 *)((void *)tg +of_cft (of)->private) = v;else - *(unsigned int *)((void *)tg +of_cft (of)->private) = v; + *(unsigned int *)((void *)tg +of_cft (of)->private) = +min_t (u64, v,UINT_MAX );tg_conf_updated (tg, false); ret =0 ;
Signature · 1 line
2.56.0
Re: [PATCH 03/98] blk-throttle: don't truncate the cgroup v1 iops limits
Quoted 5 lines · Jens Axboe
tg_set_conf()parses the limit into a u64 and stores the iops ones through an unsigned int, so anything aboveUINT_MAXis truncated and a multiple of 2^32 becomes a limit of 0. The next IO of the group then divides by it intg_within_iops_limit().
Quoted 7 lines
Clamp to
UINT_MAX, which means no limit, liketg_set_limit()does on the default hierarchy.Fixes af133ceb2610 ("blkcg: move blkio_group_conf->iops and ->bps to blk-throttle")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 4/98 block: fix use-after-free of the disk in sync_bdevs()
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
sync_bdevs() holds a reference to the bdev inode while it takes bd_disk->open_mutex. For a partition the inode doesn't keep the gendisk alive, the device reference does: part_release() drops the disk as soon as the last reference to the partition device is gone. A sync(2) racing with the last close of a partition of an already deleted disk locks a freed open_mutex.
Take a device reference before touching the disk, like blkdev_get_no_open() does, and skip the bdev if it's on its way out.
--- block/bdev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
block/bdev.c +10 -0
@@ -1366,6 +1366,15 @@ void sync_bdevs(bool wait) old_inode = inode; bdev =I_BDEV (inode); +/* +* The inode does not keep the gendisk of a partition alive, the +* device does. +*/ +if (!kobject_get_unless_zero (&bdev->bd_device.kobj)) { +spin_lock (&blockdev_superblock->s_inode_list_lock); +continue ; + } +mutex_lock (&bdev->bd_disk->open_mutex);if (!atomic_read (&bdev->bd_openers)) { ;/* skip */
@@ -1381,6 +1390,7 @@ void sync_bdevs(bool wait)filemap_fdatawrite (inode->i_mapping); }mutex_unlock (&bdev->bd_disk->open_mutex); +put_device (&bdev->bd_device);spin_lock (&blockdev_superblock->s_inode_list_lock); }
Signature · 1 line
2.56.0
Re: [PATCH 04/98] block: fix use-after-free of the disk in sync_bdevs()
Quoted 11 lines · Jens Axboe
From: Josef Bacik <[email protected]>
sync_bdevs()holds a reference to the bdev inode while it takesbd_disk->open_mutex. For a partition the inode doesn't keep the gendisk alive, the device reference does:part_release()drops the disk as soon as the last reference to the partition device is gone. A sync(2) racing with the last close of a partition of an already deleted disk locks a freedopen_mutex.Take a device reference before touching the disk, like
blkdev_get_no_open()does, and skip the bdev if it's on its way out.
Looks good:
In the long run I wonder if we should find a way to add the bdevfs special cases to sync_inodes_sb and reuse the common syncing logic instead of this. But some of the special cases might be a bit to much.
Jens Axboe
maintainerPATCH 5/98 block: don't lose the status of a failed flush request
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
flush_end_io() stores the status in fq->rq_status for whoever drops the last reference to finish the flush sequence. blk_mq_put_rq_ref() calls it with BLK_STS_OK from tag iterators and the timeout handler and overwrites an error the real completion stored. With two extra references, reading /sys/block/<disk>/inflight is enough for that, a failed flush completes everything waiting on it with success and fsync() lies to the application.
Only store a status that is an error.
--- block/blk-flush.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
block/blk-flush.c +7 -1
@@ -212,7 +212,13 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq,spin_lock_irqsave (&fq->mq_flush_lock, flags);if (!req_ref_put_and_test (flush_rq)) { - fq->rq_status = error; +/* +* Remember the status of the flush for whoever drops the last +* reference. blk_mq_put_rq_ref() gets here with BLK_STS_OK and +* must not wipe it out. +*/ +if (error !=BLK_STS_OK ) + fq->rq_status = error;spin_unlock_irqrestore (&fq->mq_flush_lock, flags);return RQ_END_IO_NONE ; }
Signature · 1 line
2.56.0
Re: [PATCH 05/98] block: don't lose the status of a failed flush request
Looks good:
Jens Axboe
maintainerPATCH 6/98 blk-mq: don't batch allocate tags from an inactive hctx
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_mq_get_tag() checks BLK_MQ_S_INACTIVE after getting a tag and gives it back if the flag is set, so the caller retries on the hctx of the CPU it runs on now. The batched allocation for plugged submitters doesn't, so a task migrated off a dying CPU can fill plug->cached_rqs from a drained hctx, and those requests never complete.
Check the flag in blk_mq_get_tags() too and release the tags if it's set. The caller falls back to blk_mq_get_tag(), which fails the same way and starts over.
--- block/blk-mq-tag.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
block/blk-mq-tag.c +12 -0
@@ -139,6 +139,18 @@ unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags,return 0 ; ret =__sbitmap_queue_get_batch (bt, nr_tags, offset); *offset += tags->nr_reserved_tags; + +/* +* Give up this allocation if the hctx is inactive, like blk_mq_get_tag() +* does. The caller falls back to it and will retry on an active hctx. +*/ +if (ret &&unlikely (test_bit (BLK_MQ_S_INACTIVE , &data->hctx->state))) { +unsigned int i; + +for_each_set_bit (i, &ret,BITS_PER_LONG ) +blk_mq_put_tag (tags, data->ctx, *offset + i); +return 0 ; + }return ret; }
Signature · 1 line
2.56.0
Re: [PATCH 06/98] blk-mq: don't batch allocate tags from an inactive hctx
block/blk-mq-tag.c unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags, · line 145
Quoted 1 line · Jens Axboe
block/blk-mq-tag.c +1
-0+* does. The caller falls back to it and will retry on an active hctx.
it I guess is blk_mq_get_tag? Maybe spell that out.
Or just use the comment from blk_mq_tag which would apply here as well:
/*
* Give up this allocation if the hctx is inactive. The caller will
* retry on an active hctx.
*/Otherwise this looks fine.
Jens Axboe
maintainerPATCH 7/98 block: keep zone write plugs with plugged BIOs alive
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The issuer of a plugged zone write hears about the completion in bio_endio(), before blk_zone_write_plug_finish_request() unplugs the zone, and may reset the zone and write it again in between. Since commit 836efd35c472 that revives the plug the reset marked dead, with the new BIO on the list and wp_offset still 0. The unplug of the first write then sees an empty zone and marks the plug dead again, BIO and all. Once that BIO is done the plug is freed and the write pointer offset is lost.
Only remove the plug of an empty zone when no BIO is left on it.
--- block/blk-zoned.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
block/blk-zoned.c +8 -1
@@ -1812,7 +1812,14 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,else if (!bio_list_empty (&zwplug->bio_list))disk_zone_wplug_schedule_work (disk, zwplug); -if (!zwplug->wp_offset ||disk_zone_wplug_is_full (disk, zwplug)) +/* +* The issuer of the write was told about its completion before we got +* here. It may already have reset the zone and started to write it +* again, in which case the zone is empty but has BIOs plugged. The plug +* must stay alive to track them. +*/ +if ((!zwplug->wp_offset &&bio_list_empty (&zwplug->bio_list)) || +disk_zone_wplug_is_full (disk, zwplug))disk_mark_zone_wplug_dead (zwplug);spin_unlock_irqrestore (&zwplug->lock, flags);
Signature · 1 line
2.56.0
Re: [PATCH 07/98] block: keep zone write plugs with plugged BIOs alive
Quoted 5 lines · Jens Axboe
From: Josef Bacik <[email protected]>
The issuer of a plugged zone write hears about the completion in
bio_endio(), beforeblk_zone_write_plug_finish_request()unplugs the zone, and may reset the zone and write it again in between.
How are supposed to a zone reset and a write in the short race window?
Quoted 7 lines
Since commit
836efd35c472that revives the plug the reset marked dead, with the new BIO on the list andwp_offsetstill 0. The unplug of the first write then sees an empty zone and marks the plug dead again, BIO and all. Once that BIO is done the plug is freed and the write pointer offset is lost.Only remove the plug of an empty zone when no BIO is left on it.
I'm not sure this is the correct thing to do. It would seem more sensible to make the blk-mq path match the bio-based one and call into the write plug machinery before finishing the last bio in a request.
Jens Axboe
maintainerPATCH 8/98 block: keep the zone write plug of a failed write
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
When a write fails, blk_zone_write_plug_bio_endio() sets BLK_ZONE_WPLUG_NEED_WP_UPDATE as the write pointer offset is now wrong. disk_zone_wplug_unplug_bio() doesn't look at that flag, so a failed write that would have filled the zone makes it look full, the plug is freed and "full" is written to zones_cond[] for a partially written zone. Reports then show it as full and zone append emulation fails with unaligned writes, even after the zone report an error requires.
Leave the plug alone while BLK_ZONE_WPLUG_NEED_WP_UPDATE is set.
--- block/blk-zoned.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
block/blk-zoned.c +6 -2
@@ -1817,9 +1817,13 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,* here. It may already have reset the zone and started to write it * again, in which case the zone is empty but has BIOs plugged. The plug * must stay alive to track them. +* +* If the write failed, the write pointer offset says nothing about the +* zone. Keep the plug until the offset was updated. */ -if ((!zwplug->wp_offset &&bio_list_empty (&zwplug->bio_list)) || -disk_zone_wplug_is_full (disk, zwplug)) +if (!(zwplug->flags &BLK_ZONE_WPLUG_NEED_WP_UPDATE ) && + ((!zwplug->wp_offset &&bio_list_empty (&zwplug->bio_list)) || +disk_zone_wplug_is_full (disk, zwplug)))disk_mark_zone_wplug_dead (zwplug);spin_unlock_irqrestore (&zwplug->lock, flags);
Signature · 1 line
2.56.0
Re: [PATCH 08/98] block: keep the zone write plug of a failed write
Quoted 9 lines · Jens Axboe
From: Josef Bacik <[email protected]>
When a write fails,
blk_zone_write_plug_bio_endio()setsBLK_ZONE_WPLUG_NEED_WP_UPDATEas the write pointer offset is now wrong.disk_zone_wplug_unplug_bio()doesn't look at that flag, so a failed write that would have filled the zone makes it look full, the plug is freed and "full" is written to zones_cond[] for a partially written zone. Reports then show it as full and zone append emulation fails with unaligned writes, even after the zone report an error requires.
I can't parse the second half of the last sentence. Can you explain what is is supposed to mean?
Otherwise the change looks sensible.
Jens Axboe
maintainerPATCH 9/98 blk-mq: end the request if getting a keyslot fails
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio by hand and frees the request. By then blk_mq_bio_to_request() has done the IO accounting, which blk_mq_free_request() doesn't undo, so the in_flight counter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.
Use blk_mq_end_request() like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.
--- block/blk-mq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
block/blk-mq.c +3 -2
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio) ret =blk_crypto_rq_get_keyslot (rq);if (ret !=BLK_STS_OK ) { -bio_endio_status (bio, ret); -blk_mq_free_request (rq); +if (bio_zone_write_plugging (bio)) +blk_zone_write_plug_init_request (rq); +blk_mq_end_request (rq, ret);return ; }
Signature · 1 line
2.56.0
Re: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails
Adding Eric for the blk-crypto code.
Quoted 24 lines · Jens Axboe
From: Josef Bacik <[email protected]>
When
blk_crypto_rq_get_keyslot()fails,blk_mq_submit_bio()ends the bio by hand and frees the request. By thenblk_mq_bio_to_request()has done the IO accounting, whichblk_mq_free_request()doesn't undo, so thein_flightcounter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.Use
blk_mq_end_request()like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/blk-mq.c +0
-0
block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200
Quoted 8 lines
block/blk-mq.c +3
-2@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio) ret =blk_crypto_rq_get_keyslot (rq);if (ret !=BLK_STS_OK ) { -bio_endio_status (bio, ret); -blk_mq_free_request (rq); +if (bio_zone_write_plugging (bio)) +blk_zone_write_plug_init_request (rq); +blk_mq_end_request (rq, ret);
If blk_mq_end_request does the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder the blk_crypto_rq_get_keyslot and blk_zone_write_plug_init_request calls so that the extra copy of the blk_zone_write_plug_init_request call is not needed.
Re: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails
Quoted 28 lines · Christoph Hellwig
Adding Eric for the blk-crypto code.
Quoted 25 lines · Jens Axboe
From: Josef Bacik <[email protected]>
When
blk_crypto_rq_get_keyslot()fails,blk_mq_submit_bio()ends the bio by hand and frees the request. By thenblk_mq_bio_to_request()has done the IO accounting, whichblk_mq_free_request()doesn't undo, so thein_flightcounter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.Use
blk_mq_end_request()like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/blk-mq.c +0
-0@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
Quoted 13 lines
block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200
Quoted 7 lines
block/blk-mq.c +3
-2ret =blk_crypto_rq_get_keyslot (rq);if (ret !=BLK_STS_OK ) { -bio_endio_status (bio, ret); -blk_mq_free_request (rq); +if (bio_zone_write_plugging (bio)) +blk_zone_write_plug_init_request (rq); +blk_mq_end_request (rq, ret);If
blk_mq_end_requestdoes the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder theblk_crypto_rq_get_keyslotandblk_zone_write_plug_init_requestcalls so that the extra copy of theblk_zone_write_plug_init_requestcall is not needed.
I don't know the blk-mq code super well. But yes, blk_zone_write_plug_init_request() has no dependency on the keyslot; in particular the back-merging relies on crypt_ctx, not crypt_keyslot. So calling it before blk_crypto_rq_get_keyslot() would be simpler than duplicating it in the success and failure cases.
And yes, blk_mq_end_request() doesn't assume blk_crypto_rq_get_keyslot() has succeeded. On failure crypt_keyslot just remains NULL.
Josef and Jens, what are your expectations about this patch? Sending it as part of a 98-patch series (that didn't get picked up by Sashiko) is kind of awkward. If you need help I'd be glad to send a revised version as a standalone patch. Or you can handle it: up to you.
- Eric
Jens Axboe
maintainerRe: [PATCH 09/98] blk-mq: end the request if getting a keyslot fails
Quoted 49 lines · Eric Biggers
Quoted 28 lines · Christoph Hellwig
Adding Eric for the blk-crypto code.
Quoted 25 lines · Jens Axboe
From: Josef Bacik <[email protected]>
When
blk_crypto_rq_get_keyslot()fails,blk_mq_submit_bio()ends the bio by hand and frees the request. By thenblk_mq_bio_to_request()has done the IO accounting, whichblk_mq_free_request()doesn't undo, so thein_flightcounter never drops and iostat shows a device that is 100% busy. If the bio went through zone write plugging the zone stays plugged as well, and every later write to it waits forever.Use
blk_mq_end_request()like the dispatch path does for a request the driver rejected, after marking it for zone write plugging.Fixes a892c8d52c02 ("block: Inline encryption support for blk-mq")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/blk-mq.c +0
-0@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)Quoted 13 lines
block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3200
Quoted 7 lines
block/blk-mq.c +3
-2ret =blk_crypto_rq_get_keyslot (rq);if (ret !=BLK_STS_OK ) { -bio_endio_status (bio, ret); -blk_mq_free_request (rq); +if (bio_zone_write_plugging (bio)) +blk_zone_write_plug_init_request (rq); +blk_mq_end_request (rq, ret);If
blk_mq_end_requestdoes the right thing (it looks like that to me, but I'd rather rely on Eric and Jens for that), we should reorder theblk_crypto_rq_get_keyslotandblk_zone_write_plug_init_requestcalls so that the extra copy of theblk_zone_write_plug_init_requestcall is not needed.I don't know the blk-mq code super well. But yes,
blk_zone_write_plug_init_request()has no dependency on the keyslot; in particular the back-merging relies oncrypt_ctx, notcrypt_keyslot. So calling it beforeblk_crypto_rq_get_keyslot()would be simpler than duplicating it in the success and failure cases.
Quoted 8 lines
And yes,
blk_mq_end_request()doesn't assumeblk_crypto_rq_get_keyslot()has succeeded. On failurecrypt_keyslotjust remains NULL.Josef and Jens, what are your expectations about this patch? Sending it as part of a 98-patch series (that didn't get picked up by Sashiko) is kind of awkward. If you need help I'd be glad to send a revised version as a standalone patch. Or you can handle it: up to you.
I should have put this in the cover letter, that's my mistake. But I've replied to other patches in this thread, and the idea is "congrats, as the maintainer, you now own this bug" and you are responsible for fixing it. If the patch is good, feel free to ack it and run with it. If it needs work, do the work and send it out as any other patch you would. If you do that, feel free to rewrite it and claim authorship as well, nobody cares about that part.
So yes, would be great if you could send a revised patch in this case. Thanks!
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 10/98 block: don't unplug a zone under a failed write
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Since commit 552c1149af7a disk_zone_wplug_abort() clears BLK_ZONE_WPLUG_PLUGGED. For a failed write that happens in bio_endio(), before the request reaches blk_zone_write_plug_finish_request(). The issuer can reset the zone and submit a new write in between, which is issued unplugged, and when the old request is finished it clears the flag under that one, so the next write goes out alongside it.
Preserve the flag across the abort, the unplug of the failed write clears it. The exception is QUEUE_FLAG_ZONED_QD1_WRITES, where the abort takes the plug off the worker's list and nobody is left to clear it.
--- block/blk-zoned.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
block/blk-zoned.c +12 -1
@@ -1871,9 +1871,20 @@ void blk_zone_write_plug_bio_endio(struct bio *bio)* needing a write pointer update. */ if (bio->bi_status !=BLK_STS_OK ) { +unsigned int plugged =0 ; +spin_lock_irqsave (&zwplug->lock, flags); +/* +* The abort unplugs the zone. But the flag still belongs to a +* write in flight (this one, until its request is finished) or +* to the BIO work, and they clear it when they are done. The +* zone write plugs worker does not rely on the flag to issue +* one write at a time and forgets the plug with the abort. +*/ +if (!blk_queue_zoned_qd1_writes (disk->queue)) + plugged = zwplug->flags &BLK_ZONE_WPLUG_PLUGGED ;disk_zone_wplug_abort (zwplug); - zwplug->flags |=BLK_ZONE_WPLUG_NEED_WP_UPDATE ; + zwplug->flags |=BLK_ZONE_WPLUG_NEED_WP_UPDATE | plugged;spin_unlock_irqrestore (&zwplug->lock, flags); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 11/98 block: check dma alignment of bvec iters in async dio
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
__blkdev_direct_IO_async() uses bio_iov_iter_set() for an ITER_BVEC iterator and never gets to the alignment check in bio_iov_iter_get_pages(), which is the one that exists for io_uring registered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misaligned READ_FIXED from brd returns the wrong data, and dm-verity with panic_on_error panics.
Move bio_iov_bvec_aligned() to blk.h and check in the async path too, failing with -EINVAL like the other direct IO paths.
--- block/bio.c | 33 --------------------------------- block/blk.h | 33 +++++++++++++++++++++++++++++++++ block/fops.c | 10 ++++++++-- 3 files changed, 41 insertions(+), 35 deletions(-)
block/bio.c +0 -33
@@ -1272,39 +1272,6 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,return 0 ; } -#ifdef CONFIG_DEBUG_KERNEL -static inline bool bio_iov_bvec_aligned (const struct bio *bio, -unsigned mem_align_mask) -{ -struct bvec_iter iter; -struct bio_vec bv; - -/* -* Correct callers never break the alignment requirements, so this -* exhaustive check is only paid for in debug builds. -*/ -for_each_mp_bvec (bv, bio->bi_io_vec, iter, bio->bi_iter) -if ((bv.bv_offset | bv.bv_len) & mem_align_mask) -return false; -return true; -} -#else -static inline bool bio_iov_bvec_aligned (const struct bio *bio, -unsigned mem_align_mask) -{ -/* -* We forward the bio_vec as-is, so ITER_BVEC callers must provide -* segments already aligned to the device's DMA alignment. The only -* unchecked user-controllable offset that reaches here is an io_uring -* registered buffer where just the first segment can be unaligned -* (the rest is virtually contiguous), so checking only that one is -* sufficient to know if the entire vector is valid. -*/ -return !(mp_bvec_iter_offset (bio->bi_io_vec, bio->bi_iter) & - mem_align_mask); -} -#endif -/** * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to
block/blk.h +33 -0
@@ -202,6 +202,39 @@ static inline bool rq_mergeable(struct request *rq)return true; } +#ifdef CONFIG_DEBUG_KERNEL +static inline bool bio_iov_bvec_aligned (const struct bio *bio, +unsigned int mem_align_mask) +{ +struct bvec_iter iter; +struct bio_vec bv; + +/* +* Correct callers never break the alignment requirements, so this +* exhaustive check is only paid for in debug builds. +*/ +for_each_mp_bvec (bv, bio->bi_io_vec, iter, bio->bi_iter) +if ((bv.bv_offset | bv.bv_len) & mem_align_mask) +return false; +return true; +} +#else +static inline bool bio_iov_bvec_aligned (const struct bio *bio, +unsigned int mem_align_mask) +{ +/* +* We forward the bio_vec as-is, so ITER_BVEC callers must provide +* segments already aligned to the device's DMA alignment. The only +* unchecked user-controllable offset that reaches here is an io_uring +* registered buffer where just the first segment can be unaligned +* (the rest is virtually contiguous), so checking only that one is +* sufficient to know if the entire vector is valid. +*/ +return !(mp_bvec_iter_offset (bio->bi_io_vec, bio->bi_iter) & + mem_align_mask); +} +#endif +/* * There are two different ways to handle DISCARD merges: * 1) If max_discard_segments > 1, the driver treats every bio as a range and
block/fops.c +8 -2
@@ -365,9 +365,15 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,* Users don't rely on the iterator being in any particular * state for async I/O returning -EIOCBQUEUED, hence we can * avoid expensive iov_iter_advance(). Bypass -* bio_iov_iter_get_pages() and set the bvec directly. +* bio_iov_iter_get_pages() and set the bvec directly, which leaves +* its alignment check to us. */ -if (!bio_iov_iter_set (bio, iter)) { +if (bio_iov_iter_set (bio, iter)) { +if (!bio_iov_bvec_aligned (bio,bdev_dma_alignment (bdev))) { + ret = -EINVAL ; + goto out_bio_put; + } + }else { ret =blkdev_iov_iter_get_pages (bio, iter, bdev);if (unlikely (ret)) goto out_bio_put;
Signature · 1 line
2.56.0
Re: [PATCH 11/98] block: check dma alignment of bvec iters in async dio
Quoted 11 lines · Jens Axboe
From: Josef Bacik <[email protected]>
__blkdev_direct_IO_async()usesbio_iov_iter_set()for anITER_BVECiterator and never gets to the alignment check inbio_iov_iter_get_pages(), which is the one that exists forio_uringregistered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misalignedREAD_FIXEDfrom brd returns the wrong data, and dm-verity withpanic_on_errorpanics.Move
bio_iov_bvec_aligned()toblk.hand check in the async path too, failing with-EINVALlike the other direct IO paths.
I think the theory was that bvec callers have to validate the offsets themselves. It looks like that doesn't always work? Should we fix that in the callers or do we decide we're unlikely to get all of them right and do the checks unconditionally?
Quoted 123 lines
Fixes 5ff3f74e145a ("block: simplify direct io validity check")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/bio.c | 33 --------------------------------- block/blk.h | 33 +++++++++++++++++++++++++++++++++ block/fops.c | 10 ++++++++-- 3 files changed, 41 insertions(+), 35 deletions(-)block/bio.c +0
-33@@ -1272,39 +1272,6 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,return 0 ; } -#ifdef CONFIG_DEBUG_KERNEL -static inline bool bio_iov_bvec_aligned (const struct bio *bio, -unsigned mem_align_mask) -{ -struct bvec_iter iter; -struct bio_vec bv; - -/* -* Correct callers never break the alignment requirements, so this -* exhaustive check is only paid for in debug builds. -*/ -for_each_mp_bvec (bv, bio->bi_io_vec, iter, bio->bi_iter) -if ((bv.bv_offset | bv.bv_len) & mem_align_mask) -return false; -return true; -} -#else -static inline bool bio_iov_bvec_aligned (const struct bio *bio, -unsigned mem_align_mask) -{ -/* -* We forward the bio_vec as-is, so ITER_BVEC callers must provide -* segments already aligned to the device's DMA alignment. The only -* unchecked user-controllable offset that reaches here is an io_uring -* registered buffer where just the first segment can be unaligned -* (the rest is virtually contiguous), so checking only that one is -* sufficient to know if the entire vector is valid. -*/ -return !(mp_bvec_iter_offset (bio->bi_io_vec, bio->bi_iter) & - mem_align_mask); -} -#endif -/** * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to block/blk.h +33
-0@@ -202,6 +202,39 @@ static inline bool rq_mergeable(struct request *rq)return true; } +#ifdef CONFIG_DEBUG_KERNEL +static inline bool bio_iov_bvec_aligned (const struct bio *bio, +unsigned int mem_align_mask) +{ +struct bvec_iter iter; +struct bio_vec bv; + +/* +* Correct callers never break the alignment requirements, so this +* exhaustive check is only paid for in debug builds. +*/ +for_each_mp_bvec (bv, bio->bi_io_vec, iter, bio->bi_iter) +if ((bv.bv_offset | bv.bv_len) & mem_align_mask) +return false; +return true; +} +#else +static inline bool bio_iov_bvec_aligned (const struct bio *bio, +unsigned int mem_align_mask) +{ +/* +* We forward the bio_vec as-is, so ITER_BVEC callers must provide +* segments already aligned to the device's DMA alignment. The only +* unchecked user-controllable offset that reaches here is an io_uring +* registered buffer where just the first segment can be unaligned +* (the rest is virtually contiguous), so checking only that one is +* sufficient to know if the entire vector is valid. +*/ +return !(mp_bvec_iter_offset (bio->bi_io_vec, bio->bi_iter) & + mem_align_mask); +} +#endif +/* * There are two different ways to handle DISCARD merges: * 1) If max_discard_segments > 1, the driver treats every bio as a range and block/fops.c +8
-2@@ -365,9 +365,15 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,* Users don't rely on the iterator being in any particular * state for async I/O returning -EIOCBQUEUED, hence we can * avoid expensive iov_iter_advance(). Bypass -* bio_iov_iter_get_pages() and set the bvec directly. +* bio_iov_iter_get_pages() and set the bvec directly, which leaves +* its alignment check to us. */ -if (!bio_iov_iter_set (bio, iter)) { +if (bio_iov_iter_set (bio, iter)) { +if (!bio_iov_bvec_aligned (bio,bdev_dma_alignment (bdev))) { + ret = -EINVAL ; + goto out_bio_put; + } + }else { ret =blkdev_iov_iter_get_pages (bio, iter, bdev);if (unlikely (ret)) goto out_bio_put;Signature · 1 line
2.56.0
---end quoted text---
Keith Busch
maintainerRe: [PATCH 11/98] block: check dma alignment of bvec iters in async dio
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Jens Axboe
From: Josef Bacik <[email protected]>
__blkdev_direct_IO_async()usesbio_iov_iter_set()for anITER_BVECiterator and never gets to the alignment check inbio_iov_iter_get_pages(), which is the one that exists forio_uringregistered buffers in the first place. A bio based driver has nobody else to enforce its DMA alignment: a misalignedREAD_FIXEDfrom brd returns the wrong data, and dm-verity withpanic_on_errorpanics.
Quoted 7 lines
Quoted 2 lines
Move
bio_iov_bvec_aligned()toblk.hand check in the async path too, failing with-EINVALlike the other direct IO paths.I think the theory was that bvec callers have to validate the offsets themselves. It looks like that doesn't always work? Should we fix that in the callers or do we decide we're unlikely to get all of them right and do the checks unconditionally?
Yes that was the idea. Not that that's happening here with vectored registered buffer usage, so the patch is correct as far as that goes. It's a bit of a bummer as the point was to avoid repeated vector walks, and I don't think we can efficiently validate while importing the vector.
The block layer will still catch the invalid alignment, but further down the stack, and I've learned the hard way that some mappers just are not handling these kinds of errors that late in the submission path.
Jens Axboe
maintainerPATCH 12/98 drbd: stop the start_resync timer of a dying device
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
start_resync_timer gets armed for up to a second and nothing ever deletes it, so drbd_destroy_device() can free the device with the timer pending. Shut it down where request_timer is shut down.
--- drivers/block/drbd/drbd_main.c | 1 + 1 file changed, 1 insertion(+)
drivers/block/drbd/drbd_main.c +1 -0
@@ -2163,6 +2163,7 @@ void drbd_destroy_device(struct kref *kref)struct drbd_peer_device *peer_device, *tmp_peer_device;timer_shutdown_sync (&device->request_timer); +timer_shutdown_sync (&device->start_resync_timer);/* paranoia asserts */ D_ASSERT (device, device->open_cnt ==0 );
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 13/98 blk-cgroup: fix root io.stat byte counts on 32-bit
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blkcg_fill_root_iostats() shifts the unsigned long sector counts by 9 before adding them to the u64 byte counts, so on 32-bit the contribution of a CPU wraps every 4GB. Widen before the shift.
--- block/blk-cgroup.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
block/blk-cgroup.c +3 -3
@@ -1118,11 +1118,11 @@ static void blkcg_fill_root_iostats(void) cpu_dkstats->ios[STAT_DISCARD ];// convert sectors to bytes tmp.bytes[BLKG_IOSTAT_READ ] += - cpu_dkstats->sectors[STAT_READ ] <<9 ; + (u64 )cpu_dkstats->sectors[STAT_READ ] <<9 ; tmp.bytes[BLKG_IOSTAT_WRITE ] += - cpu_dkstats->sectors[STAT_WRITE ] <<9 ; + (u64 )cpu_dkstats->sectors[STAT_WRITE ] <<9 ; tmp.bytes[BLKG_IOSTAT_DISCARD ] += - cpu_dkstats->sectors[STAT_DISCARD ] <<9 ; + (u64 )cpu_dkstats->sectors[STAT_DISCARD ] <<9 ; } flags =u64_stats_update_begin_irqsave (&blkg->iostat.sync);
Signature · 1 line
2.56.0
Re: [PATCH 13/98] blk-cgroup: fix root io.stat byte counts on 32-bit
Quoted 8 lines · Jens Axboe
blkcg_fill_root_iostats()shifts the unsigned long sector counts by 9 before adding them to the u64 byte counts, so on 32-bit the contribution of a CPU wraps every 4GB. Widen before the shift.Fixes ef45fe470e1e ("blk-cgroup: show global disk stats in root cgroup io.stat")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 14/98 drbd: take req_lock to skip the initial sync as peer
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
_drbd_set_state() must be called with req_lock held. drbd_adm_new_c_uuid() does, receive_uuids() on the peer doesn't, so a concurrent state change gets lost: the device is Secondary again, or its failed disk UpToDate, while the work for the lost change is already queued.
--- drivers/block/drbd/drbd_receiver.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/block/drbd/drbd_receiver.c +2 -0
@@ -4096,8 +4096,10 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info BM_LOCKED_TEST_ALLOWED, NULL);_drbd_uuid_set (device,UI_CURRENT , p_uuid[UI_CURRENT ]);_drbd_uuid_set (device,UI_BITMAP ,0 ); +spin_lock_irq (&device->resource->req_lock);_drbd_set_state (_NS2 (device, disk,D_UP_TO_DATE , pdsk,D_UP_TO_DATE ),CS_VERBOSE ,NULL ); +spin_unlock_irq (&device->resource->req_lock);drbd_md_sync (device); updated_uuids =1 ; }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 15/98 zloop: refuse zone files without read/write_iter
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
zloop_do_rw() calls ->read_iter() and ->write_iter() on the zone files without checking that they exist. A base_dir on hugetlbfs, or zone files that are links into procfs, create a device just fine and the first IO jumps to NULL. Check for both when a zone file is opened, like loop does for its backing file.
--- drivers/block/zloop.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/block/zloop.c +10 -0
@@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,return ERR_PTR (-ENOMEM ); file =filp_open (p, oflags, mode);kfree (p); +if (IS_ERR (file)) +return file; + +/* Zone files are accessed with ->read_iter() and ->write_iter(). */ +if ((oflags &O_ACCMODE ) ==O_RDWR && + (!file->f_op->read_iter || !file->f_op->write_iter)) { +fput (file); +return ERR_PTR (-EINVAL ); + } +return file; }
Signature · 1 line
2.56.0
Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter
Quoted 7 lines · Jens Axboe
From: Josef Bacik <[email protected]>
zloop_do_rw()calls ->read_iter() and ->write_iter() on the zone files without checking that they exist. Abase_diron hugetlbfs, or zone files that are links into procfs, create a device just fine and the first IO jumps to NULL. Check for both when a zone file is opened, like loop does for its backing file.
Not particularly against the patch, but I don't think you can have a zloop
zone file be a link into procfs. Well you can if you create the zloop, tear it
down again, then rm a zone file and re-create it as a link into proc. So I guess:
Quoted 14 lines
Fixes eb0570c7df23 ("block: new zoned loop block device driver")Cc Damien Le Moal <[email protected]>Cc Christoph Hellwig <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/block/zloop.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/block/zloop.c +0
-0
drivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063
Quoted 8 lines
drivers/block/zloop.c +5
-0@@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,return ERR_PTR (-ENOMEM ); file =filp_open (p, oflags, mode);kfree (p); +if (IS_ERR (file)) +return file; + +/* Zone files are accessed with ->read_iter() and ->write_iter(). */ + if ((oflags & O_ACCMODE) == O_RDWR &&
I think you can drop the O_RDWR check, the zone files are always opened O_RDWR,
only zloop_dev_exists() passes O_RDONLY to zloop_filp_open_fmt() but that is just to
check if the files exist and they're fput() right after it.
Quoted 8 lines
drivers/block/zloop.c +5
-0+ (!file->f_op->read_iter || !file->f_op->write_iter)) { +fput (file); +return ERR_PTR (-EINVAL ); + } +return file; }
Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter
In reply to Jens Axboedrivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063
Quoted 4 lines · Jens Axboe
drivers/block/zloop.c +4
-0+return file; + +/* Zone files are accessed with ->read_iter() and ->write_iter(). */ + if ((oflags & O_ACCMODE) == O_RDWR &&
I don't think this checks makes any sense.
The proper fix here is to copy and paste loop_check_backing_file from loop. I an prepare a patch for that.
Jens Axboe
maintainerRe: [PATCH 15/98] zloop: refuse zone files without read/write_iter
Quoted 10 lines · Christoph Hellwig
drivers/block/zloop.c static struct file *zloop_filp_open_fmt(int oflags, umode_t mode, · line 1063
Quoted 4 lines · Jens Axboe
drivers/block/zloop.c +4
-0+return file; + +/* Zone files are accessed with ->read_iter() and ->write_iter(). */ + if ((oflags & O_ACCMODE) == O_RDWR &&I don't think this checks makes any sense.
The proper fix here is to copy and paste
loop_check_backing_filefrom loop. I an prepare a patch for that.
Perfect, I'll drop this one.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 16/98 block/rnbd-srv: don't free the device on a sysfs error
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
rnbd_srv_create_dev_sysfs() drops the only kobject reference on its error paths, and the release function frees the rnbd_srv_dev. process_msg_open() keeps using it and ends in destroy_device() with a list_del() and a second kfree(). This doesn't need memory pressure: a client opening a device while its last mapping goes away gets -EEXIST from kobject_add() against the old directory.
Only undo kobject_add() on failure and leave the reference to destroy_device(). Initialise the kobject once, the next open retries.
--- drivers/block/rnbd/rnbd-srv-sysfs.c | 16 ++++++++++------ drivers/block/rnbd/rnbd-srv.c | 3 +++ 2 files changed, 13 insertions(+), 6 deletions(-)
drivers/block/rnbd/rnbd-srv-sysfs.c +10 -6
@@ -44,12 +44,17 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,struct kobject *bdev_kobj;int ret; - ret =kobject_init_and_add (&dev->dev_kobj, &dev_ktype, - rnbd_devs_kobj,"%pg" , bdev); -if (ret) { -kobject_put (&dev->dev_kobj); +/* +* The release function frees the device, which the caller goes on +* using and which others may share: never drop the reference of the +* kobject here. That is for destroy_device(). +*/ +if (!dev->dev_kobj.state_initialized) +kobject_init (&dev->dev_kobj, &dev_ktype); + + ret =kobject_add (&dev->dev_kobj, rnbd_devs_kobj,"%pg" , bdev); +if (ret)return ret; - } dev->dev_sessions_kobj =kobject_create_and_add ("sessions" , &dev->dev_kobj);
@@ -69,7 +74,6 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,kobject_put (dev->dev_sessions_kobj); free_dev_kobj:kobject_del (&dev->dev_kobj); -kobject_put (&dev->dev_kobj);return ret; }
drivers/block/rnbd/rnbd-srv.c +3 -0
@@ -208,6 +208,9 @@ static void destroy_device(struct kref *kref)* Destroy kobj only if it was really created. */ rnbd_srv_destroy_dev_sysfs (dev); + else if (dev->dev_kobj.state_initialized) + /* The files could not be created. Frees the name, too. */ + kobject_put(&dev->dev_kobj); elsekfree (dev ); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 17/98 block/rnbd-srv: drop the sess_dev kobject unlocked
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
rnbd_srv_create_dev_session_sysfs() drops the kobject reference when it fails. The release function is rnbd_destroy_sess_dev(), which takes srv_dev->lock, and the caller holds that lock: the worker deadlocks on itself. A client opening one device as "ram0" and "./ram0" gets there through -EEXIST from kobject_add().
Have process_msg_open() drop the reference after releasing the lock, and leave the rest of the cleanup to the release function.
--- drivers/block/rnbd/rnbd-srv-sysfs.c | 12 ++++++------ drivers/block/rnbd/rnbd-srv.c | 8 +++++++- 2 files changed, 13 insertions(+), 7 deletions(-)
drivers/block/rnbd/rnbd-srv-sysfs.c +6 -6
@@ -193,6 +193,10 @@ static const struct kobj_type rnbd_srv_sess_dev_ktype = { .release = rnbd_srv_sess_dev_release, }; +/* +* On failure the caller has to drop the reference of the kobject. The release +* function takes the lock of the device, which the caller holds here. +*/ int rnbd_srv_create_dev_session_sysfs (struct rnbd_srv_sess_dev *sess_dev) {int ret;
@@ -200,17 +204,13 @@ int rnbd_srv_create_dev_session_sysfs(struct rnbd_srv_sess_dev *sess_dev) ret =kobject_init_and_add (&sess_dev->kobj, &rnbd_srv_sess_dev_ktype, sess_dev->dev->dev_sessions_kobj,"%s" , sess_dev->sess->sessname); -if (ret) { -kobject_put (&sess_dev->kobj); +if (ret)return ret; - } ret =sysfs_create_group (&sess_dev->kobj, &rnbd_srv_default_dev_session_attr_group); -if (ret) { +if (ret)kobject_del (&sess_dev->kobj); -kobject_put (&sess_dev->kobj); - }return ret; }
drivers/block/rnbd/rnbd-srv.c +7 -1
@@ -590,6 +590,7 @@ rnbd_srv_create_set_sess_dev(struct rnbd_srv_session *srv_sess,return sdev;kref_init (&sdev->kref); +INIT_LIST_HEAD (&sdev->dev_list);strscpy (sdev->pathname, open_msg->dev_name,sizeof (sdev->pathname));
@@ -789,7 +790,12 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,rnbd_srv_err (srv_sess_dev,"Opening device failed, failed to create dev client sysfs files, err: %d\n" , ret); - goto free_srv_sess_dev; +/* +* rnbd_destroy_sess_dev() undoes everything down to the open +* of the block device. +*/ +kobject_put (&srv_sess_dev->kobj); + goto free_path; }list_add (&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 18/98 aoe: don't free a device with its work queued
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
flush() picks the devices without pending work and frees them, but since aoedev_downdev() may sleep, the check and the DEVFL_TKILL marking are no longer one step under the lock. A reply to an ATA IDENTIFY in between queues d->work, and the device is freed with it queued.
Set DEVFL_TKILL under d->lock, don't queue the work for a device that has it, and cancel the work in freedev().
--- drivers/block/aoe/aoecmd.c | 2 +- drivers/block/aoe/aoedev.c | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-)
drivers/block/aoe/aoecmd.c +1 -1
@@ -974,7 +974,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id) d->fw_ver, (long long)ssize); d->ssize = ssize; d->geo.start =0 ; -if (d->flags & (DEVFL_GDALLOC |DEVFL_NEWSIZE )) +if (d->flags & (DEVFL_GDALLOC |DEVFL_NEWSIZE |DEVFL_TKILL ))return ;if (d->gd !=NULL ) d->flags |=DEVFL_NEWSIZE ;
drivers/block/aoe/aoedev.c +4 -0
@@ -286,6 +286,8 @@ freedev(struct aoedev *d)return ;timer_delete_sync (&d->timer); +/* With DEVFL_TKILL set only the work itself still queues it */ +cancel_work_sync (&d->work);if (d->gd) {aoedisk_rm_debugfs (d);del_gendisk (d->gd);
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)spin_unlock (&d->lock);spin_unlock_irqrestore (&devlist_lock, flags);aoedev_downdev (d); +spin_lock_irqsave (&d->lock, flags); d->flags |=DEVFL_TKILL ; +spin_unlock_irqrestore (&d->lock, flags); goto restart1; cont:spin_unlock (&d->lock);
Signature · 1 line
2.56.0
Re: [PATCH 18/98] aoe: don't free a device with its work queued
drivers/block/aoe/aoedev.c flush(const char __user *str, size_t cnt, int exiting) · line 364
Quoted 8 lines · Jens Axboe
drivers/block/aoe/aoedev.c +2
-0@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)spin_unlock (&d->lock);spin_unlock_irqrestore (&devlist_lock, flags);aoedev_downdev (d); +spin_lock_irqsave (&d->lock, flags); d->flags |=DEVFL_TKILL ; +spin_unlock_irqrestore (&d->lock, flags); goto restart1;
Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race window open while aoedev_downdev(d) drops d->lock and sleeps in blk_mq_freeze_queue(d->blkq):
1. An incoming ATA IDENTIFY reply processed by ataid_complete() during that window still sees !(d->flags & DEVFL_TKILL), sets DEVFL_NEWSIZE (or DEVFL_GDALLOC), and queues d->work.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs before freedev() reaches cancel_work_sync(&d->work), it calls aoeblk_gdalloc() or sets d->flags |= DEVFL_UP again after aoedev_downdev() cleared DEVFL_UP, allowing new requests to be queued onto d and hanging del_gendisk() in freedev().
Should d->flags |= DEVFL_TKILL be set under d->lock before unlocking and calling aoedev_downdev(d) (which also prevents two concurrent aoedev_flush() callers from running aoedev_downdev() on the same device), and should aoecmd_sleepwork() bail out if DEVFL_TKILL is set in case d->work was queued between flush_workqueue(aoe_wq) and acquiring d->lock?
Re: [PATCH 18/98] aoe: don't free a device with its work queued
In reply to Jens AxboeReview flush(const char __user *str, size_t cnt, int exiting) · line 364
Quoted 8 lines · Jens Axboe
(quoted diff) +2
-0@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting) spin_unlock(&d->lock); spin_unlock_irqrestore(&devlist_lock, flags); aoedev_downdev(d); + spin_lock_irqsave(&d->lock, flags); d->flags |= DEVFL_TKILL; + spin_unlock_irqrestore(&d->lock, flags); goto restart1;
Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race window open while aoedev_downdev(d) drops d->lock and sleeps in blk_mq_freeze_queue(d->blkq):
1. ataid_complete(), rexmit_timer(), or aoeblk_gdalloc()'s error path can still see !(d->flags & DEVFL_TKILL) and queue d->work during that window.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs before freedev() reaches cancel_work_sync(&d->work), it can call aoeblk_gdalloc() (hitting WARN_ON(d->flags & DEVFL_TKILL) if DEVFL_TKILL was set in the meantime) or set d->flags |= DEVFL_UP again after aoedev_downdev() cleared DEVFL_UP, letting new I/O enter and hanging del_gendisk() in freedev().
Instead of setting DEVFL_TKILL after aoedev_downdev(d), how about:
- Serializing flush() with a static flush_mutex so a second aoedev_flush() caller cannot enter pass 2 (freedev()) and pass 3 (kfree()) on a device while the first caller is still in pass 1.
- Setting d->flags |= DEVFL_TKILL under d->lock before unlocking, then running cancel_work_sync(&d->work) before aoedev_downdev(d) (and keeping cancel_work_sync(&d->work) after timer_delete_sync(&d->timer) in freedev()).
- Checking DEVFL_TKILL at all three queue_work(aoe_wq, &d->work) sites (ataid_complete(), rexmit_timer(), and aoeblk_gdalloc()), as well as in aoecmd_sleepwork() and aoeblk_gdalloc().
Here is a replacement patch against the same baseline (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
drivers/block/aoe/aoeblk.c +6 -2
@@ -385,7 +385,10 @@ aoeblk_gdalloc(void *vp)spin_lock_irqsave (&d->lock, flags);WARN_ON (!(d->flags &DEVFL_GD_NOW ));WARN_ON (!(d->flags &DEVFL_GDALLOC )); -WARN_ON (d->flags &DEVFL_TKILL ); +if (d->flags &DEVFL_TKILL ) { +spin_unlock_irqrestore (&d->lock, flags); + goto out_disk_cleanup; + }WARN_ON (d->gd);WARN_ON (d->flags &DEVFL_UP ); d->bufpool = mp;
@@ -427,7 +430,8 @@ err_mempool: err:spin_lock_irqsave (&d->lock, flags); d->flags &= ~DEVFL_GD_NOW ; -queue_work (aoe_wq, &d->work); +if (!(d->flags &DEVFL_TKILL )) +queue_work (aoe_wq, &d->work);spin_unlock_irqrestore (&d->lock, flags); }
drivers/block/aoe/aoecmd.c +12 -3
@@ -787,7 +787,8 @@ rexmit_timer(struct timer_list *timer)*/ list_splice (&flist, &d->factive[0 ]); d->flags |=DEVFL_DEAD ; -queue_work (aoe_wq, &d->work); +if (!(d->flags &DEVFL_TKILL )) +queue_work (aoe_wq, &d->work); goto out; }
@@ -899,6 +900,13 @@ aoecmd_sleepwork(struct work_struct *work) { struct aoedev *d = container_of(work, struct aoedev, work); +spin_lock_irq (&d->lock); +if (d->flags &DEVFL_TKILL ) { +spin_unlock_irq (&d->lock); +return ; + } +spin_unlock_irq (&d->lock); +if (d->flags &DEVFL_DEAD )aoedev_downdev (d);
@@ -909,7 +917,8 @@ aoecmd_sleepwork(struct work_struct *work)set_capacity_and_notify (d->gd, d->ssize);spin_lock_irq (&d->lock); - d->flags |=DEVFL_UP ; +if (!(d->flags &DEVFL_TKILL )) + d->flags |=DEVFL_UP ; d->flags &= ~DEVFL_NEWSIZE ;spin_unlock_irq (&d->lock); }
@@ -974,7 +983,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id) d->fw_ver, (long long)ssize); d->ssize = ssize; d->geo.start =0 ; -if (d->flags & (DEVFL_GDALLOC |DEVFL_NEWSIZE )) +if (d->flags & (DEVFL_GDALLOC |DEVFL_NEWSIZE |DEVFL_TKILL ))return ;if (d->gd !=NULL ) d->flags |=DEVFL_NEWSIZE ;
drivers/block/aoe/aoedev.c +7 -1
@@ -286,6 +286,7 @@ freedev(struct aoedev *d)return ;timer_delete_sync (&d->timer); +cancel_work_sync (&d->work);if (d->gd) {aoedisk_rm_debugfs (d);del_gendisk (d->gd);
@@ -311,6 +312,8 @@ enum flush_parms { EXITING = 1, }; +static DEFINE_MUTEX (flush_mutex); +static int flush (const char __user *str,size_t cnt,int exiting) {
@@ -333,6 +336,7 @@ flush(const char __user *str, size_t cnt, int exiting) specified =1 ; } +mutex_lock (&flush_mutex);flush_workqueue (aoe_wq);/* pass one: do aoedev_downdev, which might sleep */ restart1:
@@ -353,10 +357,11 @@ restart1: || d->ref) goto cont; + d->flags |=DEVFL_TKILL ;spin_unlock (&d->lock);spin_unlock_irqrestore (&devlist_lock, flags); +cancel_work_sync (&d->work);aoedev_downdev (d); - d->flags |=DEVFL_TKILL ; goto restart1; cont:spin_unlock (&d->lock);
@@ -397,6 +402,7 @@ restart2:kfree (doomed); }spin_unlock_irqrestore (&devlist_lock, flags); +mutex_unlock (&flush_mutex);return 0 ; }
Jens Axboe
maintainerRe: [PATCH 18/98] aoe: don't free a device with its work queued
Quoted 37 lines · Bart Van Assche
drivers/block/aoe/aoedev.c flush(const char __user *str, size_t cnt, int exiting) · line 364
Quoted 8 lines · Jens Axboe
drivers/block/aoe/aoedev.c +2
-0@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)spin_unlock (&d->lock);spin_unlock_irqrestore (&devlist_lock, flags);aoedev_downdev (d); +spin_lock_irqsave (&d->lock, flags); d->flags |=DEVFL_TKILL ; +spin_unlock_irqrestore (&d->lock, flags); goto restart1;Setting
DEVFL_TKILLonly afteraoedev_downdev(d)returns leaves a race window open whileaoedev_downdev(d)dropsd->lockand sleeps inblk_mq_freeze_queue(d->blkq):1.
ataid_complete(),rexmit_timer(), or aoeblk_gdalloc()'s error path can still see !(d->flags &DEVFL_TKILL) and queued->workduring that window.
2.aoecmd_sleepwork()does not checkDEVFL_TKILLeither. If it runs beforefreedev()reachescancel_work_sync(&d->work), it can callaoeblk_gdalloc()(hittingWARN_ON(d->flags & DEVFL_TKILL)ifDEVFL_TKILLwas set in the meantime) or setd->flags|=DEVFL_UPagain afteraoedev_downdev()clearedDEVFL_UP, letting new I/O enter and hangingdel_gendisk()infreedev().Instead of setting
DEVFL_TKILLafteraoedev_downdev(d), how about:- Serializing
flush()with a staticflush_mutexso a secondaoedev_flush()caller cannot enter pass 2 (freedev()) and pass 3 (kfree()) on a device while the first caller is still in pass 1.
- Settingd->flags|=DEVFL_TKILLunderd->lockbefore unlocking, then runningcancel_work_sync(&d->work)beforeaoedev_downdev(d)(and keepingcancel_work_sync(&d->work)aftertimer_delete_sync(&d->timer)infreedev()).
- CheckingDEVFL_TKILLat all threequeue_work(aoe_wq, &d->work)sites (ataid_complete(),rexmit_timer(), andaoeblk_gdalloc()), as well as inaoecmd_sleepwork()andaoeblk_gdalloc().
Quoted 2 lines
Here is a replacement patch against the same baseline (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.
Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.
Signature · 1 line
Jens Axboe
Re: [PATCH 18/98] aoe: don't free a device with its work queued
Quoted 5 lines · Jens Axboe
Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.
Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.
Done - see also https://lore.kernel.org/linux-block/[email protected]/.
As mentioned in the description of that patch, it has only been compile-tested.
Bart.
Jens Axboe
maintainerRe: [PATCH 18/98] aoe: don't free a device with its work queued
Quoted 11 lines · Bart Van Assche
Quoted 5 lines · Jens Axboe
Great thanks Bart, but can you make it an _actual_ patch? I can't apply this, the commit message is referential to the original patch.
Just send me drop-in replacements, and I'll replace whatever gunk is in my tree now.
Done - see also https://lore.kernel.org/linux-block/[email protected]/.
As mentioned in the description of that patch, it has only been compile-tested.
I would strongly believe that was also the extent of testing that happened on the previous patch it replaces, so not a big deal. It's not like any of us use aoe...
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 19/98 blk-cgroup: don't free a blkg with its punt work queued
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blkg_async_bio_workfn() relies on the bios on the list pinning the blkg. But blkcg_punt_bio_submit() can call queue_work() after a running instance already took and completed the bio, and queue_work() succeeds on an empty list for the same reason. Nothing holds the blkg then, and free_work shares its memory with async_bio_work on top of that.
Hold a reference across blkcg_punt_bio_submit(), give the two works memory of their own, and cancel async_bio_work in blkg_free_workfn().
--- block/blk-cgroup.c | 13 ++++++++++++- block/blk-cgroup.h | 6 ++---- 2 files changed, 14 insertions(+), 5 deletions(-)
block/blk-cgroup.c +12 -1
@@ -121,6 +121,11 @@ static void blkg_free_workfn(struct work_struct *work)struct request_queue *q = blkg->q;int i; +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO +/* It may have been queued for bios which an earlier run has taken */ +cancel_work_sync (&blkg->async_bio_work); +#endif +/* * pd_free_fn() can also be called from blkcg_deactivate_policy(), * in order to make sure pd_free_fn() is called in order, the deletion
@@ -218,7 +223,7 @@ static void blkg_async_bio_workfn(struct work_struct *work)struct blk_plug plug;bool need_plug = false; -/* as long as there are pending bios, @blkg can't go away */ +/* blkg_free_workfn() waits for us */ spin_lock (&blkg->async_bio_lock);bio_list_merge_init (&bios, &blkg->async_bios);spin_unlock (&blkg->async_bio_lock);
@@ -245,10 +250,16 @@ void blkcg_punt_bio_submit(struct bio *bio)struct blkcg_gq *blkg = bio->bi_blkg;if (blkg->parent) { +/* +* Once it is on the list, a run which is under way can submit +* @bio, and its reference goes when it completes. +*/ +blkg_get (blkg);spin_lock (&blkg->async_bio_lock);bio_list_add (&blkg->async_bios, bio);spin_unlock (&blkg->async_bio_lock);queue_work (blkcg_punt_bio_wq, &blkg->async_bio_work); +blkg_put (blkg); }else {/* never bounce for the root cgroup */ submit_bio (bio);
block/blk-cgroup.h +2 -4
@@ -76,11 +76,9 @@ struct blkcg_gq {#ifdef CONFIG_BLK_CGROUP_PUNT_BIO spinlock_t async_bio_lock;struct bio_list async_bios; +struct work_struct async_bio_work;#endif -union { -struct work_struct async_bio_work; -struct work_struct free_work; - }; +struct work_struct free_work;atomic_t use_delay;atomic64_t delay_nsec;
Signature · 1 line
2.56.0
Re: [PATCH 19/98] blk-cgroup: don't free a blkg with its punt work queued
Quoted 6 lines · Jens Axboe
blkg_async_bio_workfn()relies on the bios on the list pinning the blkg. Butblkcg_punt_bio_submit()can callqueue_work()after a running instance already took and completed the bio, andqueue_work()succeeds on an empty list for the same reason. Nothing holds the blkg then, andfree_workshares its memory withasync_bio_workon top of that.
Quoted 7 lines
Hold a reference across
blkcg_punt_bio_submit(), give the two works memory of their own, and cancelasync_bio_workinblkg_free_workfn().Fixes d3f77dfdc718 ("blkcg: implement REQ_CGROUP_PUNT")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 20/98 mtip32xx: don't mark the disk dead in the irq handler
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
mtip_check_surprise_removal() calls blk_mark_disk_dead() from the interrupt handler, and that takes bd_holder_lock and calls into the holder. Only note the removal there and let the service thread tell the block layer.
--- drivers/block/mtip32xx/mtip32xx.c | 35 +++++++++++++++++++++++++------ drivers/block/mtip32xx/mtip32xx.h | 3 +++ 2 files changed, 32 insertions(+), 6 deletions(-)
drivers/block/mtip32xx/mtip32xx.c +29 -6
@@ -130,7 +130,7 @@ struct mtip_compat_ide_task_request_s {* return value * true if device removed, else false */ -static bool mtip_check_surprise_removal (struct driver_data *dd) +static bool mtip_device_removed (struct driver_data *dd) {u16 vendor_id =0 ;
@@ -141,14 +141,34 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)pci_read_config_word (dd->pdev,0x00 , &vendor_id);if (vendor_id ==0xFFFF ) { dd->sr = true; -if (dd->disk) -blk_mark_disk_dead (dd->disk);return true;/* device removed */ }return false;/* device present */ } +/* As above, and tell the block layer. May sleep. */ +static bool mtip_check_surprise_removal (struct driver_data *dd) +{ +if (!mtip_device_removed (dd)) +return false; + +if (dd->disk && !test_and_set_bit (MTIP_DDF_DISK_DEAD_BIT , &dd->dd_flag)) +blk_mark_disk_dead (dd->disk); +return true; +} + +/* For the interrupt handler: the service thread tells the block layer. */ +static bool mtip_check_surprise_removal_irq (struct driver_data *dd) +{ +if (!mtip_device_removed (dd)) +return false; + +set_bit (MTIP_PF_SR_BIT , &dd->port->flags); +wake_up_interruptible (&dd->port->svc_wait); +return true; +} +static struct mtip_cmd *mtip_cmd_from_tag (struct driver_data *dd,unsigned int tag) {
@@ -715,7 +735,7 @@ static inline void mtip_process_errors(struct driver_data *dd, u32 port_stat)dev_warn (&dd->pdev->dev,"Port stat errors %x unhandled\n" , (port_stat & ~PORT_IRQ_HANDLED )); -if (mtip_check_surprise_removal (dd)) +if (mtip_check_surprise_removal_irq (dd))return ; }if (likely (port_stat & (PORT_IRQ_TF_ERR |PORT_IRQ_IF_ERR ))) {
@@ -740,7 +760,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data)/* Acknowledge the interrupt status on the port.*/ port_stat =readl (port->mmio +PORT_IRQ_STAT );if (unlikely (port_stat ==0xFFFFFFFF )) { -mtip_check_surprise_removal (dd); +mtip_check_surprise_removal_irq (dd);return IRQ_HANDLED ; }writel (port_stat, port->mmio +PORT_IRQ_STAT );
@@ -784,7 +804,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data) }if (unlikely (port_stat &PORT_IRQ_ERR )) { -if (unlikely (mtip_check_surprise_removal (dd))) { +if (unlikely (mtip_check_surprise_removal_irq (dd))) {/* don't proceed further */ return IRQ_HANDLED ; }
@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)set_bit (MTIP_PF_SVC_THD_ACTIVE_BIT , &port->flags); +if (test_and_clear_bit (MTIP_PF_SR_BIT , &port->flags)) +mtip_check_surprise_removal (dd); + restart_eh:/* Demux bits: start with error handling */ if (test_bit (MTIP_PF_EH_ACTIVE_BIT , &port->flags)) {
drivers/block/mtip32xx/mtip32xx.h +3 -0
@@ -134,11 +134,13 @@ enum {MTIP_PF_ISSUE_CMDS_BIT =5 ,MTIP_PF_REBUILD_BIT =6 ,MTIP_PF_SVC_THD_STOP_BIT =8 , +MTIP_PF_SR_BIT =11 ,/* surprise removal seen in irq */ MTIP_PF_SVC_THD_WORK = ((1 <<MTIP_PF_EH_ACTIVE_BIT ) | (1 <<MTIP_PF_ISSUE_CMDS_BIT ) | (1 <<MTIP_PF_REBUILD_BIT ) | (1 <<MTIP_PF_SVC_THD_STOP_BIT ) | + (1 <<MTIP_PF_SR_BIT ) | (1 << MTIP_PF_TO_ACTIVE_BIT)),/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,6 +152,7 @@ enum {MTIP_DDF_RESUME_BIT =6 ,MTIP_DDF_INIT_DONE_BIT =7 ,MTIP_DDF_REBUILD_FAILED_BIT =8 , +MTIP_DDF_DISK_DEAD_BIT =9 , MTIP_DDF_STOP_IO = ((1 << MTIP_DDF_REMOVE_PENDING_BIT) | (1 << MTIP_DDF_SEC_LOCK_BIT) |
Signature · 1 line
2.56.0
Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler
drivers/block/mtip32xx/mtip32xx.c static int mtip_service_thread(void *data) · line 2518
Quoted 8 lines · Jens Axboe
drivers/block/mtip32xx/mtip32xx.c +3
-0@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)set_bit (MTIP_PF_SVC_THD_ACTIVE_BIT , &port->flags); +if (test_and_clear_bit (MTIP_PF_SR_BIT , &port->flags)) +mtip_check_surprise_removal (dd); + restart_eh:
Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but calling mtip_check_surprise_removal(dd) -> blk_mark_disk_dead(dd->disk) in mtip_service_thread() before aborting in-flight NCQ commands can deadlock when a filesystem is mounted on the device:
1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which locks sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If another task already holds s_umount while waiting for in-flight I/O on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() then calls mtip_stop_service_thread(dd) (kthread_stop()) before calling blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd), so mtip_pci_remove() blocks waiting for mtip_service_thread() before it can abort the in-flight commands holding up s_umount.
Quiescing dd->queue and aborting in-flight commands via blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd) before calling blk_mark_disk_dead(dd->disk) on surprise removal would avoid that deadlock.
Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler
In reply to Jens AxboeReview static int mtip_service_thread(void *data) · line 2518
Quoted 8 lines · Jens Axboe
(quoted diff) +3
-0@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data) set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags); + if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags)) + mtip_check_surprise_removal(dd); + restart_eh:
Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but calling blk_mark_disk_dead(dd->disk) in mtip_service_thread() before aborting in-flight NCQ commands can deadlock when a filesystem is mounted on the device:
1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which takes sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If another task already holds s_umount while waiting for in-flight I/O on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() calls mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands, so mtip_pci_remove() blocks waiting for mtip_service_thread() before it can abort the commands holding up s_umount.
Also, mtip_abort_cmd() currently calls mtip_softirq_done_fn(req) directly without transitioning the request through blk_mq_complete_request().
How about keeping mtip_check_surprise_removal() non-sleeping everywhere (setting dd->sr, a new MTIP_DDF_DEAD_BIT to fail new normal and reserved requests immediately with BLK_STS_IOERR, and waking the service thread with MTIP_PF_SR_BIT), and having a helper mtip_handle_surprise_removal() quiesce dd->queue and abort in-flight requests via blk_mq_tagset_busy_iter() + blk_mq_complete_request() before calling blk_mark_disk_dead(dd->disk)?
Patch 21/98 still applies cleanly on top of this replacement, and my reply to patch 22/98 has the matching follow-up change. Checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested on mtip32xx hardware:
---
drivers/block/mtip32xx/mtip32xx.c +35 -3
@@ -141,8 +141,11 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)pci_read_config_word (dd->pdev,0x00 , &vendor_id);if (vendor_id ==0xFFFF ) { dd->sr = true; -if (dd->disk) -blk_mark_disk_dead (dd->disk); +set_bit (MTIP_DDF_DEAD_BIT , &dd->dd_flag); +if (dd->port) { +set_bit (MTIP_PF_SR_BIT , &dd->port->flags); +wake_up_interruptible (&dd->port->svc_wait); + }return true;/* device removed */ }
@@ -2438,9 +2441,12 @@ static bool mtip_abort_cmd(struct request *req, void *data)dbg_printk (MTIP_DRV_NAME " Aborting request, tag = %d\n" , req->tag); +if (blk_mq_request_completed (req)) +return true; +clear_bit (req->tag, dd->port->cmds_to_issue); cmd->status =BLK_STS_IOERR ; -mtip_softirq_done_fn (req); +blk_mq_complete_request (req);return true; }
@@ -2462,6 +2468,18 @@ static bool mtip_queue_cmd(struct request *req, void *data)* 0 */ +static void mtip_handle_surprise_removal (struct driver_data *dd) +{ +if (dd->queue) { +blk_mq_quiesce_queue (dd->queue); +blk_mq_tagset_busy_iter (&dd->tags, mtip_abort_cmd, dd); +blk_mq_unquiesce_queue (dd->queue); + } +if (test_bit (MTIP_DDF_INIT_DONE_BIT , &dd->dd_flag) && + !test_and_set_bit (MTIP_DDF_DISK_DEAD_BIT , &dd->dd_flag)) +blk_mark_disk_dead (dd->disk); +} +static int mtip_service_thread (void *data) {struct driver_data *dd = (struct driver_data *)data;
@@ -2492,6 +2510,15 @@ static int mtip_service_thread(void *data)set_bit (MTIP_PF_SVC_THD_ACTIVE_BIT , &port->flags); +if (test_and_clear_bit (MTIP_PF_SR_BIT , &port->flags)) { +mtip_handle_surprise_removal (dd); +clear_bit (MTIP_PF_EH_ACTIVE_BIT , &port->flags); +clear_bit (MTIP_PF_TO_ACTIVE_BIT , &port->flags); +clear_bit (MTIP_PF_ISSUE_CMDS_BIT , &port->flags); +clear_bit (MTIP_PF_REBUILD_BIT , &port->flags); +continue ; + } + restart_eh:/* Demux bits: start with error handling */ if (test_bit (MTIP_PF_EH_ACTIVE_BIT , &port->flags)) {
@@ -3241,6 +3268,8 @@ static inline bool is_stopped(struct driver_data *dd, struct request *rq)return true;if (test_bit (MTIP_DDF_REBUILD_FAILED_BIT , &dd->dd_flag))return true; +if (test_bit (MTIP_DDF_DEAD_BIT , &dd->dd_flag)) +return true;return false; }
@@ -3279,6 +3308,9 @@ static blk_status_t mtip_issue_reserved_cmd(struct blk_mq_hw_ctx *hctx, dd->port->command_list +sizeof (struct mtip_cmd_hdr ) * rq->tag;struct mtip_cmd_sg *command_sg; +if (test_bit (MTIP_DDF_DEAD_BIT , &dd->dd_flag)) +return BLK_STS_IOERR ; +if (mtip_commands_active (dd->port))return BLK_STS_DEV_RESOURCE ;
drivers/block/mtip32xx/mtip32xx.h +6 -1
@@ -134,11 +134,13 @@ enum {MTIP_PF_ISSUE_CMDS_BIT =5 ,MTIP_PF_REBUILD_BIT =6 ,MTIP_PF_SVC_THD_STOP_BIT =8 , +MTIP_PF_SR_BIT =11 ,/* surprise removal seen */ MTIP_PF_SVC_THD_WORK = ((1 <<MTIP_PF_EH_ACTIVE_BIT ) | (1 <<MTIP_PF_ISSUE_CMDS_BIT ) | (1 <<MTIP_PF_REBUILD_BIT ) | (1 <<MTIP_PF_SVC_THD_STOP_BIT ) | + (1 <<MTIP_PF_SR_BIT ) | (1 << MTIP_PF_TO_ACTIVE_BIT)),/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,12 +152,15 @@ enum {MTIP_DDF_RESUME_BIT =6 ,MTIP_DDF_INIT_DONE_BIT =7 ,MTIP_DDF_REBUILD_FAILED_BIT =8 , +MTIP_DDF_DEAD_BIT =9 , +MTIP_DDF_DISK_DEAD_BIT =10 ,MTIP_DDF_STOP_IO = ((1 <<MTIP_DDF_REMOVE_PENDING_BIT ) | (1 <<MTIP_DDF_SEC_LOCK_BIT ) | (1 <<MTIP_DDF_OVER_TEMP_BIT ) | (1 <<MTIP_DDF_WRITE_PROTECT_BIT ) | - (1 << MTIP_DDF_REBUILD_FAILED_BIT)), + (1 <<MTIP_DDF_REBUILD_FAILED_BIT ) | + (1 << MTIP_DDF_DEAD_BIT)), };
Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler
In reply to Jens AxboeNot objections against the quick fixes, but maybe we should use as a pretext to remove the driver? The hardware is also supported by ahci (with reduced performance) and the driver is unmaintained and pretty gross.
Jens Axboe
maintainerPATCH 21/98 mtip32xx: abort timed out commands of a removed device
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
mtip_device_reset() returns 0 for a removed device, so the timeout handler takes that for a successful reset and reissues the commands to registers that are no longer there, forever. Return -ENODEV so they get aborted instead.
--- drivers/block/mtip32xx/mtip32xx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/mtip32xx/mtip32xx.c +1 -1
@@ -456,7 +456,7 @@ static int mtip_device_reset(struct driver_data *dd)int rv =0 ;if (mtip_check_surprise_removal (dd)) -return 0 ; +return -ENODEV ;if (mtip_hba_reset (dd) <0 ) rv = -EFAULT ;
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 22/98 mtip32xx: abort commands in flight on surprise removal
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
mtip_pci_remove() relies on blk_mark_disk_dead(), which only stops new IO. del_gendisk() waits for the requests the driver still owns, and nothing completes them for a removed device: the timeout handler leaves them to the service thread, which exits as soon as it sees MTIP_DDF_REMOVE_PENDING_BIT. The removal hangs.
Stop the service thread and abort the commands in flight before del_gendisk() if the device is gone.
--- drivers/block/mtip32xx/mtip32xx.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-)
drivers/block/mtip32xx/mtip32xx.c +23 -5
@@ -3887,6 +3887,17 @@ static int mtip_pci_probe(struct pci_dev *pdev,return rv; } +static void mtip_stop_service_thread (struct driver_data *dd) +{ +if (!dd->mtip_svc_handler) +return ; + +set_bit (MTIP_PF_SVC_THD_STOP_BIT , &dd->port->flags); +wake_up_interruptible (&dd->port->svc_wait); +kthread_stop (dd->mtip_svc_handler); + dd->mtip_svc_handler =NULL ; +} +/* * Called for each probed device when the device is removed or the * driver is unloaded.
@@ -3917,16 +3928,23 @@ static void mtip_pci_remove(struct pci_dev *pdev)set_bit (MTIP_DDF_REMOVE_PENDING_BIT , &dd->dd_flag);mutex_lock (&dd->ioctl_mutex); +/* +* Nobody will complete the commands a removed device has been given, +* and del_gendisk() waits for them. +*/ +if (dd->sr) { +mtip_stop_service_thread (dd); +blk_mq_quiesce_queue (dd->queue); +blk_mq_tagset_busy_iter (&dd->tags, mtip_abort_cmd, dd); +blk_mq_unquiesce_queue (dd->queue); + } +if (test_bit (MTIP_DDF_INIT_DONE_BIT , &dd->dd_flag))del_gendisk (dd->disk);mtip_hw_debugfs_exit (dd); -if (dd->mtip_svc_handler) { -set_bit (MTIP_PF_SVC_THD_STOP_BIT , &dd->port->flags); -wake_up_interruptible (&dd->port->svc_wait); -kthread_stop (dd->mtip_svc_handler); - } +mtip_stop_service_thread (dd);if (!dd->sr) {/*
Signature · 1 line
2.56.0
Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal
drivers/block/mtip32xx/mtip32xx.c static void mtip_pci_remove(struct pci_dev *pdev) · line 3943
Quoted 13 lines · Jens Axboe
drivers/block/mtip32xx/mtip32xx.c +11
-0+/* +* Nobody will complete the commands a removed device has been given, +* and del_gendisk() waits for them. +*/ +if (dd->sr) { +mtip_stop_service_thread (dd); +blk_mq_quiesce_queue (dd->queue); +blk_mq_tagset_busy_iter (&dd->tags, mtip_abort_cmd, dd); +blk_mq_unquiesce_queue (dd->queue); + } +if (test_bit (MTIP_DDF_INIT_DONE_BIT , &dd->dd_flag))del_gendisk (dd->disk);
Two issues with this ordering:
1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands can deadlock with patch 20/98 if mtip_service_thread() is blocked inside blk_mark_disk_dead() -> fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss occurs after mtip_pci_remove() has already checked if (dd->sr) (false) and entered del_gendisk(dd->disk), MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread() then exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT without handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving in-flight commands uncompleted until the request timer fires.
If mtip_service_thread() aborts in-flight commands when handling MTIP_PF_SR_BIT (before blk_mark_disk_dead()) and does not exit early on MTIP_DDF_REMOVE_PENDING_BIT while surprise-removal/timeout work is pending, mtip_stop_service_thread() can stay after del_gendisk() and both races are closed.
Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal
In reply to Jens AxboeReview
Quoted 13 lines · Jens Axboe
(quoted diff) +11
-0+ /* + * Nobody will complete the commands a removed device has been given, + * and del_gendisk() waits for them. + */ + if (dd->sr) { + mtip_stop_service_thread(dd); + blk_mq_quiesce_queue(dd->queue); + blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd); + blk_mq_unquiesce_queue(dd->queue); + } + if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag)) del_gendisk(dd->disk);
Two issues with this ordering:
1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting in-flight commands deadlocks with patch 20/98 if mtip_service_thread() is blocked inside blk_mark_disk_dead() -> fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss happens after mtip_pci_remove() has already checked dd->sr (false) and entered del_gendisk(dd->disk), MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread() exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT without handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving in-flight commands uncompleted forever in del_gendisk().
With the replacement proposed for patch 20/98 (where mtip_handle_surprise_removal() aborts in-flight commands before calling blk_mark_disk_dead()), we can instead keep mtip_service_thread() running until kthread_stop() after del_gendisk(dd->disk) by removing its early exit on MTIP_DDF_REMOVE_PENDING_BIT (and clearing MTIP_PF_REBUILD_BIT if remove is pending or mtip_ftl_rebuild_poll(dd) <= 0, since rebuild poll returns -EFAULT once MTIP_DDF_REMOVE_PENDING_BIT is set). That handles surprise removal both before and during del_gendisk() from the service thread without racing two tagset iterations.
Below is the replacement for 22/98 on top of the 20/98 replacement + unmodified 21/98 (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested on mtip32xx hardware):
---
drivers/block/mtip32xx/mtip32xx.c +6 -5
@@ -2504,10 +2504,6 @@ static int mtip_service_thread(void *data) test_bit(MTIP_PF_SVC_THD_STOP_BIT, &port->flags)) goto st_out; -if (unlikely (test_bit (MTIP_DDF_REMOVE_PENDING_BIT , - &dd->dd_flag))) - goto st_out; -set_bit (MTIP_PF_SVC_THD_ACTIVE_BIT , &port->flags);if (test_and_clear_bit (MTIP_PF_SR_BIT , &port->flags)) {
@@ -2588,7 +2584,9 @@ restart_eh: }if (test_bit (MTIP_PF_REBUILD_BIT , &port->flags)) { -if (mtip_ftl_rebuild_poll (dd) ==0 ) +if (test_bit (MTIP_DDF_REMOVE_PENDING_BIT , + &dd->dd_flag) || +mtip_ftl_rebuild_poll (dd) <=0 )clear_bit (MTIP_PF_REBUILD_BIT , &port->flags); } }
@@ -3926,6 +3924,9 @@ static void mtip_pci_remove(struct pci_dev *pdev)set_bit (MTIP_DDF_REMOVE_PENDING_BIT , &dd->dd_flag);mutex_lock (&dd->ioctl_mutex); +if (dd->sr && !dd->mtip_svc_handler) +mtip_handle_surprise_removal (dd); +if (test_bit (MTIP_DDF_INIT_DONE_BIT , &dd->dd_flag))del_gendisk (dd->disk);
Jens Axboe
maintainerPATCH 23/98 blk-iolatency: flush enable_work after going offline
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blkcg_iolatency_exit() flushes enable_work before it deactivates the policy, but iolatency_pd_offline() schedules it again for the last group with a target, so the blk_iolatency is freed with the work queued. Flush it after the deactivation.
--- block/blk-iolatency.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-iolatency.c +2 -1
@@ -637,8 +637,9 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos)struct blk_iolatency *blkiolat =BLKIOLATENCY (rqos);timer_shutdown_sync (&blkiolat->timer); -flush_work (&blkiolat->enable_work);blkcg_deactivate_policy (rqos->disk, &blkcg_policy_iolatency); +/* iolatency_pd_offline() schedules it for the last group with a target */ +flush_work (&blkiolat->enable_work);kfree (blkiolat); }
Signature · 1 line
2.56.0
Re: [PATCH 23/98] blk-iolatency: flush enable_work after going offline
Quoted 9 lines · Jens Axboe
blkcg_iolatency_exit()flushesenable_workbefore it deactivates the policy, butiolatency_pd_offline()schedules it again for the last group with a target, so theblk_iolatencyis freed with the work queued. Flush it after the deactivation.Fixes 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 24/98 zram: accept lz4 dictionaries of more than 64 KiB
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
LZ4_loadDict() and LZ4_loadDictHC() only use the last 64KiB of a dictionary and return how much they used, which the zram backends take for an error. Expect that instead.
--- drivers/block/zram/backend_lz4.c | 4 +++- drivers/block/zram/backend_lz4hc.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/block/zram/backend_lz4.c +3 -1
@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h>
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params) ret =LZ4_loadDict (dict_stream, params->dict, params->dict_sz); -if (ret != params->dict_sz) { +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K )) {kfree (dict_stream);return -EINVAL ; }
drivers/block/zram/backend_lz4hc.c +3 -1
@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h>
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,LZ4_resetStreamHC (zctx->cstrm, params->level); ret =LZ4_loadDictHC (zctx->cstrm, params->dict, params->dict_sz); -if (ret != params->dict_sz) +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K ))return -EINVAL ; ret =LZ4_compress_HC_continue (zctx->cstrm, req->src, req->dst, req->src_len, req->dst_len);
Signature · 1 line
2.56.0
Sergey Senozhatsky
maintainerRe: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB
On (26/10/28 10:41), Jens Axboe wrote:
[..]
Quoted 29 lines
#include <
linux/kernel.h>
#include <linux/lz4.h>drivers/block/zram/backend_lz4.c +3
-1+#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params) ret =LZ4_loadDict (dict_stream, params->dict, params->dict_sz); -if (ret != params->dict_sz) { +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K )) {kfree (dict_stream);return -EINVAL ; }drivers/block/zram/backend_lz4hc.c +1
-0@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h>
drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102
Quoted 7 lines
drivers/block/zram/backend_lz4hc.c +2
-1@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,LZ4_resetStreamHC (zctx->cstrm, params->level); ret =LZ4_loadDictHC (zctx->cstrm, params->dict, params->dict_sz); -if (ret != params->dict_sz) +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K ))return -EINVAL ;
I think this should patch lz4hc_setup_params() instead. Similarly to lz4 hunk.
Jens Axboe
maintainerRe: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB
Quoted 30 lines · Sergey Senozhatsky
On (26/10/28 10:41), Jens Axboe wrote:
[..]Quoted 28 lines
#include <
linux/kernel.h>
#include <linux/lz4.h>drivers/block/zram/backend_lz4.c +3
-1+#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params) ret =LZ4_loadDict (dict_stream, params->dict, params->dict_sz); -if (ret != params->dict_sz) { +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K )) {kfree (dict_stream);return -EINVAL ; }drivers/block/zram/backend_lz4hc.c +1
-0@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h>
Quoted 11 lines
drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102
Quoted 8 lines
drivers/block/zram/backend_lz4hc.c +2
-1@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,LZ4_resetStreamHC (zctx->cstrm, params->level); ret =LZ4_loadDictHC (zctx->cstrm, params->dict, params->dict_sz); -if (ret != params->dict_sz) +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K ))return -EINVAL ;I think this should patch
lz4hc_setup_params()instead. Similarly to lz4 hunk.
You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.
Signature · 1 line
Jens Axboe
Sergey Senozhatsky
maintainerRe: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB
Quoted 43 lines · Jens Axboe
Quoted 30 lines · Sergey Senozhatsky
On (26/10/28 10:41), Jens Axboe wrote:
[..]Quoted 28 lines
#include <
linux/kernel.h>
#include <linux/lz4.h>drivers/block/zram/backend_lz4.c +3
-1+#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params) ret =LZ4_loadDict (dict_stream, params->dict, params->dict_sz); -if (ret != params->dict_sz) { +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K )) {kfree (dict_stream);return -EINVAL ; }drivers/block/zram/backend_lz4hc.c +1
-0@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> Quoted 11 lines
drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102
Quoted 8 lines
drivers/block/zram/backend_lz4hc.c +2
-1@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,LZ4_resetStreamHC (zctx->cstrm, params->level); ret =LZ4_loadDictHC (zctx->cstrm, params->dict, params->dict_sz); -if (ret != params->dict_sz) +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K ))return -EINVAL ;I think this should patch
lz4hc_setup_params()instead. Similarly to lz4 hunk.
Quoted 4 lines
You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.
Works for me, I'll queue up a fix (will give you credit).
Jens Axboe
maintainerRe: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB
Quoted 44 lines · Sergey Senozhatsky
Quoted 43 lines · Jens Axboe
Quoted 41 lines · Sergey Senozhatsky
On (26/10/28 10:41), Jens Axboe wrote:
[..]Quoted 29 lines
#include <
linux/kernel.h>
#include <linux/lz4.h>drivers/block/zram/backend_lz4.c +3
-1+#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params) ret =LZ4_loadDict (dict_stream, params->dict, params->dict_sz); -if (ret != params->dict_sz) { +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K )) {kfree (dict_stream);return -EINVAL ; }drivers/block/zram/backend_lz4hc.c +1
-0@@ -4,6 +4,7 @@#include <linux/kernel.h> #include <linux/lz4.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/vmalloc.h> drivers/block/zram/backend_lz4hc.c static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx, · line 102
Quoted 7 lines
drivers/block/zram/backend_lz4hc.c +2
-1@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,LZ4_resetStreamHC (zctx->cstrm, params->level); ret =LZ4_loadDictHC (zctx->cstrm, params->dict, params->dict_sz); -if (ret != params->dict_sz) +/* Only the last 64 KiB of a dictionary are of use */ +if (ret !=min_t (size_t, params->dict_sz,SZ_64K ))return -EINVAL ;I think this should patch
lz4hc_setup_params()instead. Similarly to lz4 hunk.
Quoted 6 lines
Quoted 4 lines
You're the maintainer, so queue up a patch like that. Like I told Bart, I'm not sending this out to get review. Either ack it if it's good, and if not, as the maintainer of this driver, just fix it and I'll happily drop this one.
Works for me, I'll queue up a fix (will give you credit).
Perfect, thank you! I'll drop this one.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 25/98 blk-throttle: pass the I/O counts of a dead group on
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The recursive io_service_bytes and io_serviced files sum up the online groups plus what dead descendants left in aux_cnt. That worked while the counters lived in the blkg. Since commit 7ca464383aec they're in the throtl_grp and nobody passes them up, so the files go backwards when a child is removed. Add them to the parent in throtl_pd_offline().
--- block/blk-throttle.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
block/blk-throttle.c +12 -1
@@ -1670,7 +1670,18 @@ static void tg_flush_bios(struct throtl_grp *tg)static void throtl_pd_offline (struct blkg_policy_data *pd) { -tg_flush_bios (pd_to_tg (pd)); +struct throtl_grp *tg =pd_to_tg (pd); +struct throtl_grp *parent =blkg_to_tg (tg_to_blkg (tg)->parent); + +tg_flush_bios (tg); + +/* the recursive files of the ancestors go on counting what it did */ +if (parent) { +blkg_rwstat_add_aux (&parent->stat_bytes, &tg->stat_bytes); +blkg_rwstat_add_aux (&parent->stat_ios, &tg->stat_ios); +blkg_rwstat_reset (&tg->stat_bytes); +blkg_rwstat_reset (&tg->stat_ios); + } }struct blkcg_policy blkcg_policy_throtl = {
Signature · 1 line
2.56.0
Re: [PATCH 25/98] blk-throttle: pass the I/O counts of a dead group on
Quoted 10 lines · Jens Axboe
The recursive
io_service_bytesandio_servicedfiles sum up the online groups plus what dead descendants left inaux_cnt. That worked while the counters lived in the blkg. Since commit7ca464383aecthey're in thethrotl_grpand nobody passes them up, so the files go backwards when a child is removed. Add them to the parent inthrotl_pd_offline().Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 26/98 blk-throttle: make blkio.reset_stats reset I/O counts
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blkio.throttle.io_service_bytes and io_serviced were reset by blkcg_reset_stats() while the counters were part of the blkg. Now that they live in the throtl_grp, add a pd_reset_stats_fn for them.
--- block/blk-throttle.c | 9 +++++++++ 1 file changed, 9 insertions(+)
block/blk-throttle.c +9 -0
@@ -1684,6 +1684,14 @@ static void throtl_pd_offline(struct blkg_policy_data *pd) } } +static void throtl_pd_reset_stats (struct blkg_policy_data *pd) +{ +struct throtl_grp *tg =pd_to_tg (pd); + +blkg_rwstat_reset (&tg->stat_bytes); +blkg_rwstat_reset (&tg->stat_ios); +} +struct blkcg_policy blkcg_policy_throtl = { .dfl_cftypes = throtl_files, .legacy_cftypes = throtl_legacy_files,
@@ -1693,6 +1701,7 @@ struct blkcg_policy blkcg_policy_throtl = { .pd_online_fn = throtl_pd_online, .pd_offline_fn = throtl_pd_offline, .pd_free_fn = throtl_pd_free, + .pd_reset_stats_fn = throtl_pd_reset_stats, }; static void tg_cancel_writeback_bios(struct throtl_grp *tg,
Signature · 1 line
2.56.0
Re: [PATCH 26/98] blk-throttle: make blkio.reset_stats reset I/O counts
Quoted 8 lines · Jens Axboe
blkio.throttle.io_service_bytes and
io_servicedwere reset byblkcg_reset_stats()while the counters were part of the blkg. Now that they live in thethrotl_grp, add apd_reset_stats_fnfor them.Fixes 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 27/98 blk-mq: don't trust the cookie in blk_mq_poll()
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_mq_poll() indexes queue_hw_ctx[] with bio->bi_cookie unchecked. Zone write plugging keeps the segment count of a plugged write in __bi_nr_segments, which shares a union with the cookie, and clears REQ_POLLED, but bio_poll() only looks at the cookie. A plugged polled write with more segments than there are hardware queues reads past the array. The bio is also SLAB_TYPESAFE_BY_RCU, so a cookie from a bio reused for another device can meet the wrong array as well.
Ignore a cookie that is out of range or doesn't name a poll queue.
--- block/blk-mq.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
block/blk-mq.c +14 -1
@@ -5245,9 +5245,22 @@ static int blk_hctx_poll(struct request_queue *q, struct blk_mq_hw_ctx *hctx,int blk_mq_poll (struct request_queue *q,blk_qc_t cookie,struct io_comp_batch *iob,unsigned int flags) { +struct blk_mq_hw_ctx *hctx; +if (!blk_mq_can_poll (q))return 0 ; -return blk_hctx_poll (q, q->queue_hw_ctx[cookie], iob, flags); + +/* +* The cookie is no more than a hint. iocb_bio_iopoll() may have read it +* from a bio that was reused for another queue right afterwards, and +* zone write plugging keeps the number of segments in the same field. +*/ +if (cookie >= q->nr_hw_queues) +return 0 ; + hctx = q->queue_hw_ctx[cookie]; +if (hctx->type !=HCTX_TYPE_POLL ) +return 0 ; +return blk_hctx_poll (q, hctx, iob, flags); } int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
Signature · 1 line
2.56.0
Re: [PATCH 27/98] blk-mq: don't trust the cookie in blk_mq_poll()
Looks good:
Jens Axboe
maintainerPATCH 28/98 blk-iocost: check again that the disk is alive
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blkg_conf_open_bdev() checks that the disk is alive under rq_qos_mutex, but ioc_qos_write() drops the mutex to freeze the queue first. If del_gendisk() runs in between, the tag set may be gone by the time blk_mq_quiesce_queue() looks at it. Check disk_live() again after retaking the mutex.
--- block/blk-iocost.c | 5 +++++ 1 file changed, 5 insertions(+)
block/blk-iocost.c +5 -0
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, body = ctx.body; disk = ctx.bdev->bd_disk; +/* It may have gone while the mutex was not held */ +if (!disk_live (disk)) { + ret = -ENODEV ; + goto close_bdev; + }if (!queue_is_mq (disk->queue)) { ret = -EOPNOTSUPP ; goto close_bdev;
Signature · 1 line
2.56.0
Re: [PATCH 28/98] blk-iocost: check again that the disk is alive
block/blk-iocost.c static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3329
Quoted 9 lines · Jens Axboe
block/blk-iocost.c +5
-0@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, body = ctx.body; disk = ctx.bdev->bd_disk; +/* It may have gone while the mutex was not held */ +if (!disk_live (disk)) { + ret = -ENODEV ; + goto close_bdev; + }
Look at the close_bdev unwind path in ioc_qos_write():
close_bdev:
q = ctx.bdev->bd_queue; blkg_conf_close_bdev(&ctx); blk_mq_unfreeze_queue(q, memflags);
blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because del_gendisk() (and potentially put_disk()) ran while rq_qos_mutex was dropped, ctx.bdev may hold the final reference to the bdev/gendisk. blkdev_put_no_open(ctx.bdev) will then run disk_release() -> blk_put_queue(q) and free q while it is still frozen, before the next line calls blk_mq_unfreeze_queue(q, memflags) on the freed queue.
q->rq_qos_mutex needs to be unlocked and blk_mq_unfreeze_queue(q, memflags) called before blkdev_put_no_open(ctx.bdev) drops the bdev reference.
Re: [PATCH 28/98] blk-iocost: check again that the disk is alive
In reply to Jens AxboeReview static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3329
Quoted 9 lines · Jens Axboe
(quoted diff) +5
-0@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, body = ctx.body; disk = ctx.bdev->bd_disk; + /* It may have gone while the mutex was not held */ + if (!disk_live(disk)) { + ret = -ENODEV; + goto close_bdev; + }
Look at the close_bdev unwind path in ioc_qos_write():
close_bdev:
q = ctx.bdev->bd_queue; blkg_conf_close_bdev(&ctx); blk_mq_unfreeze_queue(q, memflags);
blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because del_gendisk() and put_disk() ran while rq_qos_mutex was dropped, ctx.bdev can hold the last reference to the bdev/gendisk. Calling blkg_conf_close_bdev(&ctx) before blk_mq_unfreeze_queue(q, memflags) will then run disk_release() -> blk_put_queue(q) and free q while it is still frozen, followed by a use-after-free in blk_mq_unfreeze_queue().
Since __blk_mq_unfreeze_queue() acquires q->mq_freeze_lock (which nests outside q->rq_qos_mutex), close_bdev should unlock q->rq_qos_mutex, call blk_mq_unfreeze_queue(q, memflags) while ctx.bdev still pins the disk and queue, and only then call blkdev_put_no_open(ctx.bdev).
Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
block/blk-iocost.c +7 -1
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, body = ctx.body; disk = ctx.bdev->bd_disk; +/* It may have gone while the mutex was not held */ +if (!disk_live (disk)) { + ret = -ENODEV ; + goto close_bdev; + }if (!queue_is_mq (disk->queue)) { ret = -EOPNOTSUPP ; goto close_bdev;
@@ -3437,8 +3442,9 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, close_bdev: q = ctx.bdev->bd_queue; -blkg_conf_close_bdev (&ctx); +mutex_unlock (&q->rq_qos_mutex);blk_mq_unfreeze_queue (q, memflags); +blkdev_put_no_open (ctx.bdev);return ret ?: nbytes; einval:
Jens Axboe
maintainerPATCH 29/98 blk-mq: keep sharing tags fairly after a queue update
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A hctx only gets BLK_MQ_F_TAG_QUEUE_SHARED from blk_mq_add_queue_tag_set(). The hctxs blk_mq_update_nr_hw_queues() allocates for queues that joined long ago never get it, so they aren't counted in active_queues and hctx_may_queue() lets them take all the tags. One NVMe namespace can starve the others after the first update.
Set the flag on the new hctxs.
--- block/blk-mq.c | 3 +++ 1 file changed, 3 insertions(+)
block/blk-mq.c +3 -0
@@ -5184,6 +5184,9 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, goto fallback; }blk_mq_map_swqueue (q); +/* The new hctxs don't know yet that they share their tags */ +if (set->flags &BLK_MQ_F_TAG_QUEUE_SHARED ) +queue_set_hctx_shared (q, true); } switch_back:/* The blk_mq_elv_switch_back unfreezes queue for us. */
Signature · 1 line
2.56.0
Re: [PATCH 29/98] blk-mq: keep sharing tags fairly after a queue update
Looks good:
Jens Axboe
maintainerPATCH 30/98 blk-mq: don't bring back flags of a completed request
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The timeout handler sets and clears RQF_TIMED_OUT in a request it only holds a reference to, with a plain read-modify-write, while the completion clears flags in rq_flags the same way. If the two interleave, blk_mq_finish_request() runs twice with the flags back: the zone write plug reference is dropped twice and the scheduler's ->finish_request() is called again.
Change RQF_TIMED_OUT with cmpxchg, so the handler never writes back a flag that changed under it.
--- block/blk-mq.c | 2 +- block/blk-timeout.c | 3 ++- block/blk.h | 14 ++++++++++++++ 3 files changed, 17 insertions(+), 2 deletions(-)
block/blk-mq.c +1 -1
@@ -1639,7 +1639,7 @@ EXPORT_SYMBOL_GPL(blk_mq_queue_inflight);static void blk_mq_rq_timed_out (struct request *req) { - req->rq_flags |=RQF_TIMED_OUT ; +blk_rq_change_timed_out (req, true);if (req->q->mq_ops->timeout) {enum blk_eh_timer_return ret;
block/blk-timeout.c +2 -1
@@ -137,7 +137,8 @@ void blk_add_timer(struct request *req)if (!req->timeout) req->timeout = q->rq_timeout; - req->rq_flags &= ~RQF_TIMED_OUT ; +if (req->rq_flags &RQF_TIMED_OUT ) +blk_rq_change_timed_out (req, false); expiry = jiffies + req->timeout;WRITE_ONCE (req->deadline, expiry);
block/blk.h +14 -0
@@ -365,6 +365,20 @@ static inline void bio_integrity_free(struct bio *bio)unsigned long blk_rq_timeout (unsigned long timeout);void blk_add_timer (struct request *req); +/* +* The timeout handler changes RQF_TIMED_OUT in a request which it does not +* own: whoever completes it changes rq_flags at the same time, with plain +* loads and stores. Never write back a flag that has changed under us. +*/ +static inline void blk_rq_change_timed_out (struct request *rq,bool set) +{ +req_flags_t old =READ_ONCE (rq->rq_flags), new; + +do { + new = set ? old |RQF_TIMED_OUT : old & ~RQF_TIMED_OUT ; + }while (!try_cmpxchg (&rq->rq_flags, &old, new)); +} +enum bio_merge_status {BIO_MERGE_OK ,BIO_MERGE_NONE ,
Signature · 1 line
2.56.0
Re: [PATCH 30/98] blk-mq: don't bring back flags of a completed request
block/blk.h static inline void bio_integrity_free(struct bio *bio) · line 380
Quoted 8 lines · Jens Axboe
block/blk.h +8
-0+static inline void blk_rq_change_timed_out (struct request *rq,bool set) +{ +req_flags_t old =READ_ONCE (rq->rq_flags), new; + +do { + new = set ? old |RQF_TIMED_OUT : old & ~RQF_TIMED_OUT ; + }while (!try_cmpxchg (&rq->rq_flags, &old, new)); +}
I've seen this pattern so often lately that a generic helper might be in order.
Otherwise looks good:
Jens Axboe
maintainerPATCH 31/98 blk-mq: don't forget a timer the driver has restarted
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
If ->timeout() returns BLK_EH_RESET_TIMER, blk_add_timer() gives the request a new deadline that blk_mq_timeout_work() doesn't know about when it picks the next expiry. A 2s nbd read next to a 60s one expires once and then not again for a minute. Let the new deadline take part.
--- block/blk-mq.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
block/blk-mq.c +4 -1
@@ -1710,8 +1710,11 @@ static bool blk_mq_handle_expired(struct request *rq, void *priv) {struct blk_expired_data *expired = priv; -if (blk_mq_req_expired (rq, expired)) +if (blk_mq_req_expired (rq, expired)) {blk_mq_rq_timed_out (rq); +/* If its timer has been started again, count the new deadline */ +blk_mq_req_expired (rq, expired); + }return true; }
Signature · 1 line
2.56.0
Re: [PATCH 31/98] blk-mq: don't forget a timer the driver has restarted
block/blk-mq.c static bool blk_mq_handle_expired(struct request *rq, void *priv) · line 1715
Quoted 3 lines · Jens Axboe
block/blk-mq.c +2
-0+if (blk_mq_req_expired (rq, expired)) {blk_mq_rq_timed_out (rq); +/* If its timer has been started again, count the new deadline */
Comment nit: if you capitalize the first word, it is a sentence and needs to end with a period. Also an overly long line.
Otherwise looks good.
Jens Axboe
maintainerPATCH 32/98 blk-mq-dma: initialize the dma state for P2P, too
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_rq_dma_map_iter_start() promises that the caller need not initialize the dma_iova_state, but the P2P bus address path returns before it does so. The state then holds what the last request with that tag left, and blk_rq_dma_map_coalesce() describes a transfer as one IOVA range that it isn't. Clear it on that path too.
--- block/blk-mq-dma.c | 2 ++ 1 file changed, 2 insertions(+)
block/blk-mq-dma.c +2 -0
@@ -181,6 +181,8 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,switch (pci_p2pdma_state (&iter->p2pdma, dma_dev,phys_to_page (vec.paddr))) {case PCI_P2PDMA_MAP_BUS_ADDR : +/* don't leave the state of an earlier mapping to the caller */ +memset (state,0 ,sizeof (*state));return blk_dma_map_bus (iter, &vec);case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE :/*
Signature · 1 line
2.56.0
Re: [PATCH 32/98] blk-mq-dma: initialize the dma state for P2P, too
Quoted 7 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_rq_dma_map_iter_start()promises that the caller need not initialize thedma_iova_state, but the P2P bus address path returns before it does so. The state then holds what the last request with that tag left, andblk_rq_dma_map_coalesce()describes a transfer as one IOVA range that it isn't. Clear it on that path too.
Looks good:
Jens Axboe
maintainerPATCH 33/98 blk-mq-dma: map the metadata a passthrough request has
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_rq_integrity_dma_map_iter_start() sizes the metadata from the sector count and the device format. A passthrough request has whatever userspace gave it, so too much is mapped past the allocated IOVA range and too little hands the device an unmapped length. If the data needs a second bio, the iterator moves on to it and dereferences a NULL integrity payload.
Take the size from the payload and stop at a bio without one.
--- block/blk-mq-dma.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/blk-mq-dma.c +8 -0
@@ -13,6 +13,10 @@ static bool __blk_map_iter_next(struct blk_map_iter *iter)if (!iter->bio || !iter->bio->bi_next)return false; +/* All the metadata of a passthrough request is in its first bio */ +if (iter->is_integrity && !bio_integrity (iter->bio->bi_next)) +return false; + iter->bio = iter->bio->bi_next;if (iter->is_integrity) { iter->iter =bio_integrity (iter->bio)->bip_iter;
@@ -348,6 +352,10 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req, blk_rq_sectors(req));struct bio *bio = req->bio; +/* User space decides how much metadata a passthrough request has */ +if (blk_rq_is_passthrough (req)) + len =bio_integrity (bio)->bip_iter.bi_size; + iter->iter = (struct blk_map_iter ) { .bio = bio, .iter =bio_integrity (bio)->bip_iter,
Signature · 1 line
2.56.0
Re: [PATCH 33/98] blk-mq-dma: map the metadata a passthrough request has
The subject line does not parse.
block/blk-mq-dma.c static bool __blk_map_iter_next(struct blk_map_iter *iter) · line 17
Quoted 9 lines
index
e0e7a18790b5..5df704bbd98a100644block/blk-mq-dma.c +2
-0@@ -13,6 +13,10 @@ static bool __blk_map_iter_next(struct blk_map_iter *iter)if (!iter->bio || !iter->bio->bi_next)return false; +/* All the metadata of a passthrough request is in its first bio */ +if (iter->is_integrity && !bio_integrity (iter->bio->bi_next))
The comment does not make sense. We're at a layer where we don't know if this is passthrough or not. And the block layer should make sure we never merge integrity vs non-integrity requests into a single requests. If the passthrough code breaks that somehow it needs to be fixed there.
block/blk-mq-dma.c bool blk_rq_integrity_dma_map_iter_start(struct request *req, · line 357
Quoted 12 lines
block/blk-mq-dma.c +5
-0+return false; + iter->bio = iter->bio->bi_next;if (iter->is_integrity) { iter->iter =bio_integrity (iter->bio)->bip_iter;@@ -348,6 +352,10 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req, blk_rq_sectors(req));struct bio *bio = req->bio; +/* User space decides how much metadata a passthrough request has */ +if (blk_rq_is_passthrough (req)) + len =bio_integrity (bio)->bip_iter.bi_size;
DMA mapping helpers don't have any business looking at the passthrough flag. But this should work for kernel-generated I/O just fine as far as I can tell.
Jens Axboe
maintainerPATCH 34/98 block: don't enter reclaim under rq_qos_mutex
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
ioc_qos_write() and the wbt_lat_usec writers take rq_qos_mutex with the queue frozen, blkcg_deactivate_policy() does the same with blkcg_mutex, so nothing holding either may wait for IO. blkg_conf_prep() got GFP_NOIO for the blkg in commit 5d726c4dbeed, but the radix_tree_preload() right after it, the policy init functions and the GFP_KERNEL fallback in blkcg_activate_policy() can still enter reclaim there. lockdep reports the cycle on the first write to io.max.
Use GFP_NOIO for all of them.
--- block/blk-cgroup.c | 13 +++++++++---- block/blk-iocost.c | 5 +++-- block/blk-iolatency.c | 3 ++- block/blk-throttle.c | 3 ++- 4 files changed, 16 insertions(+), 8 deletions(-)
block/blk-cgroup.c +9 -4
@@ -884,7 +884,11 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, parent =blkcg_parent (parent); } -/* Drop locks to do new blkg allocation with GFP_KERNEL. */ +/* +* Drop the spinlock to allocate the new blkg. rq_qos_mutex and +* blkcg_mutex stay held, and both are taken with the queue +* frozen elsewhere: GFP_NOIO. +*/ spin_unlock_irq (&q->queue_lock); new_blkg =blkg_alloc (pos, disk,GFP_NOIO );
@@ -893,7 +897,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, goto fail_exit; } -if (radix_tree_preload (GFP_KERNEL )) { +if (radix_tree_preload (GFP_NOIO )) {blkg_free (new_blkg); ret = -ENOMEM ; goto fail_exit;
@@ -1648,7 +1652,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)if (!pd) {/* * GFP_NOWAIT failed. Free the existing one and -* prealloc for @blkg w/ GFP_KERNEL. +* prealloc for @blkg w/ GFP_NOIO: blkcg_mutex is held, +* and a bio based queue has not been frozen above. */ if (pinned_blkg)blkg_put (pinned_blkg);
@@ -1660,7 +1665,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)if (pd_prealloc) pol->pd_free_fn (pd_prealloc); pd_prealloc = pol->pd_alloc_fn (disk, blkg->blkcg, -GFP_KERNEL ); +GFP_NOIO );if (pd_prealloc) goto retry;else
block/blk-iocost.c +3 -2
@@ -2938,11 +2938,12 @@ static int blk_iocost_init(struct gendisk *disk)struct ioc *ioc;int i, cpu, ret; - ioc =kzalloc_obj (*ioc); +/* The caller holds rq_qos_mutex, which is taken with the queue frozen */ + ioc =kzalloc_obj (*ioc,GFP_NOIO );if (!ioc)return -ENOMEM ; - ioc->pcpu_stat = alloc_percpu(struct ioc_pcpu_stat); + ioc->pcpu_stat = alloc_percpu_gfp(struct ioc_pcpu_stat, GFP_NOIO);if (!ioc->pcpu_stat) {kfree (ioc);return -ENOMEM ;
block/blk-iolatency.c +2 -1
@@ -761,7 +761,8 @@ static int blk_iolatency_init(struct gendisk *disk)struct blk_iolatency *blkiolat;int ret; - blkiolat =kzalloc_obj (*blkiolat); +/* The caller holds rq_qos_mutex, which is taken with the queue frozen */ + blkiolat =kzalloc_obj (*blkiolat,GFP_NOIO );if (!blkiolat)return -ENOMEM ;
block/blk-throttle.c +2 -1
@@ -1322,7 +1322,8 @@ static int blk_throtl_init(struct gendisk *disk)unsigned int memflags;int ret; - td =kzalloc_node (sizeof (*td),GFP_KERNEL , q->node); +/* The caller holds rq_qos_mutex, which is taken with the queue frozen */ + td =kzalloc_node (sizeof (*td),GFP_NOIO , q->node);if (!td)return -ENOMEM ;
Signature · 1 line
2.56.0
Re: [PATCH 34/98] block: don't enter reclaim under rq_qos_mutex
Quoted 8 lines · Jens Axboe
ioc_qos_write()and thewbt_lat_usecwriters takerq_qos_mutexwith the queue frozen,blkcg_deactivate_policy()does the same withblkcg_mutex, so nothing holding either may wait for IO.blkg_conf_prep()gotGFP_NOIOfor the blkg in commit5d726c4dbeed, but theradix_tree_preload()right after it, the policy init functions and theGFP_KERNELfallback inblkcg_activate_policy()can still enter reclaim there. lockdep reports the cycle on the first write to io.max.
Quoted 7 lines
Use
GFP_NOIOfor all of them.Fixes 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")Fixes 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy")Cc Tejun Heo <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thanks.
--
tejun
Jens Axboe
maintainerPATCH 35/98 blktrace: remove the debugfs files before the channel
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_trace_free() closes the relay channel, which frees it, and removes the dropped and msg files afterwards, while both of them read bt->rchan. Remove the files first, which waits for their readers, then close the channel.
--- kernel/trace/blktrace.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
kernel/trace/blktrace.c +8 -9
@@ -468,18 +468,17 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,static void blk_trace_free (struct request_queue *q,struct blk_trace *bt) { -relay_close (bt->rchan); -/* * If 'bt->dir' is not set, then both 'dropped' and 'msg' are created -* under 'q->debugfs_dir', thus lookup and remove them. +* under 'q->debugfs_dir', thus lookup and remove them. They use the +* relay channel: wait for their users before it goes away. */ -if (!bt->dir) { -debugfs_lookup_and_remove ("dropped" , q->debugfs_dir); -debugfs_lookup_and_remove ("msg" , q->debugfs_dir); - }else { -debugfs_remove (bt->dir); - } +debugfs_lookup_and_remove ("dropped" , bt->dir ?: q->debugfs_dir); +debugfs_lookup_and_remove ("msg" , bt->dir ?: q->debugfs_dir); + +relay_close (bt->rchan); +debugfs_remove (bt->dir); +free_percpu (bt->sequence);free_percpu (bt->msg_data);kfree (bt);
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 36/98 blk-mq: quiesce a queue that joins a quiesced tag set
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_mq_quiesce_tagset() only quiesces the queues on the list at that moment. A queue added before blk_mq_unquiesce_tagset() runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning in blk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.
Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be tag_list_lock, see commit 59e25ef2b413.
--- block/blk-mq.c | 22 ++++++++++++++++++++++ include/linux/blk-mq.h | 6 ++++++ 2 files changed, 28 insertions(+)
block/blk-mq.c +22 -0
@@ -336,12 +336,15 @@ void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set) {struct request_queue *q; +spin_lock (&set->quiesce_lock); + set->quiesce_depth++;rcu_read_lock ();list_for_each_entry_rcu (q, &set->tag_list, tag_set_list) {if (!blk_queue_skip_tagset_quiesce (q))blk_mq_quiesce_queue_nowait (q); }rcu_read_unlock (); +spin_unlock (&set->quiesce_lock);blk_mq_wait_quiesce_done (set); }
@@ -351,12 +354,16 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set) {struct request_queue *q; +spin_lock (&set->quiesce_lock); +if (set->quiesce_depth) + set->quiesce_depth--;rcu_read_lock ();list_for_each_entry_rcu (q, &set->tag_list, tag_set_list) {if (!blk_queue_skip_tagset_quiesce (q))blk_mq_unquiesce_queue (q); }rcu_read_unlock (); +spin_unlock (&set->quiesce_lock); }EXPORT_SYMBOL_GPL (blk_mq_unquiesce_tagset);
@@ -4346,6 +4353,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)static void blk_mq_add_queue_tag_set (struct blk_mq_tag_set *set,struct request_queue *q) { +unsigned int i; +mutex_lock (&set->tag_list_lock);/*
@@ -4359,7 +4368,18 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, }if (set->flags &BLK_MQ_F_TAG_QUEUE_SHARED )queue_set_hctx_shared (q, true); + +/* +* A queue that joins a quiesced tag set is quiesced like the others, +* and blk_mq_unquiesce_tagset() finds what it is going to undo. +*/ +spin_lock (&set->quiesce_lock);list_add_tail_rcu (&q->tag_set_list, &set->tag_list); +if (!blk_queue_skip_tagset_quiesce (q)) { +for (i =0 ; i < set->quiesce_depth; i++) +blk_mq_quiesce_queue_nowait (q); + } +spin_unlock (&set->quiesce_lock);mutex_unlock (&set->tag_list_lock); }
@@ -4928,6 +4948,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)mutex_init (&set->tag_list_lock);INIT_LIST_HEAD (&set->tag_list); +spin_lock_init (&set->quiesce_lock); + set->quiesce_depth =0 ;return 0 ;
include/linux/blk-mq.h +6 -0
@@ -523,6 +523,10 @@ enum hctx_type {* @tag_list_lock: Serializes tag_list accesses. * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. +* @quiesce_lock: Serializes blk_mq_[un]quiesce_tagset() against each other +* and against a request queue being added to @tag_list. +* @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without +* blk_mq_unquiesce_tagset(). * @srcu: Use as lock when type of the request queue is blocking * (BLK_MQ_F_BLOCKING). * @update_nr_hwq_lock:
@@ -548,6 +552,8 @@ struct blk_mq_tag_set {struct mutex tag_list_lock;struct list_head tag_list; +spinlock_t quiesce_lock; +unsigned int quiesce_depth;struct srcu_struct *srcu;struct rw_semaphore update_nr_hwq_lock;
Signature · 1 line
2.56.0
Re: [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set
Looks good:
Re: [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set
In reply to Jens AxboeHi Rust for Linux folks,
I only added a small subset of you plus the mailing list, if I missed anybody, please bring them in as necessary.
Quoted 8 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_mq_quiesce_tagset()only quiesces the queues on the list at that moment. A queue added beforeblk_mq_unquiesce_tagset()runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning inblk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.
Quoted 7 lines
Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be
tag_list_lock, see commit59e25ef2b413.Fixes 414dd48e882c ("blk-mq: add tagset quiesce interface")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
After this patch landed in -next as commit 71ec3c7d6db8 ("blk-mq:
quiesce a queue that joins a quiesced tag set"), I am seeing the following build error:
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper allmodconfig rust/kernel.o
error[E0277]: the trait bound `bindings::__BindgenOpaqueArray8<[u8; 72]>: pin_init::Zeroable` is not satisfied
--> rust/kernel/block/mq/tag_set.rs:41:49
|
41 | let tag_set: bindings::blk_mq_tag_set = pin_init::zeroed();
| ^^^^^^^^^^^^^^^^^^ the trait `pin_init::Zeroable` is not implemented for `bindings::__BindgenOpaqueArray8<[u8; 72]>`
|
= help: the following other types implement trait `pin_init::Zeroable`:
()
(A, B, C, D, E, F, G, H, I, J)
(B, C, D, E, F, G, H, I, J)
(C, D, E, F, G, H, I, J)
(D, E, F, G, H, I, J)
(E, F, G, H, I, J)
(F, G, H, I, J)
(G, H, I, J)
and 6359 others
= note: required for `bindings::spinlock` to implement `pin_init::Zeroable`
= note: 1 redundant requirement hidden
= note: required for `blk_mq_tag_set` to implement `pin_init::Zeroable`
note: required by a bound in `pin_init::zeroed`
--> rust/pin-init/src/lib.rs:1688:24
|
1688 | pub const fn zeroed<T: Zeroable>() -> T {
| ^^^^^^^^ required by this bound in `zeroed`error: aborting due to 1 previous error
For more information about this error, try rustc --explain E0277. make[4]: *** [rust/Makefile:799: rust/kernel.o] Error 1
...
Quoted 106 lines
--- block/blk-mq.c | 22 ++++++++++++++++++++++ include/linux/blk-mq.h | 6 ++++++ 2 files changed, 28 insertions(+)block/blk-mq.c +22
-0@@ -336,12 +336,15 @@ void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set) {struct request_queue *q; +spin_lock (&set->quiesce_lock); + set->quiesce_depth++;rcu_read_lock ();list_for_each_entry_rcu (q, &set->tag_list, tag_set_list) {if (!blk_queue_skip_tagset_quiesce (q))blk_mq_quiesce_queue_nowait (q); }rcu_read_unlock (); +spin_unlock (&set->quiesce_lock);blk_mq_wait_quiesce_done (set); }@@ -351,12 +354,16 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set) {struct request_queue *q; +spin_lock (&set->quiesce_lock); +if (set->quiesce_depth) + set->quiesce_depth--;rcu_read_lock ();list_for_each_entry_rcu (q, &set->tag_list, tag_set_list) {if (!blk_queue_skip_tagset_quiesce (q))blk_mq_unquiesce_queue (q); }rcu_read_unlock (); +spin_unlock (&set->quiesce_lock); }EXPORT_SYMBOL_GPL (blk_mq_unquiesce_tagset);@@ -4346,6 +4353,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)static void blk_mq_add_queue_tag_set (struct blk_mq_tag_set *set,struct request_queue *q) { +unsigned int i; +mutex_lock (&set->tag_list_lock);/* @@ -4359,7 +4368,18 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, }if (set->flags &BLK_MQ_F_TAG_QUEUE_SHARED )queue_set_hctx_shared (q, true); + +/* +* A queue that joins a quiesced tag set is quiesced like the others, +* and blk_mq_unquiesce_tagset() finds what it is going to undo. +*/ +spin_lock (&set->quiesce_lock);list_add_tail_rcu (&q->tag_set_list, &set->tag_list); +if (!blk_queue_skip_tagset_quiesce (q)) { +for (i =0 ; i < set->quiesce_depth; i++) +blk_mq_quiesce_queue_nowait (q); + } +spin_unlock (&set->quiesce_lock);mutex_unlock (&set->tag_list_lock); }@@ -4928,6 +4948,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)mutex_init (&set->tag_list_lock);INIT_LIST_HEAD (&set->tag_list); +spin_lock_init (&set->quiesce_lock); + set->quiesce_depth =0 ;return 0 ;include/linux/blk-mq.h +6
-0@@ -523,6 +523,10 @@ enum hctx_type {* @tag_list_lock: Serializes tag_list accesses. * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. +* @quiesce_lock: Serializes blk_mq_[un]quiesce_tagset() against each other +* and against a request queue being added to @tag_list. +* @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without +* blk_mq_unquiesce_tagset(). * @srcu: Use as lock when type of the request queue is blocking * (BLK_MQ_F_BLOCKING). * @update_nr_hwq_lock: @@ -548,6 +552,8 @@ struct blk_mq_tag_set {struct mutex tag_list_lock;struct list_head tag_list; +spinlock_t quiesce_lock; +unsigned int quiesce_depth;struct srcu_struct *srcu;struct rw_semaphore update_nr_hwq_lock;Signature · 1 line
2.56.0
Signature · 2 lines
Cheers, Nathan
Andreas Hindborg
reviewerRe: [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set
Quoted 52 lines · Nathan Chancellor
Hi Rust for Linux folks,
I only added a small subset of you plus the mailing list, if I missed anybody, please bring them in as necessary.
Quoted 8 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_mq_quiesce_tagset()only quiesces the queues on the list at that moment. A queue added beforeblk_mq_unquiesce_tagset()runs unquiesced, and then gets unquiesced without ever having been quiesced, which trips the warning inblk_mq_unquiesce_queue(). NVMe gets there when a controller fails while its namespaces are being scanned.Quoted 7 lines
Count the quiesce calls in the tag set and quiesce a joining queue that many times, under a spinlock that also serializes the two functions. It can't be
tag_list_lock, see commit59e25ef2b413.Fixes 414dd48e882c ("blk-mq: add tagset quiesce interface")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>After this patch landed in -next as commit
71ec3c7d6db8("blk-mq:
quiesce a queue that joins a quiesced tag set"), I am seeing the following build error:$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper allmodconfig rust/kernel.o error[E0277]: the trait bound `bindings::__BindgenOpaqueArray8<[u8; 72]>: pin_init::Zeroable` is not satisfied --> rust/kernel/block/mq/tag_set.rs:41:49 | 41 | let tag_set: bindings::blk_mq_tag_set = pin_init::zeroed(); | ^^^^^^^^^^^^^^^^^^ the trait `pin_init::Zeroable` is not implemented for `bindings::__BindgenOpaqueArray8<[u8; 72]>` | = help: the following other types implement trait `pin_init::Zeroable`: () (A, B, C, D, E, F, G, H, I, J) (B, C, D, E, F, G, H, I, J) (C, D, E, F, G, H, I, J) (D, E, F, G, H, I, J) (E, F, G, H, I, J) (F, G, H, I, J) (G, H, I, J) and 6359 others = note: required for `bindings::spinlock` to implement `pin_init::Zeroable` = note: 1 redundant requirement hidden = note: required for `blk_mq_tag_set` to implement `pin_init::Zeroable` note: required by a bound in `pin_init::zeroed` --> rust/pin-init/src/lib.rs:1688:24 | 1688 | pub const fn zeroed<T: Zeroable>() -> T { | ^^^^^^^^ required by this bound in `zeroed`
Quoted 6 lines
error: aborting due to 1 previous error
For more information about this error, try
rustc --explain E0277. make[4]: *** [rust/Makefile:799:rust/kernel.o] Error 1
...
Thanks for reporting. This was fixed over here [1].
Best regards,
Andreas Hindborg
Jens Axboe
maintainerPATCH 37/98 blk-mq: exit the hctxs of a queue under tag_list_lock
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_mq_update_nr_hw_queues() exits and replaces the hctxs of every queue on set->tag_list under tag_list_lock. blk_mq_exit_queue() exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits. del_gendisk() is kept away by update_nr_hwq_lock, disk_release() of a never added disk and blk_mq_destroy_queue() are not, and nvme_alloc_ns() drops such a disk when it finds the queues frozen by a reset.
Hold tag_list_lock for both steps.
--- block/blk-mq.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-mq.c +8 -2
@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; -mutex_lock (&set->tag_list_lock); +lockdep_assert_held (&set->tag_list_lock); +list_del_rcu (&q->tag_set_list);if (list_is_singular (&set->tag_list)) {/* just transitioned to unshared */
@@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)/* update existing queue */ blk_mq_update_tag_set_shared (set, false); } -mutex_unlock (&set->tag_list_lock); } static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; +/* +* As long as the queue is on set->tag_list, its hctxs belong to +* blk_mq_update_nr_hw_queues() as well, which exits and replaces them. +*/ +mutex_lock (&set->tag_list_lock);/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */ blk_mq_exit_hw_queues (q, set, set->nr_hw_queues);/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */ blk_mq_del_queue_tag_set (q); +mutex_unlock (&set->tag_list_lock); }static int __blk_mq_alloc_rq_maps (struct blk_mq_tag_set *set)
Signature · 1 line
2.56.0
Re: [PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock
Quoted 11 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_mq_update_nr_hw_queues()exits and replaces the hctxs of every queue onset->tag_listundertag_list_lock.blk_mq_exit_queue()exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits.del_gendisk()is kept away byupdate_nr_hwq_lock,disk_release()of a never added disk andblk_mq_destroy_queue()are not, andnvme_alloc_ns()drops such a disk when it finds the queues frozen by a reset.
Quoted 1 line
Hold
tag_list_lockfor both steps.
I'm a bit worried about cpu hotplug lock interaction as that always lead to the craziest imaginable lock chains, but otherwise this looks ok.
Quoted 51 lines
Fixes 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)block/blk-mq.c +8
-2@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; -mutex_lock (&set->tag_list_lock); +lockdep_assert_held (&set->tag_list_lock); +list_del_rcu (&q->tag_set_list);if (list_is_singular (&set->tag_list)) {/* just transitioned to unshared */ @@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)/* update existing queue */ blk_mq_update_tag_set_shared (set, false); } -mutex_unlock (&set->tag_list_lock); } static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; +/* +* As long as the queue is on set->tag_list, its hctxs belong to +* blk_mq_update_nr_hw_queues() as well, which exits and replaces them. +*/ +mutex_lock (&set->tag_list_lock);/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */ blk_mq_exit_hw_queues (q, set, set->nr_hw_queues);/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */ blk_mq_del_queue_tag_set (q); +mutex_unlock (&set->tag_list_lock); }static int __blk_mq_alloc_rq_maps (struct blk_mq_tag_set *set)Signature · 1 line
2.56.0
---end quoted text---
Nilay Shroff
Reviewed-byRe: [PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_mq_update_nr_hw_queues()exits and replaces the hctxs of every queue onset->tag_listundertag_list_lock.blk_mq_exit_queue()exits its hctxs without the lock and only then takes the queue off the list, so an update running at the same time exits them again and sets up new ones that nobody exits.del_gendisk()is kept away byupdate_nr_hwq_lock,disk_release()of a never added disk andblk_mq_destroy_queue()are not, andnvme_alloc_ns()drops such a disk when it finds the queues frozen by a reset.
Quoted 6 lines
Quoted 1 line
Hold
tag_list_lockfor both steps.I'm a bit worried about cpu hotplug lock interaction as that always lead to the craziest imaginable lock chains, but otherwise this looks ok.
Yes, CPU hotplug lock interactions have caused numerous lockdep splats in the past. However, looking at this change, the lock chains appear to remain consistent. I reviewed all possible paths leading to hctx exit and cpu_hotplug_lock and those appears to be consistent.
It seems the new order is consistent across all relevant paths:
blk_mq_update_nr_hw_queues(): update_nr_hwq_lock (write) -> tag_list_lock.
Registered-disk teardown: update_nr_hwq_lock (read) -> tag_list_lock.
Never-registered disk release and blk_mq_destroy_queue(): tag_list_lock only.
HCTX exit callbacks already run under tag_list_lock during hardware-queue resizing, so the new exit path does not introduce a new locking context.
CPU-hotplug removal remains tag_list_lock -> blk_mq_cpuhp_lock -> cpu_hotplug_lock, the same order already used by the resize path; there is no reverse acquisition.
So With that,
Quoted 52 lines
Quoted 51 lines
Fixes 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)block/blk-mq.c +8
-2@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; -mutex_lock (&set->tag_list_lock); +lockdep_assert_held (&set->tag_list_lock); +list_del_rcu (&q->tag_set_list);if (list_is_singular (&set->tag_list)) {/* just transitioned to unshared */ @@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)/* update existing queue */ blk_mq_update_tag_set_shared (set, false); } -mutex_unlock (&set->tag_list_lock); } static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set; +/* +* As long as the queue is on set->tag_list, its hctxs belong to +* blk_mq_update_nr_hw_queues() as well, which exits and replaces them. +*/ +mutex_lock (&set->tag_list_lock);/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */ blk_mq_exit_hw_queues (q, set, set->nr_hw_queues);/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */ blk_mq_del_queue_tag_set (q); +mutex_unlock (&set->tag_list_lock); }static int __blk_mq_alloc_rq_maps (struct blk_mq_tag_set *set)Signature · 1 line
2.56.0---end quoted text---
Jens Axboe
maintainerPATCH 38/98 blk-mq: set up a queue under tag_list_lock
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
blk_mq_init_allocated_queue() only takes tag_list_lock at the end, when it adds the queue to the set. Before that it sizes the queue by set->nr_hw_queues, hands out set->tags[], maps by set->map[], and blk_mq_map_swqueue() frees tags of the set. An update clearing the maps at that time makes the new queue free tags that all the other queues share. An NVMe controller reset during a namespace scan does exactly that.
Take the lock before the first look at the set. An update can be part of error handling, so the allocations under the lock must not wait for IO:memalloc_noio_save(), as in add_disk_fwnode().
--- block/blk-mq.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
block/blk-mq.c +18 -3
@@ -4355,7 +4355,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, {unsigned int i; -mutex_lock (&set->tag_list_lock); +lockdep_assert_held (&set->tag_list_lock);/* * Check to see if we're transitioning to shared (from 1 to 2 queues).
@@ -4380,8 +4380,6 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,blk_mq_quiesce_queue_nowait (q); }spin_unlock (&set->quiesce_lock); - -mutex_unlock (&set->tag_list_lock); }/* All allocations will be freed in release handler of q->mq_kobj */
@@ -4658,6 +4656,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,int blk_mq_init_allocated_queue (struct blk_mq_tag_set *set,struct request_queue *q) { +unsigned int memflags; +/* mark the queue as mq asap */ q->mq_ops = set->ops;
@@ -4676,6 +4676,16 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,INIT_LIST_HEAD (&q->unused_hctx_list);spin_lock_init (&q->unused_hctx_lock); +/* +* Keep blk_mq_update_nr_hw_queues() away until the queue is on +* set->tag_list: it changes the number of hardware queues, the tags and +* the maps, and blk_mq_map_swqueue() allocates and frees tags of the set. +* An update can be part of the error handling of a driver, so it must +* not have to wait for somebody who waits for I/O. +*/ + memflags =memalloc_noio_save (); +mutex_lock (&set->tag_list_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;
@@ -4696,9 +4706,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,blk_mq_init_cpu_queues (q, set->nr_hw_queues);blk_mq_map_swqueue (q);blk_mq_add_queue_tag_set (set, q); + +mutex_unlock (&set->tag_list_lock); +memalloc_noio_restore (memflags);return 0 ; err_hctxs: +mutex_unlock (&set->tag_list_lock); +memalloc_noio_restore (memflags);blk_mq_release (q); err_exit: q->mq_ops =NULL ;
Signature · 1 line
2.56.0
Re: [PATCH 38/98] blk-mq: set up a queue under tag_list_lock
Quoted 53 lines · Jens Axboe
From: Josef Bacik <[email protected]>
blk_mq_init_allocated_queue()only takestag_list_lockat the end, when it adds the queue to the set. Before that it sizes the queue byset->nr_hw_queues, hands out set->tags[], maps by set->map[], andblk_mq_map_swqueue()frees tags of the set. An update clearing the maps at that time makes the new queue free tags that all the other queues share. An NVMe controller reset during a namespace scan does exactly that.Take the lock before the first look at the set. An update can be part of error handling, so the allocations under the lock must not wait for IO:
memalloc_noio_save(), as inadd_disk_fwnode().Fixes 868f2f0b7206 ("blk-mq: dynamic h/w context count")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)block/blk-mq.c +3
-3@@ -4355,7 +4355,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, {unsigned int i; -mutex_lock (&set->tag_list_lock); +lockdep_assert_held (&set->tag_list_lock);/* * Check to see if we're transitioning to shared (from 1 to 2 queues). @@ -4380,8 +4380,6 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,blk_mq_quiesce_queue_nowait (q); }spin_unlock (&set->quiesce_lock); - -mutex_unlock (&set->tag_list_lock); }/* All allocations will be freed in release handler of q->mq_kobj */ @@ -4658,6 +4656,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,int blk_mq_init_allocated_queue (struct blk_mq_tag_set *set,struct request_queue *q) { +unsigned int memflags; +/* mark the queue as mq asap */ q->mq_ops = set->ops;
block/blk-mq.c int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, · line 4682
Quoted 7 lines
block/blk-mq.c +4
-0@@ -4676,6 +4676,16 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,INIT_LIST_HEAD (&q->unused_hctx_list);spin_lock_init (&q->unused_hctx_lock); +/* +* Keep blk_mq_update_nr_hw_queues() away until the queue is on +* set->tag_list: it changes the number of hardware queues, the tags and +* the maps, and blk_mq_map_swqueue() allocates and frees tags of the set.
Overly long line inside an otherwise nicely formatted block comment :(
Otherwise similarly a slight worry about lock chains, otherwise this looks fine.
Jens Axboe
maintainerPATCH 39/98 rnbd-srv: fix use-after-free on two CLOSE messages
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
process_msg_close() looks the sess_dev up with a kref, drops it right away, and only then takes the session mutex for the teardown. A session has several connections handled by different workers, so two CLOSE messages for the same ID both finish the lookup before either one tears the sess_dev down, and the second one works on freed memory. Any host that can open a device on the server can do that.
Take the mutex first and look the ID up with xa_load() under it. The second CLOSE then finds nothing.
--- drivers/block/rnbd/rnbd-srv.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -7
@@ -360,14 +360,15 @@ static void process_msg_close(struct rnbd_srv_session *srv_sess,trace_process_msg_close (srv_sess, close_msg); - sess_dev =rnbd_get_sess_dev (le32_to_cpu (close_msg->device_id), - srv_sess); -if (IS_ERR (sess_dev)) -return ; - -rnbd_put_sess_dev (sess_dev); +/* +* Each teardown of a sess_dev runs under srv_sess->lock and erases the +* ID, so an entry that is found under the lock is alive. +*/ mutex_lock (&srv_sess->lock); -rnbd_srv_destroy_dev_session_sysfs (sess_dev); + sess_dev =xa_load (&srv_sess->index_idr, +le32_to_cpu (close_msg->device_id)); +if (sess_dev) +rnbd_srv_destroy_dev_session_sysfs (sess_dev);mutex_unlock (&srv_sess->lock); }
Signature · 1 line
2.56.0
Re: [PATCH 39/98] rnbd-srv: fix use-after-free on two CLOSE messages
Quoted 10 lines · Jens Axboe
From: Chris Mason <[email protected]>
process_msg_close()looks thesess_devup with a kref, drops it right away, and only then takes the session mutex for the teardown. A session has several connections handled by different workers, so two CLOSE messages for the same ID both finish the lookup before either one tears thesess_devdown, and the second one works on freed memory. Any host that can open a device on the server can do that.
Quoted 8 lines
Take the mutex first and look the ID up with
xa_load()under it. The second CLOSE then finds nothing.Fixes 2de6c8de192b ("block/rnbd: server: main functionality")Cc "Md. Haris Iqbal" <[email protected]>Cc Jack Wang <[email protected]>Signed-off-by Chris Mason <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thx!
Quoted 34 lines
--- drivers/block/rnbd/rnbd-srv.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)drivers/block/rnbd/rnbd-srv.c +8
-8@@ -360,14 +360,15 @@ static void process_msg_close(struct rnbd_srv_session *srv_sess,trace_process_msg_close (srv_sess, close_msg); - sess_dev =rnbd_get_sess_dev (le32_to_cpu (close_msg->device_id), - srv_sess); -if (IS_ERR (sess_dev)) -return ; - -rnbd_put_sess_dev (sess_dev); +/* +* Each teardown of a sess_dev runs under srv_sess->lock and erases the +* ID, so an entry that is found under the lock is alive. +*/ mutex_lock (&srv_sess->lock); -rnbd_srv_destroy_dev_session_sysfs (sess_dev); + sess_dev =xa_load (&srv_sess->index_idr, +le32_to_cpu (close_msg->device_id)); +if (sess_dev) +rnbd_srv_destroy_dev_session_sysfs (sess_dev);mutex_unlock (&srv_sess->lock); } --2.56.0
Jens Axboe
maintainerPATCH 40/98 rnbd-srv: publish the sess_dev at the end of OPEN
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
rnbd_sess_dev_alloc() stores the empty sess_dev in the XArray as its first step. An IO message with the new ID from another connection can find it before bdev_file is set, or hold a reference while a failing OPEN frees it after a mere RCU grace period. Both are reachable from any host that can open a device.
Reserve the ID with a NULL entry and store the sess_dev once nothing can fail anymore. The error path still erases the ID, and no longer needs synchronize_rcu().
--- drivers/block/rnbd/rnbd-srv.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/block/rnbd/rnbd-srv.c +8 -2
@@ -431,7 +431,7 @@ static struct rnbd_srv_sess_devif (!sess_dev)return ERR_PTR (-ENOMEM ); - error =xa_alloc (&srv_sess->index_idr, &sess_dev->device_id, sess_dev, + error =xa_alloc (&srv_sess->index_idr, &sess_dev->device_id,NULL , xa_limit_32b,GFP_NOWAIT );if (error <0 ) {pr_warn ("Allocating idr failed, err: %d\n" , error);
@@ -802,6 +802,13 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,list_add (&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);mutex_unlock (&srv_dev->lock); +/* +* Nothing fails from here on, so let I/O messages find the sess_dev. +* The ID is reserved, so this store allocates nothing. +*/ +xa_store (&srv_sess->index_idr, srv_sess_dev->device_id, srv_sess_dev, +GFP_KERNEL ); +rnbd_srv_info (srv_sess_dev,"Opened device '%s'\n" , srv_dev->name);kfree (full_path);
@@ -813,7 +820,6 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess, free_srv_sess_dev:xa_erase (&srv_sess->index_idr, srv_sess_dev->device_id); -synchronize_rcu ();kfree (srv_sess_dev); srv_dev_put:if (open_msg->access_mode !=RNBD_ACCESS_RO ) {
Signature · 1 line
2.56.0
Re: [PATCH 40/98] rnbd-srv: publish the sess_dev at the end of OPEN
Quoted 13 lines · Jens Axboe
From: Chris Mason <[email protected]>
rnbd_sess_dev_alloc()stores the emptysess_devin the XArray as its first step. An IO message with the new ID from another connection can find it beforebdev_fileis set, or hold a reference while a failing OPEN frees it after a mere RCU grace period. Both are reachable from any host that can open a device.Reserve the ID with a NULL entry and store the
sess_devonce nothing can fail anymore. The error path still erases the ID, and no longer needssynchronize_rcu().
Quoted 5 lines
Fixes 2de6c8de192b ("block/rnbd: server: main functionality")Cc "Md. Haris Iqbal" <[email protected]>Cc Jack Wang <[email protected]>Signed-off-by Chris Mason <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Thx!
Quoted 42 lines
--- drivers/block/rnbd/rnbd-srv.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/block/rnbd/rnbd-srv.c +8
-3@@ -431,7 +431,7 @@ static struct rnbd_srv_sess_devif (!sess_dev)return ERR_PTR (-ENOMEM ); - error =xa_alloc (&srv_sess->index_idr, &sess_dev->device_id, sess_dev, + error =xa_alloc (&srv_sess->index_idr, &sess_dev->device_id,NULL , xa_limit_32b,GFP_NOWAIT );if (error <0 ) {pr_warn ("Allocating idr failed, err: %d\n" , error);@@ -802,6 +802,13 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,list_add (&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);mutex_unlock (&srv_dev->lock); +/* +* Nothing fails from here on, so let I/O messages find the sess_dev. +* The ID is reserved, so this store allocates nothing. +*/ +xa_store (&srv_sess->index_idr, srv_sess_dev->device_id, srv_sess_dev, +GFP_KERNEL ); +rnbd_srv_info (srv_sess_dev,"Opened device '%s'\n" , srv_dev->name);kfree (full_path);@@ -813,7 +820,6 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess, free_srv_sess_dev:xa_erase (&srv_sess->index_idr, srv_sess_dev->device_id); -synchronize_rcu ();kfree (srv_sess_dev); srv_dev_put:if (open_msg->access_mode !=RNBD_ACCESS_RO ) { --2.56.0
Jens Axboe
maintainerPATCH 41/98 zram: check a bio's range against the device size
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
bio_check_eod() counts whole sectors, so a bio shorter than a sector, or one whose tail is shorter than a sector, passes it even at or past the end of the device. zram takes such a piece to the slot of its sector and reads, locks and modifies an entry outside the table, or dereferences a NULL table on a device without a size. valid_io_request() refused these until commit 9fe95babc742 removed it, and any in-kernel builder of bios that takes size and sector from a peer can feed them in.
Check the range at the top of zram_submit_bio(), with the size rounded up to sectors, and fail with an IO error like bio_check_eod() does.
--- drivers/block/zram/zram_drv.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
drivers/block/zram/zram_drv.c +22 -0
@@ -2794,6 +2794,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)bio_endio (bio); } +/* +* bio_check_eod() counts whole sectors: it does not look at a bio that is +* shorter than a sector, nor at a tail that is shorter than one. zram takes +* such a piece to the slot of its sector, so round the size up. An empty bio +* has to start inside the device as well: the loops for reads and writes run +* once before their test. +*/ +static bool zram_bio_in_range (struct zram *zram,struct bio *bio) +{ +sector_t nr_sectors =get_capacity (zram->disk); +sector_t start = bio->bi_iter.bi_sector; +sector_t len = ((sector_t )bio->bi_iter.bi_size +SECTOR_SIZE -1 ) >> +SECTOR_SHIFT ; + +return start < nr_sectors && len <= nr_sectors - start; +} +/* * Handler function for all zram I/O requests. */
@@ -2801,6 +2818,11 @@ static void zram_submit_bio(struct bio *bio) {struct zram *zram = bio->bi_bdev->bd_disk->private_data; +if (unlikely (!zram_bio_in_range (zram, bio))) { +bio_io_error (bio); +return ; + } +switch (bio_op (bio)) {case REQ_OP_READ :zram_bio_read (zram, bio);
Signature · 1 line
2.56.0
Re: [PATCH 41/98] zram: check a bio's range against the device size
Quoted 6 lines · Jens Axboe
drivers/block/zram/zram_drv.c +6
-0+/* +* bio_check_eod() counts whole sectors: it does not look at a bio that is +* shorter than a sector, nor at a tail that is shorter than one. zram takes +* such a piece to the slot of its sector, so round the size up. An empty bio +* has to start inside the device as well: the loops for reads and writes run +* once before their test.
drivers/block/zram/zram_drv.c static void zram_bio_write(struct zram *zram, struct bio *bio) · line 2812
Quoted 10 lines
drivers/block/zram/zram_drv.c +10
-0+*/ +static bool zram_bio_in_range (struct zram *zram,struct bio *bio) +{ +sector_t nr_sectors =get_capacity (zram->disk); +sector_t start = bio->bi_iter.bi_sector; +sector_t len = ((sector_t )bio->bi_iter.bi_size +SECTOR_SIZE -1 ) >> +SECTOR_SHIFT ; + +return start < nr_sectors && len <= nr_sectors - start; +}
There are two problems with zram_bio_in_range():
1. It explicitly allows an empty REQ_OP_READ or REQ_OP_WRITE bio (bi_size == 0) as long as start < nr_sectors. Both zram_bio_read() and zram_bio_write() use a do { ... } while (iter.bi_size) loop whose first statement is bio_iter_iovec(bio, iter), which immediately dereferences NULL bio->bi_io_vec on an empty bio.
2. It allows sub-4K / unaligned bios as long as they fall within nr_sectors. However, ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on PAGE_SIZE == 4096 kernels is_partial_io() is hardcoded to return false:
- zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index, bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes of bv_page.
- zram_bvec_write() calls zram_write_page(zram, bvec->bv_page, index), clobbering the rest of the 4 KiB zram slot and compressing up to 4095 bytes of adjacent kernel page memory into the slot where a subsequent read will return it.
Shouldn't this reinstate valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE alignment checks on bi_sector and bi_size (as well as rejecting bi_size == 0)?
Re: [PATCH 41/98] zram: check a bio's range against the device size
In reply to Jens AxboeReview
Quoted 9 lines · Jens Axboe
(quoted diff) +9
-0+static bool zram_bio_in_range(struct zram *zram, struct bio *bio) +{ + sector_t nr_sectors = get_capacity(zram->disk); + sector_t start = bio->bi_iter.bi_sector; + sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >> + SECTOR_SHIFT; + + return start < nr_sectors && len <= nr_sectors - start; +}
There are two problems with zram_bio_in_range():
1. It allows an empty REQ_OP_READ or REQ_OP_WRITE bio (bi_size == 0) as long as start < nr_sectors. Both zram_bio_read() and zram_bio_write() use a do { ... } while (iter.bi_size) loop whose first statement is bio_iter_iovec(bio, iter), dereferencing NULL bio->bi_io_vec on an empty bio. (Empty REQ_OP_FLUSH bios are already completed in submit_bio_noacct() because zram does not set a write cache.)
2. It allows sub-4K / unaligned bios as long as they fall within nr_sectors. ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on PAGE_SIZE == 4096 kernels is_partial_io() is hardcoded to return false:
- zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index, bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes of bv_page.
- zram_bvec_write() calls zram_write_page(zram, bvec->bv_page, index), clobbering the rest of the 4 KiB zram slot and compressing up to 4095 bytes of adjacent kernel page memory into the slot where a subsequent read can read it back.
Reinstating valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE alignment checks on bi_sector and bi_size alongside rejecting bi_size == 0 and out-of-range bios closes all of these cases.
Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
drivers/block/zram/zram_drv.c +22 -0
@@ -2796,6 +2796,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)bio_endio (bio); } +static bool zram_valid_io_request (struct zram *zram,struct bio *bio) +{ +sector_t nr_sectors =get_capacity (zram->disk); +sector_t start = bio->bi_iter.bi_sector; +unsigned int size = bio->bi_iter.bi_size; + +if (!size) +return false; +if (start & (ZRAM_SECTOR_PER_LOGICAL_BLOCK -1 )) +return false; +if (size & (ZRAM_LOGICAL_BLOCK_SIZE -1 )) +return false; + +return start < nr_sectors && + (size >>SECTOR_SHIFT ) <= nr_sectors - start; +} +/* * Handler function for all zram I/O requests. */
@@ -2803,6 +2820,11 @@ static void zram_submit_bio(struct bio *bio) {struct zram *zram = bio->bi_bdev->bd_disk->private_data; +if (unlikely (!zram_valid_io_request (zram, bio))) { +bio_io_error (bio); +return ; + } +switch (bio_op (bio)) {case REQ_OP_READ :zram_bio_read (zram, bio);
Jens Axboe
maintainerPATCH 42/98 block: partitions/mac: compute the sectors in 64 bits
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
start_block and block_count are converted to 512 byte sectors in 32-bit arithmetic, although put_partition() takes a sector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.
--- block/partitions/mac.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/partitions/mac.c +2 -2
@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)if (be16_to_cpu (part->signature) !=MAC_PARTITION_MAGIC )break ;put_partition (state, slot, -be32_to_cpu (part->start_block) * (secsize/512 ), -be32_to_cpu (part->block_count) * (secsize/512 )); + (sector_t )be32_to_cpu (part->start_block) * (secsize/512 ), + (sector_t )be32_to_cpu (part->block_count) * (secsize/512 ));if (!strncasecmp (part->type,"Linux_RAID" ,10 )) state->parts[slot].flags =ADDPART_FLAG_RAID ;
Signature · 1 line
2.56.0
Re: [PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits
Quoted 19 lines · Jens Axboe
From: Josef Bacik <[email protected]>
start_blockandblock_countare converted to 512 byte sectors in 32-bit arithmetic, althoughput_partition()takes asector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/partitions/mac.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)block/partitions/mac.c +0
-0
block/partitions/mac.c int mac_partition(struct parsed_partitions *state) · line 101
Quoted 7 lines
block/partitions/mac.c +2
-2@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)if (be16_to_cpu (part->signature) !=MAC_PARTITION_MAGIC )break ;put_partition (state, slot, -be32_to_cpu (part->start_block) * (secsize/512 ), -be32_to_cpu (part->block_count) * (secsize/512 )); + (sector_t )be32_to_cpu (part->start_block) * (secsize/512 ), + (sector_t )be32_to_cpu (part->block_count) * (secsize/512 ));
PLease just put this into a helper..
Jens Axboe
maintainerRe: [PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Jens Axboe
From: Josef Bacik <[email protected]>
start_blockandblock_countare converted to 512 byte sectors in 32-bit arithmetic, althoughput_partition()takes asector_t. With a block size above 512 bytes anything beyond 2TiB wraps and the partition lands on top of another one.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/partitions/mac.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 14 lines
block/partitions/mac.c int mac_partition(struct parsed_partitions *state) · line 101
Quoted 12 lines
block/partitions/mac.c +2
-2@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)if (be16_to_cpu (part->signature) !=MAC_PARTITION_MAGIC )break ;put_partition (state, slot, -be32_to_cpu (part->start_block) * (secsize/512 ), -be32_to_cpu (part->block_count) * (secsize/512 )); + (sector_t )be32_to_cpu (part->start_block) * (secsize/512 ), + (sector_t )be32_to_cpu (part->block_count) * (secsize/512 ));PLease just put this into a helper..
For this and others, we can add helpers later.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 43/98 blk-mq: free the bitmaps of tags in the SRCU callback
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the iterators under tags_srcu are gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks, blk_mq_tagset_busy_iter() may be about to walk tags that __blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.
Free the bitmaps with the rest in the callback.
--- block/blk-mq-tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq-tag.c +4 -4
@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)kmemleak_free (page_address (page));__free_pages (page, page->private); } +/* An iterator may walk the bitmaps for as long as it may see the tags */ +sbitmap_queue_free (&tags->bitmap_tags); +sbitmap_queue_free (&tags->breserved_tags);kfree (tags); }void blk_mq_free_tags (struct blk_mq_tags *tags) { -sbitmap_queue_free (&tags->bitmap_tags); -sbitmap_queue_free (&tags->breserved_tags); -/* if tags pages is not allocated yet, free tags directly */ if (list_empty (&tags->page_list)) { -kfree (tags); +blk_mq_free_tags_callback (&tags->rcu_head);return ; }
Signature · 1 line
2.56.0
Re: [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback
Quoted 35 lines · Jens Axboe
From: Josef Bacik <[email protected]>
Commit
ad0d05dbddc1keeps structblk_mq_tagsand the requests until the iterators undertags_srcuare gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks,blk_mq_tagset_busy_iter()may be about to walk tags that__blk_mq_update_nr_hw_queues()frees.sbitmap_free()clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.Free the bitmaps with the rest in the callback.
Fixes ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq-tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)block/blk-mq-tag.c +3
-0@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)kmemleak_free (page_address (page));__free_pages (page, page->private); } +/* An iterator may walk the bitmaps for as long as it may see the tags */ +sbitmap_queue_free (&tags->bitmap_tags); +sbitmap_queue_free (&tags->breserved_tags);kfree (tags); }void blk_mq_free_tags (struct blk_mq_tags *tags) {
block/blk-mq-tag.c static void blk_mq_free_tags_callback(struct rcu_head *head) · line 639
Quoted 10 lines
block/blk-mq-tag.c +1
-4-sbitmap_queue_free (&tags->bitmap_tags); -sbitmap_queue_free (&tags->breserved_tags); -/* if tags pages is not allocated yet, free tags directly */ if (list_empty (&tags->page_list)) { -kfree (tags); +blk_mq_free_tags_callback (&tags->rcu_head);return ; }
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback
In reply to Jens AxboeQuoted 24 lines · Jens Axboe
From: Josef Bacik <[email protected]>
Commit
ad0d05dbddc1keeps structblk_mq_tagsand the requests until the iterators undertags_srcuare gone, but the two bitmaps they walk are still freed right away. When the number of hardware queues shrinks,blk_mq_tagset_busy_iter()may be about to walk tags that__blk_mq_update_nr_hw_queues()frees.sbitmap_free()clears the map pointer, so that's a NULL dereference or a use-after-free, depending on the timing.Free the bitmaps with the rest in the callback.
Fixes ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/blk-mq-tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)block/blk-mq-tag.c +0
-0
block/blk-mq-tag.c static void blk_mq_free_tags_callback(struct rcu_head *head) · line 626
Quoted 4 lines
block/blk-mq-tag.c +1
-0@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)kmemleak_free (page_address (page));__free_pages (page, page->private); } +/* An iterator may walk the bitmaps for as long as it may see the tags */
Missing sentence ending and overly long line. Otherwise looks good.
Jens Axboe
maintainerPATCH 44/98 block: apply the segment size to the integrity buffer
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The integrity buffer is a single segment and must not exceed max_segment_size. max_integrity_io_size() applies that limit to the data size instead, which is off by a factor of 64 for 8 bytes of PI per 512 bytes of data. A SCSI host that doesn't set a maximum segment size gets 64KiB, lpfc and qla2xxx among them, so a DIX disk behind one of those does no IO above 64KiB.
Limit the buffer and derive the data size from that, which makes it 4MiB for that case.
--- include/linux/blkdev.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
include/linux/blkdev.h +5 -4
@@ -1819,11 +1819,12 @@ static inline int bio_split_rw_at(struct bio *bio,*/ static inline unsigned int max_integrity_io_size (struct queue_limits *lim) { -u64 max_intervals; + unsigned int buf_size = min_t(unsigned int, lim->max_segment_size, +BLK_INTEGRITY_MAX_SIZE ); - max_intervals =BLK_INTEGRITY_MAX_SIZE / lim->integrity.metadata_size; -return min_t (u64, lim->max_segment_size, - max_intervals << lim->integrity.interval_exp); +return min_t (u64,UINT_MAX , + (u64)(buf_size / lim->integrity.metadata_size) << + lim->integrity.interval_exp); }#define DEFINE_IO_COMP_BATCH (name) struct io_comp_batch name = { }
Signature · 1 line
2.56.0
Re: [PATCH 44/98] block: apply the segment size to the integrity buffer
Quoted 8 lines · Jens Axboe
From: Josef Bacik <[email protected]>
The integrity buffer is a single segment and must not exceed
max_segment_size.max_integrity_io_size()applies that limit to the data size instead, which is off by a factor of 64 for 8 bytes of PI per 512 bytes of data. A SCSI host that doesn't set a maximum segment size gets 64KiB, lpfc and qla2xxx among them, so a DIX disk behind one of those does no IO above 64KiB.
Looks good:
Jens Axboe
maintainerPATCH 45/98 nvmet: don't truncate the length of big discards
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_bdev_execute_discard() shifts the 32-bit nlb of a DSM range into sectors before widening it to sector_t, so on a 4k namespace the top bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards nothing, and both complete with success. The target announces no DMRSL, and hosts other than Linux (and nvme dsm passthrough) do send ranges that big.
Widen before the shift.
--- drivers/nvme/target/io-cmd-bdev.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/io-cmd-bdev.c +2 -1
@@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)if (status)break ; - nr_sects =le32_to_cpu (range.nlb) << (ns->blksize_shift -9 ); + nr_sects = (sector_t )le32_to_cpu (range.nlb) << + (ns->blksize_shift -9 );__blkdev_issue_discard (ns->bdev,nvmet_lba_to_sect (ns, range.slba), nr_sects,GFP_KERNEL , &bio);
Signature · 1 line
2.56.0
Re: [PATCH 45/98] nvmet: don't truncate the length of big discards
Quoted 26 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_bdev_execute_discard()shifts the 32-bit nlb of a DSM range into sectors before widening it tosector_t, so on a 4k namespace the top bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards nothing, and both complete with success. The target announces no DMRSL, and hosts other than Linux (and nvme dsm passthrough) do send ranges that big.Widen before the shift.
Fixes a07b4970f464 ("nvmet: add a generic NVMe target")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/io-cmd-bdev.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/target/io-cmd-bdev.c +0
-0
drivers/nvme/target/io-cmd-bdev.c static void nvmet_bdev_execute_discard(struct nvmet_req *req) · line 396
Quoted 9 lines
drivers/nvme/target/io-cmd-bdev.c +2
-1@@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)if (status)break ; - nr_sects =le32_to_cpu (range.nlb) << (ns->blksize_shift -9 ); + nr_sects = (sector_t )le32_to_cpu (range.nlb) << + (ns->blksize_shift -9 );__blkdev_issue_discard (ns->bdev,nvmet_lba_to_sect (ns, range.slba), nr_sects,GFP_KERNEL , &bio);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 46/98 nvme-loop: delete the controller if a reset fails
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
If the reconnect of a reset fails, nvme_loop_reset_ctrl_work() tears the controller down by hand with nvme_uninit_ctrl(). Since that was split from nvme_stop_ctrl() it no longer removes the namespaces, which hold references to the controller. A reset while the target no longer accepts the host (allow_any_host 0) is enough to get there: a sysfs reset hangs in kernfs_drain(), an ioctl reset leaves a controller stuck in CONNECTING with its block devices, the cntlid can't be reused, and module unload deletes the controller a second time and hits a use-after-free.
Do what the other fabrics drivers do and call nvme_delete_ctrl(). The delete path quiesces the admin queue and only unquiesces it if the queue is live, which it isn't after a failed reset, so unquiesce it unconditionally before destroying it so a stuck request can't block the freeze.
--- drivers/nvme/target/loop.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/target/loop.c +3 -2
@@ -422,6 +422,8 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)static void nvme_loop_delete_ctrl_host (struct nvme_ctrl *ctrl) {nvme_loop_shutdown_ctrl (to_loop_ctrl (ctrl)); +/* Left quiesced if a failed reset has left no admin queue behind */ +nvme_unquiesce_admin_queue (ctrl);nvme_remove_admin_tag_set (ctrl); }
@@ -485,9 +487,8 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)nvme_cancel_admin_tagset (&ctrl->ctrl);nvme_loop_destroy_admin_queue (ctrl); out_disable: -nvme_remove_admin_tag_set (&ctrl->ctrl);dev_warn (ctrl->ctrl.device,"Removing after reset failure\n" ); -nvme_uninit_ctrl (&ctrl->ctrl); +nvme_delete_ctrl (&ctrl->ctrl); }static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 47/98 nvme: refuse metadata sizes above 255 bytes
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The metadata size of an LBA format is a 16-bit field, but blk_integrity->metadata_size is an unsigned char. nvme_init_integrity()
truncates: a 264-byte format tells the block layer 8, so the integrity buffer is 8 bytes per block while the controller transfers 264, and a read overflows it. 256 bytes becomes 0 and every request fails in nvme_setup_rw().
Treat such a format like the other unsupported ones: capacity 0, passthrough only.
--- drivers/nvme/host/core.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +4 -0
@@ -1867,6 +1867,10 @@ static bool nvme_init_integrity(struct nvme_ns_head *head, !(head->features & NVME_NS_METADATA_SUPPORTED))return nvme_ns_has_pi (head); +/* struct blk_integrity has eight bits for the metadata size. */ +if (head->ms >U8_MAX ) +return false; +switch (head->pi_type) {case NVME_NS_DPS_PI_TYPE3 :switch (head->guard_type) {
Signature · 1 line
2.56.0
Keith Busch
maintainerRe: [PATCH 47/98] nvme: refuse metadata sizes above 255 bytes
drivers/nvme/host/core.c static bool nvme_init_integrity(struct nvme_ns_head *head, · line 1872
Quoted 3 lines · Jens Axboe
drivers/nvme/host/core.c +3
-0+/* struct blk_integrity has eight bits for the metadata size. */ +if (head->ms >U8_MAX ) +return false;
I've never seen a device with 256 or greater metadata, but if the specs allow it, should the block layer accomodate? It's just the queue_limits, so widening the type isn't a big deal; it fits in the existing padding anyway so the struct doesn't even have to grow.
There's potential issues with logical_block_metadata_cap, but that's just user space reporting and doesn't affect the kernel behavior. Seems better than throttling the capacity to 0 for an unsupported format that we can totally support.
Jens Axboe
maintainerPATCH 48/98 nvmet-tcp: rewind the iterator after non-data records
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A TLS record that isn't application data (an Alert, say) lands in whatever buffer was waiting for a PDU header, command data or a digest. nvmet_tcp_tls_record_ok() fails the receive and the bytes aren't counted, but the iterator has already advanced. tls_alert_recv() then expects the iterator at the start of the alert, and if the alert filled the kvec it has stepped off the end of it:
BUG: KASAN: stack-out-of-bounds in tls_alert_recv+0x33/0x170 nvmet_tcp_tls_record_ok+0x139/0x1a0 nvmet_tcp_try_recv_pdu+0x37f/0x12a0
Six bytes of a header followed by an alert from the host is all it takes. For command data the following bytes are stored behind the dropped record and the command runs before its data is complete.
Rewind the iterator by what was received.
--- drivers/nvme/target/tcp.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/nvme/target/tcp.c +7 -5
@@ -1174,7 +1174,7 @@ static inline bool nvmet_tcp_pdu_valid(u8 type) }static int nvmet_tcp_tls_record_ok (struct nvmet_tcp_queue *queue, -struct msghdr *msg,char *cbuf) +struct msghdr *msg,char *cbuf,int len) {struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;u8 ctype, level, description;
@@ -1187,6 +1187,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,case TLS_RECORD_TYPE_DATA :break ;case TLS_RECORD_TYPE_ALERT : +iov_iter_revert (&msg->msg_iter, len);tls_alert_recv (queue->sock->sk, msg, &level, &description);if (level ==TLS_ALERT_LEVEL_FATAL ) { pr_err("queue %d: TLS Alert desc %u\n",
@@ -1200,6 +1201,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,break ;default :/* discard this record type */ +iov_iter_revert (&msg->msg_iter, len);pr_err ("queue %d: TLS record %d unhandled\n" , queue->idx, ctype); ret = -EAGAIN ;
@@ -1228,7 +1230,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)if (unlikely (len <0 ))return len;if (queue->tls_pskid) { - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret; }
@@ -1290,7 +1292,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)return len;if (queue->tls_pskid) { ret =nvmet_tcp_tls_record_ok (cmd->queue, - &cmd->recv_msg, cmd->recv_cbuf); + &cmd->recv_msg, cmd->recv_cbuf, len);if (ret <0 )return ret; }
@@ -1331,7 +1333,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)if (unlikely (len <0 ))return len;if (queue->tls_pskid) { - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret; }
@@ -1803,7 +1805,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)return len; } - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret;
Signature · 1 line
2.56.0
Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records
Quoted 3 lines · Jens Axboe
drivers/nvme/target/tcp.c +1
-0case TLS_RECORD_TYPE_ALERT : +iov_iter_revert (&msg->msg_iter, len);tls_alert_recv (queue->sock->sk, msg, &level, &description);
...
drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1295
Quoted 4 lines
drivers/nvme/target/tcp.c +1
-1if (queue->tls_pskid) { ret =nvmet_tcp_tls_record_ok (cmd->queue, - &cmd->recv_msg, cmd->recv_cbuf); + &cmd->recv_msg, cmd->recv_cbuf, len);
Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT is still broken in two ways here:
1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC initialized with iov_iter_bvec(), whereas tls_alert_recv() (net/handshake/alert.c) unconditionally reads union member msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases bvec->bv_page (struct page *), so tls_alert_recv() reads the first two bytes of struct page instead of the received alert payload and silently ignores fatal alerts when byte 0 != TLS_ALERT_LEVEL_FATAL. (Also, up to 2 bytes of the command's SG page were overwritten before iov_iter_revert().)
2. If queue->left == 1 when a 2-byte TLS alert arrives, sock_recvmsg() returns len == 1, and tls_alert_recv() reads data[0] and data[1] without checking len >= 2.
tls_alert_recv() either needs to handle non-kvec iterators and len < 2 properly, or nvmet-tcp should treat any TLS alert record during live PDU/data transfer as fatal without calling tls_alert_recv() on cmd->recv_msg.
Hannes Reinecke
maintainerRe: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records
Quoted 23 lines · Bart Van Assche
Quoted 3 lines · Jens Axboe
case
TLS_RECORD_TYPE_ALERT:drivers/nvme/target/tcp.c +1
-0+iov_iter_revert (&msg->msg_iter, len);tls_alert_recv (queue->sock->sk, msg, &level, &description);...
drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1295
Quoted 4 lines
if (queue->tls_pskid) { ret = nvmet_tcp_tls_record_ok(cmd->queue,drivers/nvme/target/tcp.c +1
-1- &cmd->recv_msg, cmd->recv_cbuf); + &cmd->recv_msg, cmd->recv_cbuf, len);Rewinding
msg_iterfixes the iterator advance, butTLS_RECORD_TYPE_ALERTis still broken in two ways here:1. In
nvmet_tcp_try_recv_data(),cmd->recv_msg.msg_iteris anITER_BVECinitialized withiov_iter_bvec(), whereastls_alert_recv()(net/handshake/alert.c) unconditionally reads union membermsg->msg_iter.kvec->iov_base. For anITER_BVEC,iov->iov_basealiasesbvec->bv_page(struct page *), sotls_alert_recv()reads the first two bytes of struct page instead of the received alert payload and silently ignores fatal alerts when byte 0 !=TLS_ALERT_LEVEL_FATAL. (Also, up to 2 bytes of the command's SG page were overwritten beforeiov_iter_revert().)
Quoted 9 lines
2. If
queue->left== 1 when a 2-byte TLS alert arrives,sock_recvmsg()returns len == 1, andtls_alert_recv()reads data[0] and data[1] without checking len >= 2.
tls_alert_recv()either needs to handle non-kvec iterators and len < 2 properly, or nvmet-tcp should treat any TLS alert record during live PDU/data transfer as fatal without callingtls_alert_recv()oncmd->recv_msg.
There is a larger rework under way for handling TLS alerts, so we should defer this until the rework is merged.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records
In reply to Jens AxboeQuoted 3 lines · Jens Axboe
(quoted diff) +1
-0case TLS_RECORD_TYPE_ALERT: + iov_iter_revert(&msg->msg_iter, len); tls_alert_recv(queue->sock->sk, msg, &level, &description);
...
Review
Quoted 4 lines
(quoted diff) +1
-1if (queue->tls_pskid) { ret = nvmet_tcp_tls_record_ok(cmd->queue, - &cmd->recv_msg, cmd->recv_cbuf); + &cmd->recv_msg, cmd->recv_cbuf, len);
Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT is still broken in two ways here:
1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC initialized with iov_iter_bvec(), whereas tls_alert_recv() in net/handshake/alert.c unconditionally reads union member msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases bvec->bv_page (struct page *), so tls_alert_recv() reads the first two bytes of struct page instead of the received alert payload and ignores fatal alerts whenever byte 0 != TLS_ALERT_LEVEL_FATAL.
2. If queue->left == 1 when a 2-byte TLS alert arrives (or a peer sends a 1-byte malformed alert record), sock_recvmsg() returns len == 1, and tls_alert_recv() reads data[0] and data[1] without checking that 2 bytes were received.
To fix this inside nvmet-tcp without cross-tree changes to net/handshake/alert.c, nvmet_tcp_tls_record_ok() can extract the alert bytes from a local ITER_SOURCE copy of the rewound msg->msg_iter via copy_from_iter_full(), pull the second byte from queue->sock when len == 1 and !(msg->msg_flags & (MSG_EOR | MSG_PEEK)), reject < 2-byte alerts with -EPROTO, and pass a local 2-byte ITER_KVEC msghdr to tls_alert_recv() so its tracepoint still works.
Note: patch 51/98 will need a trivial context adjustment around the nvmet_tcp_tls_alert_recv() call in nvmet_tcp_tls_record_ok() (and propagating MSG_EOR from rem_msg lets 51/98's nvmet_tcp_tls_drop_rest() check work unchanged).
Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
drivers/nvme/target/tcp.c +57 -6
@@ -1173,8 +1173,54 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)return false; } +static int nvmet_tcp_tls_alert_recv (struct nvmet_tcp_queue *queue, +struct msghdr *msg,int len,u8 *level,u8 *description) +{ +struct iov_iter iter = msg->msg_iter; +size_t n =min_t (size_t, len,2 ); +u8 alert[2 ]; +struct kvec kvec = { + .iov_base = alert, + .iov_len =sizeof (alert), + }; +struct msghdr alert_msg = {}; + + iter.data_source =ITER_SOURCE ; +if (!copy_from_iter_full (alert, n, &iter)) +return -EPROTO ; + +if (n <sizeof (alert)) { +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +struct kvec rem = { + .iov_base = alert + n, + .iov_len =sizeof (alert) - n, + }; +struct msghdr rem_msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT , + }; +int ret; + +if (msg->msg_flags & (MSG_EOR |MSG_PEEK )) +return -EPROTO ; + ret =kernel_recvmsg (queue->sock, &rem_msg, &rem,1 , + rem.iov_len, rem_msg.msg_flags); +if (ret != rem.iov_len || +tls_get_record_type (queue->sock->sk, + (struct cmsghdr *)cbuf) != +TLS_RECORD_TYPE_ALERT ) +return -EPROTO ; + msg->msg_flags |= rem_msg.msg_flags &MSG_EOR ; + } + +iov_iter_kvec (&alert_msg.msg_iter,ITER_DEST , &kvec,1 ,sizeof (alert)); +tls_alert_recv (queue->sock->sk, &alert_msg, level, description); +return 0 ; +} +static int nvmet_tcp_tls_record_ok (struct nvmet_tcp_queue *queue, -struct msghdr *msg,char *cbuf) +struct msghdr *msg,char *cbuf,int len) {struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;u8 ctype, level, description;
@@ -1187,7 +1233,11 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,case TLS_RECORD_TYPE_DATA :break ;case TLS_RECORD_TYPE_ALERT : -tls_alert_recv (queue->sock->sk, msg, &level, &description); +iov_iter_revert (&msg->msg_iter, len); + ret =nvmet_tcp_tls_alert_recv (queue, msg, len, &level, + &description); +if (ret) +break ;if (level ==TLS_ALERT_LEVEL_FATAL ) {pr_err ("queue %d: TLS Alert desc %u\n" , queue->idx, description);
@@ -1200,6 +1250,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,break ;default :/* discard this record type */ +iov_iter_revert (&msg->msg_iter, len);pr_err ("queue %d: TLS record %d unhandled\n" , queue->idx, ctype); ret = -EAGAIN ;
@@ -1228,7 +1279,7 @@ recv:if (unlikely (len <0 ))return len;if (queue->tls_pskid) { - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret; }
@@ -1290,7 +1341,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)return len;if (queue->tls_pskid) { ret =nvmet_tcp_tls_record_ok (cmd->queue, - &cmd->recv_msg, cmd->recv_cbuf); + &cmd->recv_msg, cmd->recv_cbuf, len);if (ret <0 )return ret; }
@@ -1331,7 +1382,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)if (unlikely (len <0 ))return len;if (queue->tls_pskid) { - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret; }
@@ -1803,7 +1854,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)return len; } - ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf); + ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 )return ret;
Jens Axboe
maintainerPATCH 49/98 nvmet-tcp: ask for the TLS record type on every receive
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_tcp_alloc_cmd() points recv_msg.msg_control at recv_cbuf once. The first receive that stores a record type there advances msg_control and leaves msg_controllen at 0 for the lifetime of the command. From then on nvmet_tcp_try_recv_data() sees the record type of that first
receive: if it was an alert, all later data is dropped as alerts and the queue stalls, if it was data, a later alert can't be reported and tls_sw_recvmsg() fails with -EIO.
Set up the control buffer before each receive, like the other receive paths do.
--- drivers/nvme/target/tcp.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/nvme/target/tcp.c +5 -0
@@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)int len, ret;while (msg_data_left (&cmd->recv_msg)) { +if (queue->tls_pskid) { +/* The last receive has used it up */ + cmd->recv_msg.msg_control = cmd->recv_cbuf; + cmd->recv_msg.msg_controllen =sizeof (cmd->recv_cbuf); + } len =sock_recvmsg (cmd->queue->sock, &cmd->recv_msg, cmd->recv_msg.msg_flags);if (len <=0 )
Signature · 1 line
2.56.0
Re: [PATCH 49/98] nvmet-tcp: ask for the TLS record type on every receive
Quoted 29 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_tcp_alloc_cmd()points recv_msg.msg_control atrecv_cbufonce. The first receive that stores a record type there advancesmsg_controland leavesmsg_controllenat 0 for the lifetime of the command. From then onnvmet_tcp_try_recv_data()sees the record type of that first
receive: if it was an alert, all later data is dropped as alerts and the queue stalls, if it was data, a later alert can't be reported andtls_sw_recvmsg()fails with-EIO.Set up the control buffer before each receive, like the other receive paths do.
Fixes a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/tcp.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/nvme/target/tcp.c +0
-0@@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)int len, ret;
drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) · line 1296
Quoted 10 lines
drivers/nvme/target/tcp.c +5
-0while (msg_data_left (&cmd->recv_msg)) { +if (queue->tls_pskid) { +/* The last receive has used it up */ + cmd->recv_msg.msg_control = cmd->recv_cbuf; + cmd->recv_msg.msg_controllen =sizeof (cmd->recv_cbuf); + } len =sock_recvmsg (cmd->queue->sock, &cmd->recv_msg, cmd->recv_msg.msg_flags);if (len <=0 )
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 50/98 nvme-tcp: wait for an in-flight TLS handshake callback
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
If the wait for the TLS handshake times out or is interrupted, nvme_tcp_start_tls() cancels the request and returns. tls_handshake_cancel() returns false if the handshake already completed, in which case nvme_tcp_tls_done() may still be running while nvme_tcp_alloc_queue() tears the queue down. On the first connect that frees the controller, including the queue array the callback writes to.
If the cancel lost, wait for the callback to signal completion before returning, like the SUNRPC callers do.
--- drivers/nvme/host/tcp.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +6 -1
@@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,dev_err (nctrl->device,"queue %d: TLS handshake failed, error %d\n" , qid, ret); -tls_handshake_cancel (queue->sock->sk); +/* +* If the cancellation lost to handshake_complete(), +* nvme_tcp_tls_done() is in flight and still uses the queue. +*/ +if (!tls_handshake_cancel (queue->sock->sk)) +wait_for_completion (&queue->tls_complete); }else {if (queue->tls_err) { dev_err(nctrl->device,
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 50/98] nvme-tcp: wait for an in-flight TLS handshake callback
Quoted 30 lines · Jens Axboe
From: Josef Bacik <[email protected]>
If the wait for the TLS handshake times out or is interrupted,
nvme_tcp_start_tls()cancels the request and returns.tls_handshake_cancel()returns false if the handshake already completed, in which casenvme_tcp_tls_done()may still be running whilenvme_tcp_alloc_queue()tears the queue down. On the first connect that frees the controller, including the queue array the callback writes to.If the cancel lost, wait for the callback to signal completion before returning, like the SUNRPC callers do.
Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")Cc Keith Busch <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/host/tcp.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,dev_err (nctrl->device,"queue %d: TLS handshake failed, error %d\n" , qid, ret);
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1838
Quoted 10 lines
drivers/nvme/host/tcp.c +6
-1-tls_handshake_cancel (queue->sock->sk); +/* +* If the cancellation lost to handshake_complete(), +* nvme_tcp_tls_done() is in flight and still uses the queue. +*/ +if (!tls_handshake_cancel (queue->sock->sk)) +wait_for_completion (&queue->tls_complete); }else {if (queue->tls_err) { dev_err(nctrl->device,
This has already been reported to the linux-nvme list
(https://lore.kernel.org/linux-nvme/[email protected]/)
Please drop.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 51/98 nvmet-tcp: look again after a short read with TLS
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The receive helpers return -EAGAIN both for a short read and for a dropped non-data TLS record, and nvmet_tcp_try_recv() takes that as "socket empty" and waits for the next data_ready. On plain TCP that's right. tls_sw_recvmsg() stops where the record type changes though, so what follows an alert or a handshake record may already be queued and nobody looks at it until the host sends something else.
Return 1 in those cases so nvmet_tcp_try_recv() counts it against the budget and tries again until the socket itself says -EAGAIN. While at it, drain the rest of a dropped record using MSG_EOR instead of pulling it in PDU-sized pieces, and ratelimit the messages since the host controls how many there are.
--- drivers/nvme/target/tcp.c | 54 +++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 11 deletions(-)
drivers/nvme/target/tcp.c +43 -11
@@ -1173,6 +1173,24 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)return false; } +/* +* The buffer took as much of the record as the PDU still wants, which may be +* one byte. The rest comes with the type of the record, never with that +* which follows: throw it away at once. +*/ +static void nvmet_tcp_tls_drop_rest (struct nvmet_tcp_queue *queue) +{ +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT , + }; + +iov_iter_discard (&msg.msg_iter,ITER_DEST ,TLS_MAX_PAYLOAD_SIZE ); +sock_recvmsg (queue->sock, &msg, msg.msg_flags); +} +static int nvmet_tcp_tls_record_ok (struct nvmet_tcp_queue *queue,struct msghdr *msg,char *cbuf,int len) {
@@ -1190,26 +1208,38 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,iov_iter_revert (&msg->msg_iter, len);tls_alert_recv (queue->sock->sk, msg, &level, &description);if (level ==TLS_ALERT_LEVEL_FATAL ) { -pr_err ("queue %d: TLS Alert desc %u\n" , - queue->idx, description); +pr_err_ratelimited ("queue %d: TLS Alert desc %u\n" , + queue->idx, description); ret = -ENOTCONN ; }else { -pr_warn ("queue %d: TLS Alert desc %u\n" , - queue->idx, description); +pr_warn_ratelimited ("queue %d: TLS Alert desc %u\n" , + queue->idx, description); ret = -EAGAIN ; }break ;default :/* discard this record type */ iov_iter_revert (&msg->msg_iter, len); -pr_err ("queue %d: TLS record %d unhandled\n" , - queue->idx, ctype); +pr_err_ratelimited ("queue %d: TLS record %d unhandled\n" , + queue->idx, ctype); ret = -EAGAIN ;break ; } +if (ret == -EAGAIN && !(msg->msg_flags &MSG_EOR )) +nvmet_tcp_tls_drop_rest (queue);return ret; } +/* +* A receive from a TLS socket ends with the record, or where the type of the +* records changes. There may be more behind a record that has been dropped, +* and nobody will tell: have nvmet_tcp_try_recv() look again. +*/ +static int nvmet_tcp_tls_again (int ret) +{ +return ret == -EAGAIN ?1 : ret; +} +static int nvmet_tcp_try_recv_pdu (struct nvmet_tcp_queue *queue) {struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
@@ -1224,6 +1254,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)if (queue->tls_pskid) { msg.msg_control = cbuf; msg.msg_controllen =sizeof (cbuf); + msg.msg_flags &= ~MSG_EOR ; } len =kernel_recvmsg (queue->sock, &msg, &iov,1 , iov.iov_len, msg.msg_flags);
@@ -1232,13 +1263,13 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)if (queue->tls_pskid) { ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 ) -return ret; +return nvmet_tcp_tls_again (ret); } queue->offset += len; queue->left -= len;if (queue->left) -return -EAGAIN ; +return queue->tls_pskid && len ?1 : -EAGAIN ;if (queue->offset ==sizeof (struct nvme_tcp_hdr )) {u8 hdgst =nvmet_tcp_hdgst_len (queue);
@@ -1290,6 +1321,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)/* The last receive has used it up */ cmd->recv_msg.msg_control = cmd->recv_cbuf; cmd->recv_msg.msg_controllen =sizeof (cmd->recv_cbuf); + cmd->recv_msg.msg_flags &= ~MSG_EOR ; } len =sock_recvmsg (cmd->queue->sock, &cmd->recv_msg, cmd->recv_msg.msg_flags);
@@ -1299,7 +1331,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue) ret =nvmet_tcp_tls_record_ok (cmd->queue, &cmd->recv_msg, cmd->recv_cbuf, len);if (ret <0 ) -return ret; +return nvmet_tcp_tls_again (ret); } cmd->pdu_recv += len;
@@ -1340,13 +1372,13 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)if (queue->tls_pskid) { ret =nvmet_tcp_tls_record_ok (queue, &msg, cbuf, len);if (ret <0 ) -return ret; +return nvmet_tcp_tls_again (ret); } queue->offset += len; queue->left -= len;if (queue->left) -return -EAGAIN ; +return queue->tls_pskid && len ?1 : -EAGAIN ;if (queue->data_digest && cmd->exp_ddgst != cmd->recv_ddgst) { pr_err("queue %d: cmd %d pdu (%d) data digest error: recv %#x expected %#x\n",
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 52/98 nvme-tcp: keep the socket while the queue is polled
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Quiescing keeps ->queue_rq() out, not ->poll(): bio_poll() enters even a frozen queue. So nvme_tcp_poll() can run while error recovery or a reset stops and frees the queues, and keeps running afterwards for the requeued requests whose bios still carry a cookie. The NVME_TCP_Q_LIVE check doesn't help, queue->sock->sk is read before it and nvme_tcp_free_queue() has set queue->sock to NULL:
BUG: KASAN: null-ptr-deref in nvme_tcp_poll+0x42/0x1c0
The poller dies holding a queue reference and the controller never reconnects.
nvme_tcp_stop_queue() clears the bit under queue_lock. Trylock it in nvme_tcp_poll() too and check the bit and the socket under it; whoever holds it is either polling or stopping this queue. That means the lock can't live with the queue, so initialize it with the controller.
--- drivers/nvme/host/tcp.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-)
drivers/nvme/host/tcp.c +22 -7
@@ -1530,7 +1530,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)kfree (queue->pdu);mutex_destroy (&queue->send_mutex); -mutex_destroy (&queue->queue_lock);mutex_destroy (&queue->pf_cache_lock); }
@@ -1855,7 +1854,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,int ret, rcv_pdu_size;struct file *sock_file; -mutex_init (&queue->queue_lock); queue->ctrl = ctrl;init_llist_head (&queue->req_list);INIT_LIST_HEAD (&queue->send_list);
@@ -1993,7 +1991,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, queue->sock =NULL ; err_destroy_mutex:mutex_destroy (&queue->send_mutex); -mutex_destroy (&queue->queue_lock);mutex_destroy (&queue->pf_cache_lock);return ret; }
@@ -2922,18 +2919,33 @@ static void nvme_tcp_map_queues(struct blk_mq_tag_set *set)static int nvme_tcp_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) {struct nvme_tcp_queue *queue = hctx->driver_data; -struct sock *sk = queue->sock->sk; -int ret; +struct sock *sk; +int ret =0 ;if (!test_bit (NVME_TCP_Q_LIVE , &queue->flags))return 0 ; +/* +* Nothing in the block layer keeps a poller out while the queue is +* torn down. The lock keeps the socket until we are through; if +* somebody else has it, he polls, or the queue is being stopped. +*/ +if (!mutex_trylock (&queue->queue_lock)) +return 0 ; +if (!test_bit (NVME_TCP_Q_LIVE , &queue->flags)) + goto unlock; + + sk = queue->sock->sk;set_bit (NVME_TCP_Q_POLLING , &queue->flags);if (sk_can_busy_loop (sk) &&skb_queue_empty_lockless (&sk->sk_receive_queue))sk_busy_loop (sk, true); ret =nvme_tcp_try_recv (queue);clear_bit (NVME_TCP_Q_POLLING , &queue->flags); -return ret <0 ? ret : queue->nr_cqe; +if (ret >=0 ) + ret = queue->nr_cqe; +unlock: +mutex_unlock (&queue->queue_lock); +return ret; }static int nvme_tcp_get_address (struct nvme_ctrl *ctrl,char *buf,int size)
@@ -3020,7 +3032,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) {struct nvme_tcp_ctrl *ctrl; -int ret; +int ret, i; ctrl =kzalloc_obj (*ctrl);if (!ctrl)
@@ -3089,6 +3101,9 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, ret = -ENOMEM ; goto out_free_ctrl; } +/* A poller may try the lock of a queue that has been freed */ +for (i =0 ; i < ctrl->ctrl.queue_count; i++) +mutex_init (&ctrl->queues[i].queue_lock); ctrl->net =get_net (current->nsproxy->net_ns);
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 53/98 nvme-tcp: don't get stuck on a TLS control record
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The host receives with read_sock(), and tls_sw_read_sock() won't hand out a non-data record: it puts it back and returns -EINVAL. nvme_tcp_io_work() bails on the error and finds the same record again next time, so nothing behind it is ever read. One warning alert or NewSessionTicket after the ICResp kills the queue: commands time out and the controller resets. The Linux target sends none, but others may.
Fetch such a record with recvmsg() instead. A fatal alert starts error recovery, anything else is dropped whole, with a log line, and the queue carries on. io_work and a poller may both see the record, so a bit makes sure only one of them deals with it, and that one retries read_sock() afterwards.
--- drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +86 -1
@@ -95,6 +95,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 ,NVME_TCP_Q_IO_CPU_SET =3 , + NVME_TCP_Q_TLS_CTRL = 4, }; enum nvme_tcp_recv_state {
@@ -1418,7 +1419,7 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)return ret; } -static int nvme_tcp_try_recv (struct nvme_tcp_queue *queue) +static int nvme_tcp_read_sock (struct nvme_tcp_queue *queue) {struct socket *sock = queue->sock;struct sock *sk = sock->sk;
@@ -1431,6 +1432,90 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe =0 ; consumed = sock->ops->read_sock (sk, &rd_desc, nvme_tcp_recv_skb);release_sock (sk); +return consumed; +} + +/* +* read_sock() leaves a TLS record that is no data where it is, in front of +* all that follows. Take it out of the way. +*/ +static int nvme_tcp_recv_tls_ctrl (struct nvme_tcp_queue *queue) +{ +struct nvme_ctrl *ctrl = &queue->ctrl->ctrl; +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT , + }; +u8 ctype, level, description, data[2 ] = {}; +struct kvec iov = { .iov_base = data, .iov_len =sizeof (data) }; +int len; + +/* +* io_work and a poller may both have met the record, and recvmsg() +* would take data as well. Only one of them goes on, and he looks +* again: nobody else removes such a record from now on. +*/ +if (test_and_set_bit_lock (NVME_TCP_Q_TLS_CTRL , &queue->flags)) +return 0 ; + len =nvme_tcp_read_sock (queue); +if (len != -EINVAL || !queue->rd_enabled) + goto out; + + len =kernel_recvmsg (queue->sock, &msg, &iov,1 , iov.iov_len, + msg.msg_flags); +if (len <0 ) + goto out; + + ctype =tls_get_record_type (queue->sock->sk, (struct cmsghdr *)cbuf); +switch (ctype) { +case 0 : +case TLS_RECORD_TYPE_DATA : + goto fail; +case TLS_RECORD_TYPE_ALERT : +iov_iter_revert (&msg.msg_iter, len); +tls_alert_recv (queue->sock->sk, &msg, &level, &description); +dev_warn_ratelimited (ctrl->device, +"queue %d: TLS Alert level %u desc %u\n" , +nvme_tcp_queue_id (queue), level, + description); +if (level ==TLS_ALERT_LEVEL_FATAL ) + goto fail; +break ; +default : +dev_warn_ratelimited (ctrl->device, +"queue %d: TLS record %u dropped\n" , +nvme_tcp_queue_id (queue), ctype); +break ; + } + +if (!(msg.msg_flags &MSG_EOR )) { + msg.msg_control = cbuf; + msg.msg_controllen =sizeof (cbuf); +iov_iter_discard (&msg.msg_iter,ITER_DEST , +TLS_MAX_PAYLOAD_SIZE ); +sock_recvmsg (queue->sock, &msg,MSG_DONTWAIT ); + } + len =1 ; +out: +clear_bit_unlock (NVME_TCP_Q_TLS_CTRL , &queue->flags); +return len; + +fail: + queue->rd_enabled = false; +nvme_tcp_error_recovery (ctrl); + len = -ENOTCONN ; + goto out; +} + +static int nvme_tcp_try_recv (struct nvme_tcp_queue *queue) +{ +int consumed =nvme_tcp_read_sock (queue); + +if (consumed == -EINVAL &&nvme_tcp_queue_tls (queue) && + queue->rd_enabled) + consumed =nvme_tcp_recv_tls_ctrl (queue);return consumed == -EAGAIN ?0 : consumed; }
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 53/98] nvme-tcp: don't get stuck on a TLS control record
Quoted 21 lines · Jens Axboe
From: Josef Bacik <[email protected]>
The host receives with
read_sock(), andtls_sw_read_sock()won't hand out a non-data record: it puts it back and returns-EINVAL.nvme_tcp_io_work()bails on the error and finds the same record again next time, so nothing behind it is ever read. One warning alert or NewSessionTicket after the ICResp kills the queue: commands time out and the controller resets. The Linux target sends none, but others may.Fetch such a record with
recvmsg()instead. A fatal alert starts error recovery, anything else is dropped whole, with a log line, and the queue carries on.io_workand a poller may both see the record, so a bit makes sure only one of them deals with it, and that one retriesread_sock()afterwards.Fixes be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")Cc Keith Busch <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 4 lines
--- drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)
TLS record handling is currently being reworked, so please drop.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 54/98 nvme-tcp: start error recovery after a TLS KeyUpdate
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Once the controller sends a KeyUpdate, kTLS fails every receive with -EKEYEXPIRED until a new key is installed, and nobody on the host side does that. nvme_tcp_io_work() returns on the error and the queue is dead until a command or the keep-alive times out.
The connection is unusable at that point, say so and start error recovery right away.
--- drivers/nvme/host/tcp.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/nvme/host/tcp.c +8 -0
@@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)if (consumed == -EINVAL &&nvme_tcp_queue_tls (queue) && queue->rd_enabled) consumed =nvme_tcp_recv_tls_ctrl (queue); +if (consumed == -EKEYEXPIRED && queue->rd_enabled) { +/* Nobody is going to install the new key */ +dev_err (queue->ctrl->ctrl.device, +"queue %d: TLS KeyUpdate is not supported\n" , +nvme_tcp_queue_id (queue)); + queue->rd_enabled = false; +nvme_tcp_error_recovery (&queue->ctrl->ctrl); + }return consumed == -EAGAIN ?0 : consumed; }
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 54/98] nvme-tcp: start error recovery after a TLS KeyUpdate
Quoted 29 lines · Jens Axboe
From: Josef Bacik <[email protected]>
Once the controller sends a KeyUpdate, kTLS fails every receive with
-EKEYEXPIREDuntil a new key is installed, and nobody on the host side does that.nvme_tcp_io_work()returns on the error and the queue is dead until a command or the keep-alive times out.The connection is unusable at that point, say so and start error recovery right away.
Fixes 0471b1093e3a ("tls: block decryption when a rekey is pending")Cc Keith Busch <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/host/tcp.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/nvme/host/tcp.c +1
-0@@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)if (consumed == -EINVAL &&nvme_tcp_queue_tls (queue) && queue->rd_enabled) consumed =nvme_tcp_recv_tls_ctrl (queue); +if (consumed == -EKEYEXPIRED && queue->rd_enabled) {
drivers/nvme/host/tcp.c static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) · line 1529
Quoted 10 lines
drivers/nvme/host/tcp.c +7
-0+/* Nobody is going to install the new key */ +dev_err (queue->ctrl->ctrl.device, +"queue %d: TLS KeyUpdate is not supported\n" , +nvme_tcp_queue_id (queue)); + queue->rd_enabled = false; +nvme_tcp_error_recovery (&queue->ctrl->ctrl); + }return consumed == -EAGAIN ?0 : consumed; }
Already been reported (https://lore.kernel.org/linux-nvme/[email protected]/).
Please drop.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 55/98 nvme-tcp: don't act on PDUs for a command being sent
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A request belongs to the send path from the moment it's put on req_list until its last byte is handed to the socket, but the receive path acts on whatever the controller sends for it. A response completes the request while the send side is still reading from its bio:
BUG: KASAN: slab-use-after-free in nvme_tcp_try_send_data+0x175/0x6b0 nvme_tcp_try_send+0x4af/0x600 nvme_tcp_io_work+0x249/0x380
and an R2T resets and requeues it, since the queued check only looks at the two lists and a request being sent is on neither, so it ends up completed twice. An honest controller doesn't do that, but its answer can legitimately overtake a sender that has handed over the last byte and not yet cleared queue->request.
Make such a PDU wait: don't consume the bytes that complete its header, so it's seen again by the next receive. For that nvme_tcp_req_busy() has to find the request in one of the three places at all times, so nvme_tcp_fetch_request() sets queue->request before taking it off the lists. io_work reruns when the sender was io_work or the socket was full, waits for send_mutex when the submitter was sending, and a poller just polls again.
--- drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++------- 1 file changed, 73 insertions(+), 14 deletions(-)
drivers/nvme/host/tcp.c +73 -14
@@ -123,6 +123,7 @@ struct nvme_tcp_queue {size_t data_remaining;size_t ddgst_remaining;unsigned int nr_cqe; +bool rcv_parked;/* send state */ struct nvme_tcp_request *request;
@@ -465,11 +466,29 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)return NULL ; } +WRITE_ONCE (queue->request, req); +/* nvme_tcp_req_busy() must find the request in one place at any time */ +smp_wmb ();list_del_init (&req->entry);init_llist_node (&req->lentry);return req; } +/* +* Does the send path still hold @req? As long as it does the request must +* neither be completed nor be queued once more, whatever the controller says. +*/ +static bool nvme_tcp_req_busy (struct nvme_tcp_queue *queue, +struct nvme_tcp_request *req) +{ +if (llist_on_list (&req->lentry) || !list_empty (&req->entry)) +return true; +/* pairs with the barrier in nvme_tcp_fetch_request() */ +smp_rmb (); +/* pairs with the release in nvme_tcp_done_send_req() */ +return smp_load_acquire (&queue->request) == req; +} +#define NVME_TCP_CRC_SEED (~0) static inline void nvme_tcp_ddgst_update(u32 *crcp,
@@ -672,6 +691,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, } req =blk_mq_rq_to_pdu (rq); +if (unlikely (nvme_tcp_req_busy (queue, req))) +return -EBUSY ; +if (req->status ==cpu_to_le16 (NVME_SC_SUCCESS )) req->status = cqe->status;
@@ -707,6 +729,9 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue, } req =blk_mq_rq_to_pdu (rq); +if (unlikely (nvme_tcp_req_busy (queue, req))) +return -EBUSY ; +if (!blk_rq_payload_bytes (rq) || !req->curr_bio || !req->data_len) { dev_err(queue->ctrl->ctrl.device, "queue %d tag %#x unexpected data\n",
@@ -800,6 +825,8 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,return -ENOENT ; } req =blk_mq_rq_to_pdu (rq); +if (unlikely (nvme_tcp_req_busy (queue, req))) +return -EBUSY ;if (unlikely (rq_data_dir (rq) !=WRITE )) { dev_err(queue->ctrl->ctrl.device,
@@ -829,14 +856,6 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,return -EPROTO ; } -if (llist_on_list (&req->lentry) || - !list_empty (&req->entry)) { -dev_err (queue->ctrl->ctrl.device, -"req %d unexpected r2t while processing request\n" , - rq->tag); -return -EPROTO ; - } - req->pdu_len =0 ; req->h2cdata_left = r2t_length; req->h2cdata_offset = r2t_offset;
@@ -939,17 +958,32 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,switch (hdr->type) {case nvme_tcp_c2h_data: -return nvme_tcp_handle_c2h_data (queue, (void *)queue->pdu); + ret =nvme_tcp_handle_c2h_data (queue, (void *)queue->pdu); +break ;case nvme_tcp_rsp: -nvme_tcp_init_recv_ctx (queue); -return nvme_tcp_handle_comp (queue, (void *)queue->pdu); + ret =nvme_tcp_handle_comp (queue, (void *)queue->pdu); +break ;case nvme_tcp_r2t: -nvme_tcp_init_recv_ctx (queue); -return nvme_tcp_handle_r2t (queue, (void *)queue->pdu); + ret =nvme_tcp_handle_r2t (queue, (void *)queue->pdu); +break ;default : goto unsupported_pdu; } +if (unlikely (ret == -EBUSY )) { +/* +* The command is still being sent. Leave the end of the header +* in the socket and look at it again when that is done. +*/ + queue->pdu_remaining += rcv_len; + queue->pdu_offset -= rcv_len; + *offset -= rcv_len; + *len += rcv_len; + }else if (hdr->type != nvme_tcp_c2h_data) { +nvme_tcp_init_recv_ctx (queue); + } +return ret; + unsupported_pdu:dev_err (queue->ctrl->ctrl.device,"unsupported pdu type (%d)\n" , hdr->type);
@@ -1110,6 +1144,10 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,default : result = -EFAULT ; } +if (unlikely (result == -EBUSY )) { + queue->rcv_parked = true; +return consumed - len; + }if (result) {dev_err (queue->ctrl->ctrl.device,"receive failed: %d\n" , result);
@@ -1182,7 +1220,8 @@ static void nvme_tcp_state_change(struct sock *sk)static inline void nvme_tcp_done_send_req (struct nvme_tcp_queue *queue) { - queue->request =NULL ; +/* pairs with nvme_tcp_req_busy() */ +smp_store_release (&queue->request,NULL ); }static void nvme_tcp_fail_request (struct nvme_tcp_request *req)
@@ -1430,6 +1469,7 @@ static int nvme_tcp_read_sock(struct nvme_tcp_queue *queue) rd_desc.count =1 ;lock_sock (sk); queue->nr_cqe =0 ; + queue->rcv_parked = false; consumed = sock->ops->read_sock (sk, &rd_desc, nvme_tcp_recv_skb);release_sock (sk);return consumed;
@@ -1527,6 +1567,22 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)return consumed == -EAGAIN ?0 : consumed; } +/* +* A PDU has been left in the socket because its command is still being sent. +* A sender in another context does not wake us when it is done: wait for it. +* If there is something left to send then, either the socket is full and tells +* us when it has room, or io_work has been queued. +*/ +static bool nvme_tcp_recv_again (struct nvme_tcp_queue *queue) +{ +bool again; + +mutex_lock (&queue->send_mutex); + again = !queue->request && !nvme_tcp_queue_has_pending (queue); +mutex_unlock (&queue->send_mutex); +return again; +} +static void nvme_tcp_io_work (struct work_struct *w) { struct nvme_tcp_queue *queue =
@@ -1552,6 +1608,9 @@ static void nvme_tcp_io_work(struct work_struct *w)else if (unlikely (result <0 ))return ; +if (unlikely (queue->rcv_parked) &&nvme_tcp_recv_again (queue)) + pending = true; +/* did we get some space after spending time in recv? */ if (nvme_tcp_queue_has_pending (queue) &&sk_stream_is_writeable (queue->sock->sk))
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 56/98 nvme-tcp: don't queue the AER while it is being sent
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Every AER completion runs nvme_async_event_work(), which has nvme_tcp_submit_async_event() set up and queue ctrl->async_req again. A controller that completes the AER twice can do that while the request is still on req_list or send_list, or being sent. Added to req_list twice the list becomes a ring and nvme_tcp_process_req_list() never ends:
WARNING: lib/list_debug.c:35 at __list_add_valid_or_report+0xd8/0xf0 nvme_tcp_try_send+0x51b/0x600
Leave the AER alone while the send path holds it, there's one on its way already. Initialize the list nodes at allocation so the check is valid from the start; nvme_tcp_fetch_request() leaves them that way.
--- drivers/nvme/host/tcp.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +11 -4
@@ -766,11 +766,14 @@ static int nvme_tcp_handle_comp(struct nvme_tcp_queue *queue,* for them but rather special case them here. */ if (unlikely (nvme_is_aen_req (nvme_tcp_queue_id (queue), - cqe->command_id))) + cqe->command_id))) { +if (nvme_tcp_req_busy (queue, &queue->ctrl->async_req)) +return -EBUSY ;nvme_complete_async_event (&queue->ctrl->ctrl, cqe->status, &cqe->result); -else + }else { ret =nvme_tcp_process_nvme_cqe (queue, cqe); + }return ret; }
@@ -1646,6 +1649,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)return -ENOMEM ; async->queue = &ctrl->queues[0 ]; +init_llist_node (&async->lentry); +INIT_LIST_HEAD (&async->entry);return 0 ; }
@@ -2884,6 +2889,10 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)struct nvme_command *cmd = &pdu->cmd;u8 hdgst =nvme_tcp_hdgst_len (queue); +/* A controller which completes it twice makes us come here twice */ +if (nvme_tcp_req_busy (queue, &ctrl->async_req)) +return ; +memset (pdu,0 ,sizeof (*pdu)); pdu->hdr.type = nvme_tcp_cmd;if (queue->hdr_digest)
@@ -2900,8 +2909,6 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.offset =0 ; ctrl->async_req.curr_bio =NULL ; ctrl->async_req.data_len =0 ; -init_llist_node (&ctrl->async_req.lentry); -INIT_LIST_HEAD (&ctrl->async_req.entry);nvme_tcp_queue_request (&ctrl->async_req, true); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 57/98 nvme-tcp: don't send SIGPIPE to whoever submits I/O
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvme_tcp_queue_request() tries to send inline, so the first PDU of a command goes out in the context of the submitting task, and none of the sends pass MSG_NOSIGNAL. Once a connection is down and error recovery hasn't stopped the queue yet, tcp_sendmsg() returns -EPIPE and sk_stream_error() sends SIGPIPE to current. A lost connection that the driver would recover from transparently instead kills whatever is doing I/O on the namespace: with 16 dd processes and a connection cut every other second, a few die within a minute.
Pass MSG_NOSIGNAL, like nbd, drbd, ceph, cifs and sunrpc do.
--- drivers/nvme/host/tcp.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/nvme/host/tcp.c +10 -5
@@ -1249,7 +1249,8 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)while (true) {struct bio_vec bvec;struct msghdr msg = { - .msg_flags =MSG_DONTWAIT |MSG_SPLICE_PAGES , + .msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL | +MSG_SPLICE_PAGES , };struct page *page =nvme_tcp_req_cur_page (req);size_t offset =nvme_tcp_req_cur_offset (req);
@@ -1308,7 +1309,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)struct nvme_tcp_queue *queue = req->queue;struct nvme_tcp_cmd_pdu *pdu =nvme_tcp_req_cmd_pdu (req);struct bio_vec bvec; -struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_SPLICE_PAGES , }; +struct msghdr msg = { + .msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL |MSG_SPLICE_PAGES , + };bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset;
@@ -1349,7 +1352,9 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)struct nvme_tcp_queue *queue = req->queue;struct nvme_tcp_data_pdu *pdu =nvme_tcp_req_data_pdu (req);struct bio_vec bvec; -struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_MORE , }; +struct msghdr msg = { + .msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL |MSG_MORE , + };u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) - req->offset + hdgst;int ret;
@@ -1384,7 +1389,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)size_t offset = req->offset;u32 h2cdata_left = req->h2cdata_left;int ret; -struct msghdr msg = { .msg_flags =MSG_DONTWAIT }; +struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL };struct kvec iov = { .iov_base = (u8 *)&req->ddgst + req->offset, .iov_len =NVME_TCP_DIGEST_LENGTH - req->offset
@@ -1696,7 +1701,7 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)struct nvme_tcp_icresp_pdu *icresp;char cbuf[CMSG_LEN (sizeof (char ))] = {};u8 ctype; -struct msghdr msg = {}; +struct msghdr msg = { .msg_flags =MSG_NOSIGNAL };struct kvec iov;bool ctrl_hdgst, ctrl_ddgst;u32 maxh2cdata;
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 58/98 nvme-fabrics: reconnect if the association is gone
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A reconnect connects the admin queue, gets a controller ID, and connects the I/O queues with it. If the controller goes away in between, the Connect of an I/O queue fails with Connect Invalid Parameters pointing at the cntlid, with DNR set:
nvme nvme0: Connect Invalid Data Parameter, cntlid: 1 nvme nvme0: Failed reconnect attempt 1/600 nvme nvme0: Removing controller (16770)...
nvmf_should_reconnect() honours DNR and the host deletes the controller with 599 attempts to go. DNR is right in that resending that command is pointless, but the next attempt would create a new association with a new cntlid. This is not a verdict on the host like a failed admin Connect is.
Treat it as a lost association and return -ENOTCONN, so the retries are bounded by ctrl_loss_tmo like after any other loss.
--- drivers/nvme/host/fabrics.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/nvme/host/fabrics.c +13 -2
@@ -542,8 +542,19 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)NVME_SUBMIT_RESERVED | NVME_SUBMIT_NOWAIT);if (ret) { -nvmf_log_connect_error (ctrl, ret,le32_to_cpu (res.u32), - &cmd, data); + result =le32_to_cpu (res.u32); +nvmf_log_connect_error (ctrl, ret, result, &cmd, data); +/* +* The controller ID we got from the admin queue's Connect is +* not known any more: the association has gone since. DNR is +* right about this command and says nothing about the next +* association. Don't let nvmf_should_reconnect() see it. +*/ +if (ret >0 && + (ret &NVME_SCT_SC_MASK ) ==NVME_SC_CONNECT_INVALID_PARAM && + result >>16 && (result &0xffff ) == + offsetof(struct nvmf_connect_data , cntlid)) + ret = -ENOTCONN ; goto out_free_data; } result =le32_to_cpu (res.u32);
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 59/98 nvmet-auth: serialize the commands of a controller
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The host and controller keys, the DH context and the hash of DH-HMAC-CHAP live in the controller, and the authentication commands of all its queues use them without a lock. A host can send AUTH_Negotiate on the admin queue at any time, and nvmet_setup_auth() then frees and reloads the keys while another queue computes a response from them:
BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250 nvmet_auth_host_hash+0x10e/0x530 nvmet_execute_auth_send+0xa4c/0x1a00 Freed by task 194: nvmet_setup_auth+0x166/0x3a0 nvmet_execute_auth_send+0x3e7/0x1a00
Neither queue needs to have authenticated for that.
Add a controller-wide mutex and hold it for the duration of an authentication command, dropping it before the request is completed. nvmet_alloc_ctrl() takes it around nvmet_setup_auth() too, as the controller is already on the subsystem list by then and an I/O queue can connect.
[axboe: upstream already has the per-queue auth_lock and expiry handling
from 53f6efe00cea, so only the controller lock is added here. It nests inside sq->auth_lock.]
--- drivers/nvme/target/core.c | 4 ++++ drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++ drivers/nvme/target/nvmet.h | 1 + 3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +4 -0
@@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)if (!ctrl) goto out_put_subsystem;mutex_init (&ctrl->lock); +mutex_init (&ctrl->auth_lock); ctrl->port = args->port; ctrl->ops = args->ops;
@@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)nvmet_start_keep_alive_timer (ctrl); +/* An I/O queue may connect as soon as the controller is on the list. */ +mutex_lock (&ctrl->auth_lock); dhchap_status =nvmet_setup_auth (ctrl, args->sq, false); +mutex_unlock (&ctrl->auth_lock);if (dhchap_status) {pr_err ("Failed to setup authentication, dhchap status %u\n" , dhchap_status);
drivers/nvme/target/fabrics-cmd-auth.c +6 -0
@@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)u8 dhchap_status;mutex_lock (&req->sq->auth_lock); +mutex_lock (&ctrl->auth_lock);if (req->cmd->auth_send.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ;
@@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) }if (unlikely (tl != req->transfer_len)) {pr_debug ("%s: transfer length mismatch (%u)\n" , __func__, tl); +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_check_transfer_len (req, tl);return ;
@@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)nvmet_ctrl_fatal_error (ctrl); complete: +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_req_complete (req, status); }
@@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)u16 status =0 ;mutex_lock (&req->sq->auth_lock); +mutex_lock (&ctrl->auth_lock);if (req->cmd->auth_receive.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ;
@@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) }if (unlikely (al != req->transfer_len)) {pr_debug ("%s: transfer length mismatch (%u)\n" , __func__, al); +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_check_transfer_len (req, al);return ;
@@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)nvmet_auth_sq_free (req->sq);nvmet_ctrl_fatal_error (ctrl); } +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_req_complete (req, status); }
drivers/nvme/target/nvmet.h +1 -0
@@ -307,6 +307,7 @@ struct nvmet_ctrl {struct nvme_error_slot slots[NVMET_ERROR_LOG_SLOTS ];bool pi_support;bool concat; +struct mutex auth_lock;/* one AUTH command at a time */ #ifdef CONFIG_NVME_TARGET_AUTH struct nvme_dhchap_key *host_key;struct nvme_dhchap_key *ctrl_key;
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 59/98] nvmet-auth: serialize the commands of a controller
Quoted 120 lines · Jens Axboe
From: Josef Bacik <[email protected]>
The host and controller keys, the DH context and the hash of DH-HMAC-CHAP live in the controller, and the authentication commands of all its queues use them without a lock. A host can send
AUTH_Negotiateon the admin queue at any time, andnvmet_setup_auth()then frees and reloads the keys while another queue computes a response from them:BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250 nvmet_auth_host_hash+0x10e/0x530 nvmet_execute_auth_send+0xa4c/0x1a00 Freed by task 194: nvmet_setup_auth+0x166/0x3a0 nvmet_execute_auth_send+0x3e7/0x1a00Neither queue needs to have authenticated for that.
Add a controller-wide mutex and hold it for the duration of an authentication command, dropping it before the request is completed.
nvmet_alloc_ctrl()takes it aroundnvmet_setup_auth()too, as the controller is already on the subsystem list by then and an I/O queue can connect.[axboe: upstream already has the per-queue
auth_lockand expiry handling
from53f6efe00cea, so only the controller lock is added here. It nests insidesq->auth_lock.]Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Cc Hannes Reinecke <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/core.c | 4 ++++ drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++ drivers/nvme/target/nvmet.h | 1 + 3 files changed, 11 insertions(+)drivers/nvme/target/core.c +4
-0@@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)if (!ctrl) goto out_put_subsystem;mutex_init (&ctrl->lock); +mutex_init (&ctrl->auth_lock); ctrl->port = args->port; ctrl->ops = args->ops;@@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)nvmet_start_keep_alive_timer (ctrl); +/* An I/O queue may connect as soon as the controller is on the list. */ +mutex_lock (&ctrl->auth_lock); dhchap_status =nvmet_setup_auth (ctrl, args->sq, false); +mutex_unlock (&ctrl->auth_lock);if (dhchap_status) {pr_err ("Failed to setup authentication, dhchap status %u\n" , dhchap_status);drivers/nvme/target/fabrics-cmd-auth.c +6
-0@@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)u8 dhchap_status;mutex_lock (&req->sq->auth_lock); +mutex_lock (&ctrl->auth_lock);if (req->cmd->auth_send.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ;@@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) }if (unlikely (tl != req->transfer_len)) {pr_debug ("%s: transfer length mismatch (%u)\n" , __func__, tl); +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_check_transfer_len (req, tl);return ;@@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)nvmet_ctrl_fatal_error (ctrl); complete: +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_req_complete (req, status); }@@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)u16 status =0 ;mutex_lock (&req->sq->auth_lock); +mutex_lock (&ctrl->auth_lock);if (req->cmd->auth_receive.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ;@@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) }if (unlikely (al != req->transfer_len)) {pr_debug ("%s: transfer length mismatch (%u)\n" , __func__, al); +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_check_transfer_len (req, al);return ;@@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)nvmet_auth_sq_free (req->sq);nvmet_ctrl_fatal_error (ctrl); } +mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);nvmet_req_complete (req, status); }drivers/nvme/target/nvmet.h +0
-0
drivers/nvme/target/nvmet.h struct nvmet_ctrl { · line 313
Quoted 7 lines
drivers/nvme/target/nvmet.h +1
-0@@ -307,6 +307,7 @@ struct nvmet_ctrl {struct nvme_error_slot slots[NVMET_ERROR_LOG_SLOTS ];bool pi_support;bool concat; +struct mutex auth_lock;/* one AUTH command at a time */ #ifdef CONFIG_NVME_TARGET_AUTH struct nvme_dhchap_key *host_key;struct nvme_dhchap_key *ctrl_key;
Hmm. Not particularly happy with this.
In the end, it renders sq->auth_lock pretty much pointless (the only
place where it remains on its own is in nvmet_auth_expiry_work() ).
And it would trigger if a authentication reset while another authentication (eg on the I/O queues) is still ongoing. With this patch we would switch the authentication variables
in the middle of the authentication protocol run for the I/O
queue, most likely causing that authentication to fail.
We should rather fail ongoing authentications whenever we call nvmet_auth_setup().
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 60/98 nvmet-auth: keep the negotiated hash per queue
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Every queue runs its own DH-HMAC-CHAP transaction, but the hash that AUTH_Negotiate settles on is stored in the controller. A host that doesn't offer the configured hash gets a fallback, and that changes ctrl->shash_id under the transactions of the other queues, whose challenges have the length of the old hash:
qid 1: AUTH_Negotiate (SHA-256), 32 byte challenge qid 0: AUTH_Negotiate (SHA-512 only)
qid 1: DH-HMAC-CHAP_Reply with a hash length of 64
BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530
No secret is needed for that.
Store the negotiated hash in the queue and use it for the rest of the transaction. ctrl->shash_id stays the configured preference.
--- drivers/nvme/target/auth.c | 35 +++++++++++++------------- drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++----- drivers/nvme/target/nvmet.h | 1 + 3 files changed, 25 insertions(+), 23 deletions(-)
drivers/nvme/target/auth.c +18 -17
@@ -299,14 +299,14 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,if (IS_ERR (transformed_key))return PTR_ERR (transformed_key); - ret =nvme_auth_hmac_init (&hmac, ctrl->shash_id, transformed_key->key, - transformed_key->len); + ret =nvme_auth_hmac_init (&hmac, req->sq->shash_id, + transformed_key->key, transformed_key->len);if (ret) goto out_free_response; -if (shash_len !=nvme_auth_hmac_hash_len (ctrl->shash_id)) { +if (shash_len !=nvme_auth_hmac_hash_len (req->sq->shash_id)) {pr_err ("%s: hash len mismatch (len %u digest %zu)\n" , __func__, - shash_len,nvme_auth_hmac_hash_len (ctrl->shash_id)); + shash_len,nvme_auth_hmac_hash_len (req->sq->shash_id)); ret = -EINVAL ; goto out_free_response; }
@@ -317,7 +317,7 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response, ret = -ENOMEM ; goto out_free_response; } - ret = nvme_auth_augmented_challenge(ctrl->shash_id, + ret = nvme_auth_augmented_challenge(req->sq->shash_id, req->sq->dhchap_skey, req->sq->dhchap_skey_len, req->sq->dhchap_c1,
@@ -371,14 +371,14 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,if (IS_ERR (transformed_key))return PTR_ERR (transformed_key); - ret =nvme_auth_hmac_init (&hmac, ctrl->shash_id, transformed_key->key, - transformed_key->len); + ret =nvme_auth_hmac_init (&hmac, req->sq->shash_id, + transformed_key->key, transformed_key->len);if (ret) goto out_free_response; -if (shash_len !=nvme_auth_hmac_hash_len (ctrl->shash_id)) { +if (shash_len !=nvme_auth_hmac_hash_len (req->sq->shash_id)) {pr_err ("%s: hash len mismatch (len %u digest %zu)\n" , __func__, - shash_len,nvme_auth_hmac_hash_len (ctrl->shash_id)); + shash_len,nvme_auth_hmac_hash_len (req->sq->shash_id)); ret = -EINVAL ; goto out_free_response; }
@@ -389,7 +389,7 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response, ret = -ENOMEM ; goto out_free_response; } - ret = nvme_auth_augmented_challenge(ctrl->shash_id, + ret = nvme_auth_augmented_challenge(req->sq->shash_id, req->sq->dhchap_skey, req->sq->dhchap_skey_len, req->sq->dhchap_c2,
@@ -456,7 +456,8 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,struct nvmet_ctrl *ctrl = req->sq->ctrl;int ret; - req->sq->dhchap_skey_len =nvme_auth_hmac_hash_len (ctrl->shash_id); + req->sq->dhchap_skey_len = +nvme_auth_hmac_hash_len (req->sq->shash_id); req->sq->dhchap_skey =kzalloc (req->sq->dhchap_skey_len,GFP_KERNEL );if (!req->sq->dhchap_skey)return -ENOMEM ;
@@ -464,7 +465,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req, pkey, pkey_size, req->sq->dhchap_skey, req->sq->dhchap_skey_len, - ctrl->shash_id); + req->sq->shash_id);if (ret)pr_debug ("failed to compute session key, err %d\n" , ret);#ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
@@ -478,7 +479,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,void nvmet_auth_insert_psk (struct nvmet_sq *sq) { -int hash_len =nvme_auth_hmac_hash_len (sq->ctrl->shash_id); +int hash_len =nvme_auth_hmac_hash_len (sq->shash_id);u8 *psk, *tls_psk;char *digest;size_t psk_len;
@@ -487,7 +488,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)struct key *tls_key =NULL ;#endif - ret = nvme_auth_generate_psk(sq->ctrl->shash_id, + ret = nvme_auth_generate_psk(sq->shash_id, sq->dhchap_skey, sq->dhchap_skey_len, sq->dhchap_c1, sq->dhchap_c2,
@@ -497,7 +498,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq) __func__, sq->ctrl->cntlid, sq->qid, ret);return ; } - ret =nvme_auth_generate_digest (sq->ctrl->shash_id, psk, psk_len, + ret =nvme_auth_generate_digest (sq->shash_id, psk, psk_len, sq->ctrl->subsys->subsysnqn, sq->ctrl->hostnqn, &digest);if (ret) {
@@ -505,7 +506,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq) __func__, sq->ctrl->cntlid, sq->qid, ret); goto out_free_psk; } - ret =nvme_auth_derive_tls_psk (sq->ctrl->shash_id, psk, psk_len, + ret =nvme_auth_derive_tls_psk (sq->shash_id, psk, psk_len, digest, &tls_psk);if (ret) { pr_warn("%s: ctrl %d qid %d failed to derive TLS PSK, error %d\n",
@@ -515,7 +516,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)#ifdef CONFIG_NVME_TARGET_TCP_TLS tls_key =nvme_tls_psk_refresh (NULL , sq->ctrl->hostnqn, sq->ctrl->subsys->subsysnqn, - sq->ctrl->shash_id, tls_psk, psk_len, + sq->shash_id, tls_psk, psk_len, digest);if (IS_ERR (tls_key)) { pr_warn("%s: ctrl %d qid %d failed to refresh key, error %ld\n",
drivers/nvme/target/fabrics-cmd-auth.c +6 -6
@@ -111,8 +111,9 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)pr_debug ("%s: ctrl %d qid %d: no usable hash found, falling back to %s\n" , __func__, ctrl->cntlid, req->sq->qid,nvme_auth_hmac_name (fallback_hash_id)); - ctrl->shash_id = fallback_hash_id; + hash_id = fallback_hash_id; } + req->sq->shash_id = hash_id; dhgid =-1 ; fallback_dhgid =-1 ;
@@ -454,7 +455,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)struct nvmf_auth_dhchap_challenge_data *data = d;struct nvmet_ctrl *ctrl = req->sq->ctrl;int ret =0 ; -int hash_len =nvme_auth_hmac_hash_len (ctrl->shash_id); +int hash_len =nvme_auth_hmac_hash_len (req->sq->shash_id);int data_size =sizeof (*data) + hash_len;if (ctrl->dh_tfm)
@@ -469,7 +470,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al) data->auth_type =NVME_AUTH_DHCHAP_MESSAGES ; data->auth_id =NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE ; data->t_id =cpu_to_le16 (req->sq->dhchap_tid); - data->hashid = ctrl->shash_id; + data->hashid = req->sq->shash_id; data->hl = hash_len; data->seqnum =cpu_to_le32 (req->sq->dhchap_s1); req->sq->dhchap_c1 =kmalloc (data->hl,GFP_KERNEL );
@@ -493,7 +494,7 @@ static int nvmet_auth_success1(struct nvmet_req *req, void *d, int al) {struct nvmf_auth_dhchap_success1_data *data = d;struct nvmet_ctrl *ctrl = req->sq->ctrl; -int hash_len =nvme_auth_hmac_hash_len (ctrl->shash_id); +int hash_len =nvme_auth_hmac_hash_len (req->sq->shash_id);WARN_ON (al <sizeof (*data));memset (data,0 ,sizeof (*data));
@@ -530,7 +531,6 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al)u32 nvmet_auth_receive_data_len (struct nvmet_req *req) { -struct nvmet_ctrl *ctrl = req->sq->ctrl;u32 al =le32_to_cpu (req->cmd->auth_receive.al);u32 min_len;
@@ -544,7 +544,7 @@ u32 nvmet_auth_receive_data_len(struct nvmet_req *req)case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1 : min_len =sizeof (struct nvmf_auth_dhchap_success1_data );if (req->sq->dhchap_c2) - min_len +=nvme_auth_hmac_hash_len (ctrl->shash_id); + min_len +=nvme_auth_hmac_hash_len (req->sq->shash_id);break ;default : min_len =sizeof (struct nvmf_auth_dhchap_failure_data );
drivers/nvme/target/nvmet.h +1 -0
@@ -167,6 +167,7 @@ struct nvmet_sq {bool auth_expiry_active;u16 dhchap_tid;u8 sc_c; +u8 shash_id;u8 dhchap_status;u8 dhchap_step;u8 *dhchap_c1;
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 60/98] nvmet-auth: keep the negotiated hash per queue
Quoted 26 lines · Jens Axboe
From: Josef Bacik <[email protected]>
Every queue runs its own DH-HMAC-CHAP transaction, but the hash that
AUTH_Negotiatesettles on is stored in the controller. A host that doesn't offer the configured hash gets a fallback, and that changesctrl->shash_idunder the transactions of the other queues, whose challenges have the length of the old hash:qid 1:
AUTH_Negotiate(SHA-256), 32 byte challenge qid 0:AUTH_Negotiate(SHA-512 only)
qid 1: DH-HMAC-CHAP_Reply with a hash length of 64BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530No secret is needed for that.
Store the negotiated hash in the queue and use it for the rest of the transaction.
ctrl->shash_idstays the configured preference.Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 6 lines
--- drivers/nvme/target/auth.c | 35 +++++++++++++------------- drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++----- drivers/nvme/target/nvmet.h | 1 + 3 files changed, 25 insertions(+), 23 deletions(-)
Weird. But spec seems to be okay with that, I guess. But then we don't really need to the shash_id from the controller, so it should be removed from there.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 61/98 nvmet-auth: authenticate the I/O queues again
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Since commit c91a20129185 the target neither asks a host to authenticate an I/O queue nor checks that it did: nvmet_check_auth_status() is true for everything but the admin queue. Nothing else ties an I/O queue to the admin queue's authentication. The Connect of an I/O queue names its controller by subsystem NQN, host NQN and cntlid, so anyone who can reach the port and knows the NQNs can attach a queue to an authenticated controller and do I/O. Property Set isn't subject to authentication either, so an unauthenticated host can enable its own controller and connect an I/O queue to that.
Ask for authentication in the Connect response of an I/O queue and refuse its commands until it has passed, as before that commit. The Linux host authenticates every queue that is asked to. A queue with a TLS session is exempt, its peer held a PSK the port accepts, and since commit ecf4d2d88351 such a controller does get a host key when the admin queue re-authenticates.
--- drivers/nvme/target/auth.c | 14 +++++++------- drivers/nvme/target/fabrics-cmd.c | 4 ---- 2 files changed, 7 insertions(+), 11 deletions(-)
drivers/nvme/target/auth.c +7 -7
@@ -275,13 +275,13 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)bool nvmet_check_auth_status (struct nvmet_req *req) { -if (req->sq->ctrl->host_key) { -if (req->sq->qid >0 ) -return true; -if (!req->sq->authenticated) -return false; - } -return true; +struct nvmet_sq *sq = req->sq; + +if (!sq->ctrl->host_key || sq->authenticated) +return true; + +/* Whoever set up the TLS session of an I/O queue knew the PSK. */ +return sq->qid &&nvmet_queue_tls_keyid (sq); } int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
drivers/nvme/target/fabrics-cmd.c +0 -4
@@ -251,10 +251,6 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq)bool needs_auth =nvmet_has_auth (ctrl, sq);key_serial_t keyid =nvmet_queue_tls_keyid (sq); -/* Do not authenticate I/O queues */ -if (sq->qid) - needs_auth = false; -if (keyid) pr_debug("%s: ctrl %d qid %d should %sauthenticate, tls psk %08x\n", __func__, ctrl->cntlid, sq->qid,
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 62/98 nvmet: don't mix queues with and without TLS
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
Since commit fa2e0f8bbc68 a controller whose admin queue uses TLS gets no host key, the PSK stands in for DH-HMAC-CHAP. But a port with TREQ "not required" takes plain connections too, as does any other port the subsystem is linked to, and the Connect of an I/O queue names its controller by NQNs and cntlid only. So anyone who reaches such a port can add a plain queue to a TLS controller, and with no host key nothing is asked of it. The other way round, a TLS queue skips DH-HMAC-CHAP on a controller whose other queues had to pass it.
A host uses TLS for all queues or none. Refuse an I/O queue Connect that differs from the admin queue in that.
--- drivers/nvme/target/core.c | 1 + drivers/nvme/target/fabrics-cmd.c | 9 +++++++++ drivers/nvme/target/nvmet.h | 1 + 3 files changed, 11 insertions(+)
drivers/nvme/target/core.c +1 -0
@@ -1646,6 +1646,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) ctrl->port = args->port; ctrl->ops = args->ops; + ctrl->tls = args->sq &&nvmet_queue_tls_keyid (args->sq);#ifdef CONFIG_NVME_TARGET_PASSTHRU /* By default, set loop targets to clear IDS by default */
drivers/nvme/target/fabrics-cmd.c +9 -0
@@ -377,6 +377,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) goto out_ctrl_put; } +/* All queues of a controller use TLS, or none of them does. */ +if (ctrl->tls != !!nvmet_queue_tls_keyid (req->sq)) { +pr_warn ("ctrl %d: queue %d with%s TLS\n" , ctrl->cntlid, qid, + ctrl->tls ?"out" :"" ); + status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; + req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (cntlid); + goto out_ctrl_put; + } + status =nvmet_install_queue (ctrl, req);if (status) goto out_ctrl_put;
drivers/nvme/target/nvmet.h +1 -0
@@ -308,6 +308,7 @@ struct nvmet_ctrl {struct nvme_error_slot slots[NVMET_ERROR_LOG_SLOTS ];bool pi_support;bool concat; +bool tls;struct mutex auth_lock;/* one AUTH command at a time */ #ifdef CONFIG_NVME_TARGET_AUTH struct nvme_dhchap_key *host_key;
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 62/98] nvmet: don't mix queues with and without TLS
Quoted 63 lines · Jens Axboe
From: Josef Bacik <[email protected]>
Since commit
fa2e0f8bbc68a controller whose admin queue uses TLS gets no host key, the PSK stands in for DH-HMAC-CHAP. But a port with TREQ "not required" takes plain connections too, as does any other port the subsystem is linked to, and the Connect of an I/O queue names its controller by NQNs and cntlid only. So anyone who reaches such a port can add a plain queue to a TLS controller, and with no host key nothing is asked of it. The other way round, a TLS queue skips DH-HMAC-CHAP on a controller whose other queues had to pass it.A host uses TLS for all queues or none. Refuse an I/O queue Connect that differs from the admin queue in that.
Fixes fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/core.c | 1 + drivers/nvme/target/fabrics-cmd.c | 9 +++++++++ drivers/nvme/target/nvmet.h | 1 + 3 files changed, 11 insertions(+)drivers/nvme/target/core.c +1
-0@@ -1646,6 +1646,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) ctrl->port = args->port; ctrl->ops = args->ops; + ctrl->tls = args->sq &&nvmet_queue_tls_keyid (args->sq);#ifdef CONFIG_NVME_TARGET_PASSTHRU /* By default, set loop targets to clear IDS by default */ drivers/nvme/target/fabrics-cmd.c +9
-0@@ -377,6 +377,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) goto out_ctrl_put; } +/* All queues of a controller use TLS, or none of them does. */ +if (ctrl->tls != !!nvmet_queue_tls_keyid (req->sq)) { +pr_warn ("ctrl %d: queue %d with%s TLS\n" , ctrl->cntlid, qid, + ctrl->tls ?"out" :"" ); + status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; + req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (cntlid); + goto out_ctrl_put; + } + status =nvmet_install_queue (ctrl, req);if (status) goto out_ctrl_put;drivers/nvme/target/nvmet.h +0
-0
drivers/nvme/target/nvmet.h struct nvmet_ctrl { · line 314
Quoted 7 lines
drivers/nvme/target/nvmet.h +1
-0@@ -308,6 +308,7 @@ struct nvmet_ctrl {struct nvme_error_slot slots[NVMET_ERROR_LOG_SLOTS ];bool pi_support;bool concat; +bool tls;struct mutex auth_lock;/* one AUTH command at a time */ #ifdef CONFIG_NVME_TARGET_AUTH struct nvme_dhchap_key *host_key;
Hmm. Technically TLS is a feature of the TCP transport and only applicable there. We should delegate this step to the TCP transport.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 63/98 nvmet-auth: refuse a reply without a DH value
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
With a DH group configured, nvmet_auth_host_hash() augments the challenge with the queue's session key, but nvmet_auth_reply() only computes that key if the DH-HMAC-CHAP_Reply carries a DH value. Without one, whatever the queue holds is used: nothing on the first transaction, so the DH exchange is skipped, and after nvmet_auth_sq_free() a NULL pointer with the hash length still set:
BUG: KASAN: null-ptr-deref in __hmac_sha256_preparekey+0xa5/0x250 nvme_auth_hmac+0xb7/0x160 nvmet_auth_host_hash+0x2a5/0x530
Refuse such a reply, as the host refuses a challenge without a DH value, and clear the length along with the pointer.
--- drivers/nvme/target/auth.c | 1 + drivers/nvme/target/fabrics-cmd-auth.c | 3 +++ 2 files changed, 4 insertions(+)
drivers/nvme/target/auth.c +1 -0
@@ -237,6 +237,7 @@ void nvmet_auth_sq_free(struct nvmet_sq *sq) sq->dhchap_c2 =NULL ;kfree (sq->dhchap_skey); sq->dhchap_skey =NULL ; + sq->dhchap_skey_len =0 ; }void nvmet_auth_sq_destroy (struct nvmet_sq *sq)
drivers/nvme/target/fabrics-cmd-auth.c +3 -0
@@ -180,6 +180,9 @@ static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)if (nvmet_auth_ctrl_sesskey (req, data->rval +2 * data->hl, dhvlen) <0 )return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE ; + }else if (ctrl->dh_gid !=NVME_AUTH_DHGROUP_NULL ) { +/* No DH value, no session key for the augmented challenge. */ +return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD ; } response =kmalloc (data->hl,GFP_KERNEL );
Signature · 1 line
2.56.0
Re: [PATCH 63/98] nvmet-auth: refuse a reply without a DH value
Quoted 23 lines · Jens Axboe
From: Josef Bacik <[email protected]>
With a DH group configured,
nvmet_auth_host_hash()augments the challenge with the queue's session key, butnvmet_auth_reply()only computes that key if the DH-HMAC-CHAP_Reply carries a DH value. Without one, whatever the queue holds is used: nothing on the first transaction, so the DH exchange is skipped, and afternvmet_auth_sq_free()a NULL pointer with the hash length still set:BUG: KASAN: null-ptr-deref in __hmac_sha256_preparekey+0xa5/0x250 nvme_auth_hmac+0xb7/0x160 nvmet_auth_host_hash+0x2a5/0x530Refuse such a reply, as the host refuses a challenge without a DH value, and clear the length along with the pointer.
Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 5 lines
--- drivers/nvme/target/auth.c | 1 + drivers/nvme/target/fabrics-cmd-auth.c | 3 +++ 2 files changed, 4 insertions(+)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 64/98 nvmet-auth: keep the old keys if the new ones fail
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
AUTH_Negotiate on the admin queue has nvmet_setup_auth() reload the host's keys, and it frees the old host key first. If decoding the new secret fails, or an allocation does, the controller is left without a host key, which means no authentication at all: nvmet_check_auth_status() passes everything and I/O queue Connects no longer ask for it. configfs accepts any string starting with "DHHC-1:" and a digit, so one typo while changing a secret opens the namespaces to any connected admin queue.
Decode the new keys first and only then replace the old ones.
--- drivers/nvme/target/auth.c | 69 ++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 41 deletions(-)
drivers/nvme/target/auth.c +28 -41
@@ -135,6 +135,7 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset)int ret =0 ;struct nvmet_host_link *p;struct nvmet_host *host =NULL ; +struct nvme_dhchap_key *host_key, *ctrl_key =NULL ;down_read (&nvmet_config_sem);if (nvmet_is_disc_subsys (ctrl->subsys))
@@ -172,55 +173,41 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset) goto out_unlock; } -if (host->dhchap_hash_id == ctrl->shash_id) { -pr_debug ("Re-use existing hash ID %d\n" , - ctrl->shash_id); - }else { - ctrl->shash_id = host->dhchap_hash_id; - } - -/* Skip the 'DHHC-1:XX:' prefix */ -nvme_auth_free_key (ctrl->host_key); - ctrl->host_key =nvme_auth_extract_key (host->dhchap_secret +10 , - host->dhchap_key_hash); -if (IS_ERR (ctrl->host_key)) { +/* +* Skip the 'DHHC-1:XX:' prefix. Keep the old keys until the new ones +* are there: a controller without a host key needs no authentication. +*/ + host_key =nvme_auth_extract_key (host->dhchap_secret +10 , + host->dhchap_key_hash); +if (IS_ERR (host_key)) { ret =NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE ; - ctrl->host_key =NULL ; - goto out_free_hash; + goto out_unlock; } +if (host->dhchap_ctrl_secret) { + ctrl_key =nvme_auth_extract_key (host->dhchap_ctrl_secret +10 , + host->dhchap_ctrl_key_hash); +if (IS_ERR (ctrl_key)) { +nvme_auth_free_key (host_key); + ret =NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE ; + goto out_unlock; + } + } +nvme_auth_free_key (ctrl->host_key); + ctrl->host_key = host_key; +nvme_auth_free_key (ctrl->ctrl_key); + ctrl->ctrl_key = ctrl_key; + ctrl->shash_id = host->dhchap_hash_id;#ifdef CONFIG_NVME_TARGET_AUTH_DEBUG pr_debug ("%s: using hash %s key %*ph\n" , __func__, ctrl->host_key->hash >0 ?nvme_auth_hmac_name (ctrl->host_key->hash) :"none" , (int )ctrl->host_key->len, ctrl->host_key->key); +if (ctrl->ctrl_key) +pr_debug ("%s: using ctrl hash %s key %*ph\n" , __func__, + ctrl->ctrl_key->hash >0 ? +nvme_auth_hmac_name (ctrl->ctrl_key->hash) :"none" , + (int )ctrl->ctrl_key->len, ctrl->ctrl_key->key);#endif -nvme_auth_free_key (ctrl->ctrl_key); -if (!host->dhchap_ctrl_secret) { - ctrl->ctrl_key =NULL ; - goto out_unlock; - } - - ctrl->ctrl_key =nvme_auth_extract_key (host->dhchap_ctrl_secret +10 , - host->dhchap_ctrl_key_hash); -if (IS_ERR (ctrl->ctrl_key)) { - ret =NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE ; - ctrl->ctrl_key =NULL ; - goto out_free_hash; - } -#ifdef CONFIG_NVME_TARGET_AUTH_DEBUG -pr_debug ("%s: using ctrl hash %s key %*ph\n" , __func__, - ctrl->ctrl_key->hash >0 ? -nvme_auth_hmac_name (ctrl->ctrl_key->hash) :"none" , - (int )ctrl->ctrl_key->len, ctrl->ctrl_key->key); -#endif -out_free_hash: -if (ret) { -if (ctrl->host_key) { -nvme_auth_free_key (ctrl->host_key); - ctrl->host_key =NULL ; - } - ctrl->shash_id =0 ; - } out_unlock:up_read (&nvmet_config_sem);
Signature · 1 line
2.56.0
Re: [PATCH 64/98] nvmet-auth: keep the old keys if the new ones fail
Quoted 20 lines · Jens Axboe
From: Josef Bacik <[email protected]>
AUTH_Negotiateon the admin queue hasnvmet_setup_auth()reload the host's keys, and it frees the old host key first. If decoding the new secret fails, or an allocation does, the controller is left without a host key, which means no authentication at all:nvmet_check_auth_status()passes everything and I/O queue Connects no longer ask for it. configfs accepts any string starting with "DHHC-1:" and a digit, so one typo while changing a secret opens the namespaces to any connected admin queue.Decode the new keys first and only then replace the old ones.
Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 4 lines
--- drivers/nvme/target/auth.c | 69 ++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 41 deletions(-)
That discussion again. I give in.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 65/98 nvmet-auth: take only AUTH_Negotiate for a negotiation
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_execute_auth_send() hands a common-type message to nvmet_auth_negotiate() if its ID is AUTH_Negotiate or matches the step the queue is in. The second case is wrong: the only other common-type message a host sends is AUTH_Failure2, and no queue waits in that step. A host can name the step on purpose though, and its message is then treated as a new AUTH_Negotiate without the work that goes with a new transaction, like reloading the keys on the admin queue.
Fail every other common-type message, as AUTH_Failure2 already is.
--- drivers/nvme/target/fabrics-cmd-auth.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/fabrics-cmd-auth.c +2 -1
@@ -350,8 +350,9 @@ void nvmet_execute_auth_send(struct nvmet_req *req) } req->sq->dhchap_step =NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE ; - }else if (data->auth_id != req->sq->dhchap_step) + }else { goto done_failure1; + }/* Validate negotiation parameters */ dhchap_status =nvmet_auth_negotiate (req, d, tl);if (dhchap_status ==0 )
Signature · 1 line
2.56.0
Re: [PATCH 65/98] nvmet-auth: take only AUTH_Negotiate for a negotiation
Quoted 28 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_execute_auth_send()hands a common-type message tonvmet_auth_negotiate()if its ID isAUTH_Negotiateor matches the step the queue is in. The second case is wrong: the only other common-type message a host sends isAUTH_Failure2, and no queue waits in that step. A host can name the step on purpose though, and its message is then treated as a newAUTH_Negotiatewithout the work that goes with a new transaction, like reloading the keys on the admin queue.Fail every other common-type message, as
AUTH_Failure2already is.Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/fabrics-cmd-auth.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/target/fabrics-cmd-auth.c +0
-0
drivers/nvme/target/fabrics-cmd-auth.c void nvmet_execute_auth_send(struct nvmet_req *req) · line 358
Quoted 10 lines
drivers/nvme/target/fabrics-cmd-auth.c +2
-1@@ -350,8 +350,9 @@ void nvmet_execute_auth_send(struct nvmet_req *req) } req->sq->dhchap_step =NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE ; - }else if (data->auth_id != req->sq->dhchap_step) + }else { goto done_failure1; + }/* Validate negotiation parameters */ dhchap_status =nvmet_auth_negotiate (req, d, tl);if (dhchap_status ==0 )
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 66/98 nvmet-auth: free the old data on a new AUTH_Negotiate
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A host may restart the negotiation at any step. The challenge and the session key of the abandoned transaction stay in the queue, and nvmet_auth_challenge() and nvmet_auth_ctrl_sesskey() overwrite the pointers with new allocations, leaking one or two hash-sized buffers per round for anyone who can reach the port. The controller challenge stays too, and if the new transaction brings none but asks for the controller's response, the old one is hashed with the new hash length:
BUG: KASAN: slab-out-of-bounds in nvmet_auth_ctrl_hash+0x2d5/0x4b0
Free the old transaction's data when a new one begins.
--- drivers/nvme/target/fabrics-cmd-auth.c | 1 + 1 file changed, 1 insertion(+)
drivers/nvme/target/fabrics-cmd-auth.c +1 -0
@@ -336,6 +336,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)/* Restart negotiation */ pr_debug ("%s: ctrl %d qid %d reset negotiation\n" , __func__, ctrl->cntlid, req->sq->qid); +nvmet_auth_sq_free (req->sq);if (!req->sq->qid) { dhchap_status =nvmet_setup_auth (ctrl, req->sq, true);
Signature · 1 line
2.56.0
Re: [PATCH 66/98] nvmet-auth: free the old data on a new AUTH_Negotiate
Quoted 30 lines · Jens Axboe
From: Josef Bacik <[email protected]>
A host may restart the negotiation at any step. The challenge and the session key of the abandoned transaction stay in the queue, and
nvmet_auth_challenge()andnvmet_auth_ctrl_sesskey()overwrite the pointers with new allocations, leaking one or two hash-sized buffers per round for anyone who can reach the port. The controller challenge stays too, and if the new transaction brings none but asks for the controller's response, the old one is hashed with the new hash length:BUG: KASAN: slab-out-of-bounds in nvmet_auth_ctrl_hash+0x2d5/0x4b0Free the old transaction's data when a new one begins.
Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/fabrics-cmd-auth.c | 1 + 1 file changed, 1 insertion(+)drivers/nvme/target/fabrics-cmd-auth.c +0
-0
drivers/nvme/target/fabrics-cmd-auth.c void nvmet_execute_auth_send(struct nvmet_req *req) · line 342
Quoted 7 lines
drivers/nvme/target/fabrics-cmd-auth.c +1
-0@@ -336,6 +336,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)/* Restart negotiation */ pr_debug ("%s: ctrl %d qid %d reset negotiation\n" , __func__, ctrl->cntlid, req->sq->qid); +nvmet_auth_sq_free (req->sq);if (!req->sq->qid) { dhchap_status =nvmet_setup_auth (ctrl, req->sq, true);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 67/98 nvmet-auth: fix the check of the offered DH groups
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_auth_negotiate() matches the first DH group in the host's list that is NOT the configured one, and looks for a fallback among those that are. So a host offering only groups the target isn't configured for passes and gets a challenge for a group it never offered, while a host offering only the NULL group is refused by a target configured for NULL, since there's no KPP algorithm to fall back to. A fallback can't work anyway, the DH context and public key belong to the configured group.
Accept if the configured group is in the list, fail with "DH group unusable" otherwise.
--- drivers/nvme/target/fabrics-cmd-auth.c | 27 +++++--------------------- 1 file changed, 5 insertions(+), 22 deletions(-)
drivers/nvme/target/fabrics-cmd-auth.c +5 -22
@@ -47,7 +47,7 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl) {struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvmf_auth_dhchap_negotiate_data *data = d; -int i, hash_id =0 , fallback_hash_id =0 , dhgid, fallback_dhgid; +int i, hash_id =0 , fallback_hash_id =0 ;if (tl <sizeof (*data) +sizeof (struct nvmf_auth_dhchap_protocol_descriptor ))
@@ -115,32 +115,15 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl) } req->sq->shash_id = hash_id; - dhgid =-1 ; - fallback_dhgid =-1 ;for (i =0 ; i < data->auth_protocol[0 ].dhchap.dhlen; i++) { -int tmp_dhgid = data->auth_protocol[0 ].dhchap.idlist[i +30 ]; - -if (tmp_dhgid != ctrl->dh_gid) { - dhgid = tmp_dhgid; +if (data->auth_protocol[0 ].dhchap.idlist[i +30 ] == ctrl->dh_gid)break ; - } -if (fallback_dhgid <0 ) { -const char *kpp =nvme_auth_dhgroup_kpp (tmp_dhgid); - -if (crypto_has_kpp (kpp,0 ,0 )) - fallback_dhgid = tmp_dhgid; - } } -if (dhgid <0 ) { -if (fallback_dhgid <0 ) { -pr_debug ("%s: ctrl %d qid %d: no usable DH group found\n" , - __func__, ctrl->cntlid, req->sq->qid); -return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE ; - } +if (i == data->auth_protocol[0 ].dhchap.dhlen) {pr_debug ("%s: ctrl %d qid %d: configured DH group %s not found\n" , __func__, ctrl->cntlid, req->sq->qid, -nvme_auth_dhgroup_name (fallback_dhgid)); - ctrl->dh_gid = fallback_dhgid; +nvme_auth_dhgroup_name (ctrl->dh_gid)); +return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE ; }if (ctrl->dh_gid ==NVME_AUTH_DHGROUP_NULL && ctrl->concat) { pr_debug("%s: ctrl %d qid %d: NULL DH group invalid "
Signature · 1 line
2.56.0
Re: [PATCH 67/98] nvmet-auth: fix the check of the offered DH groups
Quoted 21 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_auth_negotiate()matches the first DH group in the host's list that is NOT the configured one, and looks for a fallback among those that are. So a host offering only groups the target isn't configured for passes and gets a challenge for a group it never offered, while a host offering only the NULL group is refused by a target configured for NULL, since there's no KPP algorithm to fall back to. A fallback can't work anyway, the DH context and public key belong to the configured group.Accept if the configured group is in the list, fail with "DH group unusable" otherwise.
Fixes 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 4 lines
--- drivers/nvme/target/fabrics-cmd-auth.c | 27 +++++--------------------- 1 file changed, 5 insertions(+), 22 deletions(-)
Yeah, the fallback idea was not great.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 68/98 nvmet: accept I/O queues on the controller's port only
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_ctrl_find_get() looks the subsystem up through the port of the request, but then takes the controller from the subsystem list whatever port it was created on. A controller belongs to one port and one
transport: ctrl->ops is that of the admin queue, and both ->install_queue() and ->delete_ctrl() assume the queues are their own. So an I/O queue arriving on another port of the same subsystem, possibly on another transport, is installed into a controller it has no part in, nvmet_tcp_install_queue() writes to it as a TCP queue, and an error on it deletes that controller. The Connect comes before any authentication.
Skip controllers of other ports, and compare the transport too since a port's transport can be changed while it's disabled.
--- drivers/nvme/target/core.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/nvme/target/core.c +5 -0
@@ -1507,6 +1507,11 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn,pr_warn ("hostnqn mismatch.\n" );continue ; } +/* All queues of a controller come in by its port. */ +if (ctrl->port != req->port || ctrl->ops != req->ops) { +pr_warn ("port mismatch.\n" ); +continue ; + }if (!kref_get_unless_zero (&ctrl->ref))continue ;
Signature · 1 line
2.56.0
Re: [PATCH 68/98] nvmet: accept I/O queues on the controller's port only
Quoted 31 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_ctrl_find_get()looks the subsystem up through the port of the request, but then takes the controller from the subsystem list whatever port it was created on. A controller belongs to one port and one
transport:ctrl->opsis that of the admin queue, and both ->install_queue() and ->delete_ctrl() assume the queues are their own. So an I/O queue arriving on another port of the same subsystem, possibly on another transport, is installed into a controller it has no part in,nvmet_tcp_install_queue()writes to it as a TCP queue, and an error on it deletes that controller. The Connect comes before any authentication.Skip controllers of other ports, and compare the transport too since a port's transport can be changed while it's disabled.
Fixes a07b4970f464 ("nvmet: add a generic NVMe target")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/core.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/nvme/target/core.c +0
-0@@ -1507,6 +1507,11 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn,pr_warn ("hostnqn mismatch.\n" );
drivers/nvme/target/core.c struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn, · line 1517
Quoted 10 lines
drivers/nvme/target/core.c +5
-0continue ; } +/* All queues of a controller come in by its port. */ +if (ctrl->port != req->port || ctrl->ops != req->ops) { +pr_warn ("port mismatch.\n" ); +continue ; + }if (!kref_get_unless_zero (&ctrl->ref))continue ;
Ouch.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 69/98 nvmet: confine fatal errors of unauthenticated queues
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The Connect of an I/O queue names its controller by NQNs and cntlid and needs no secret, DH-HMAC-CHAP comes after it. But a fatal error on any queue, a failed authentication or a transport error like an unknown PDU type, takes the whole controller down. So anyone who can reach the port and knows the NQNs can attach a queue to an authenticated host's controller and kick that host out, repeatedly.
Add a ->disconnect_queue() transport op and, for an I/O queue that hasn't passed the authentication asked of it, disconnect just that queue. Transports without the op (fc, loop) take the controller down as before. A queue that had authenticated and fails to do so again is the host's own, there the controller goes down as before too.
[axboe: resolved against the per-queue auth_lock and expiry work that
is upstream now]
--- drivers/nvme/target/auth.c | 9 ++++++--- drivers/nvme/target/core.c | 16 ++++++++++++++++ drivers/nvme/target/fabrics-cmd-auth.c | 20 ++++++++++++++++---- drivers/nvme/target/nvmet.h | 7 +++++++ drivers/nvme/target/rdma.c | 9 ++++++++- drivers/nvme/target/tcp.c | 11 ++++++++++- 6 files changed, 63 insertions(+), 9 deletions(-)
drivers/nvme/target/auth.c +6 -3
@@ -261,10 +261,8 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)#endif } -bool nvmet_check_auth_status (struct nvmet_req *req) +bool nvmet_sq_auth_status (struct nvmet_sq *sq) { -struct nvmet_sq *sq = req->sq; -if (!sq->ctrl->host_key || sq->authenticated)return true;
@@ -272,6 +270,11 @@ bool nvmet_check_auth_status(struct nvmet_req *req)return sq->qid &&nvmet_queue_tls_keyid (sq); } +bool nvmet_check_auth_status (struct nvmet_req *req) +{ +return nvmet_sq_auth_status (req->sq); +} +int nvmet_auth_host_hash (struct nvmet_req *req,u8 *response,unsigned int shash_len) {
drivers/nvme/target/core.c +16 -0
@@ -1828,6 +1828,22 @@ void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvmet_ctrl_fatal_error); +/* +* A fatal error on one queue. The Connect command of an I/O queue needs no +* secret, so until the queue has authenticated it may be anybody's: do not +* let it take down the controller of the host whose NQN it has named. +*/ +void nvmet_sq_fatal_error (struct nvmet_sq *sq) +{ +struct nvmet_ctrl *ctrl = sq->ctrl; + +if (sq->qid && ctrl->ops->disconnect_queue && !nvmet_sq_auth_status (sq)) + ctrl->ops->disconnect_queue (sq); +else +nvmet_ctrl_fatal_error (ctrl); +} +EXPORT_SYMBOL_GPL (nvmet_sq_fatal_error); +ssize_t nvmet_ctrl_host_traddr (struct nvmet_ctrl *ctrl,char *traddr,size_t traddr_len) {
drivers/nvme/target/fabrics-cmd-auth.c +16 -4
@@ -33,6 +33,14 @@ static void nvmet_auth_expired_work(struct work_struct *work)mutex_unlock (&sq->auth_lock); } +static void nvmet_auth_fatal_error (struct nvmet_sq *sq,bool passed) +{ +if (passed) +nvmet_ctrl_fatal_error (sq->ctrl); +else +nvmet_sq_fatal_error (sq); +} +void nvmet_auth_sq_init (struct nvmet_sq *sq) {/* Initialize in-band authentication */
@@ -253,11 +261,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)void *d;u32 tl;u16 status =0 ; +bool passed;u8 dhchap_status;mutex_lock (&req->sq->auth_lock);mutex_lock (&ctrl->auth_lock); - +/* A queue that fails to authenticate again is no stranger's. */ + passed =nvmet_sq_auth_status (req->sq);if (req->cmd->auth_send.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
@@ -430,7 +440,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)/* Final states, clear up variables */ nvmet_auth_sq_free (req->sq);if (req->sq->dhchap_step ==NVME_AUTH_DHCHAP_MESSAGE_FAILURE2 ) -nvmet_ctrl_fatal_error (ctrl); +nvmet_auth_fatal_error (req->sq, passed); complete:mutex_unlock (&ctrl->auth_lock);
@@ -551,10 +561,12 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)void *d;u32 al;u16 status =0 ; +bool passed;mutex_lock (&req->sq->auth_lock);mutex_lock (&ctrl->auth_lock); - +/* A queue that fails to authenticate again is no stranger's. */ + passed =nvmet_sq_auth_status (req->sq);if (req->cmd->auth_receive.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
@@ -641,7 +653,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)nvmet_auth_sq_free (req->sq); else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {nvmet_auth_sq_free (req->sq); -nvmet_ctrl_fatal_error (ctrl); +nvmet_auth_fatal_error (req->sq, passed); }mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);
drivers/nvme/target/nvmet.h +7 -0
@@ -441,6 +441,7 @@ struct nvmet_fabrics_ops {int (*add_port)(struct nvmet_port *port);void (*remove_port)(struct nvmet_port *port);void (*delete_ctrl)(struct nvmet_ctrl *ctrl); +void (*disconnect_queue)(struct nvmet_sq *sq);void (*disc_traddr)(struct nvmet_req *req,struct nvmet_port *port,char *traddr); ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
@@ -622,6 +623,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);int nvmet_sq_init (struct nvmet_sq *sq,struct nvmet_cq *cq);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl); +void nvmet_sq_fatal_error (struct nvmet_sq *sq);void nvmet_update_cc (struct nvmet_ctrl *ctrl,u32 new);
@@ -989,6 +991,7 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);void nvmet_auth_sq_free (struct nvmet_sq *sq);void nvmet_auth_sq_destroy (struct nvmet_sq *sq);int nvmet_setup_dhgroup (struct nvmet_ctrl *ctrl,u8 dhgroup_id); +bool nvmet_sq_auth_status (struct nvmet_sq *sq);bool nvmet_check_auth_status (struct nvmet_req *req);int nvmet_auth_host_hash (struct nvmet_req *req,u8 *response,unsigned int hash_len);
@@ -1015,6 +1018,10 @@ static inline void nvmet_auth_sq_init(struct nvmet_sq *sq)static inline void nvmet_destroy_auth (struct nvmet_ctrl *ctrl) {};static inline void nvmet_auth_sq_free (struct nvmet_sq *sq) {};static inline void nvmet_auth_sq_destroy (struct nvmet_sq *sq) {}; +static inline bool nvmet_sq_auth_status (struct nvmet_sq *sq) +{ +return true; +}static inline bool nvmet_check_auth_status (struct nvmet_req *req) {return true;
drivers/nvme/target/rdma.c +8 -1
@@ -687,7 +687,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)static void nvmet_rdma_error_comp (struct nvmet_rdma_queue *queue) {if (queue->nvme_sq.ctrl) { -nvmet_ctrl_fatal_error (queue->nvme_sq.ctrl); +nvmet_sq_fatal_error (&queue->nvme_sq); }else {/* * we didn't setup the controller yet in case
@@ -1848,6 +1848,12 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,return ret; } +static void nvmet_rdma_disconnect_queue (struct nvmet_sq *sq) +{ +nvmet_rdma_queue_disconnect (container_of(sq ,struct nvmet_rdma_queue , +nvme_sq )); +} +static void nvmet_rdma_delete_ctrl (struct nvmet_ctrl *ctrl) {struct nvmet_rdma_queue *queue, *n;
@@ -2078,6 +2084,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = { .remove_port = nvmet_rdma_remove_port, .queue_response = nvmet_rdma_queue_response, .delete_ctrl = nvmet_rdma_delete_ctrl, + .disconnect_queue = nvmet_rdma_disconnect_queue, .disc_traddr = nvmet_rdma_disc_port_addr, .host_traddr = nvmet_rdma_host_port_addr, .get_mdts = nvmet_rdma_get_mdts,
drivers/nvme/target/tcp.c +10 -1
@@ -412,7 +412,7 @@ static void nvmet_tcp_socket_error(struct nvmet_tcp_queue *queue, int status)if (status == -EPIPE || status == -ECONNRESET || !queue->nvme_sq.ctrl)kernel_sock_shutdown (queue->sock,SHUT_RDWR );else -nvmet_ctrl_fatal_error (queue->nvme_sq.ctrl); +nvmet_sq_fatal_error (&queue->nvme_sq); }static int nvmet_tcp_map_data (struct nvmet_tcp_cmd *cmd)
@@ -2223,6 +2223,14 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)kfree (port); } +static void nvmet_tcp_disconnect_queue (struct nvmet_sq *sq) +{ +struct nvmet_tcp_queue *queue = + container_of(sq, struct nvmet_tcp_queue, nvme_sq); + +kernel_sock_shutdown (queue->sock,SHUT_RDWR ); +} +static void nvmet_tcp_delete_ctrl (struct nvmet_ctrl *ctrl) {struct nvmet_tcp_queue *queue;
@@ -2300,6 +2308,7 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = { .remove_port = nvmet_tcp_remove_port, .queue_response = nvmet_tcp_queue_response, .delete_ctrl = nvmet_tcp_delete_ctrl, + .disconnect_queue = nvmet_tcp_disconnect_queue, .install_queue = nvmet_tcp_install_queue, .disc_traddr = nvmet_tcp_disc_port_addr, .host_traddr = nvmet_tcp_host_port_addr,
Signature · 1 line
2.56.0
Hannes Reinecke
maintainerRe: [PATCH 69/98] nvmet: confine fatal errors of unauthenticated queues
Quoted 257 lines · Jens Axboe
From: Josef Bacik <[email protected]>
The Connect of an I/O queue names its controller by NQNs and cntlid and needs no secret, DH-HMAC-CHAP comes after it. But a fatal error on any queue, a failed authentication or a transport error like an unknown PDU type, takes the whole controller down. So anyone who can reach the port and knows the NQNs can attach a queue to an authenticated host's controller and kick that host out, repeatedly.
Add a ->disconnect_queue() transport op and, for an I/O queue that hasn't passed the authentication asked of it, disconnect just that queue. Transports without the op (fc, loop) take the controller down as before. A queue that had authenticated and fails to do so again is the host's own, there the controller goes down as before too.
[axboe: resolved against the per-queue
auth_lockand expiry work that
is upstream now]Fixes db1312dd9548 ("nvmet: implement basic In-Band Authentication")Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/auth.c | 9 ++++++--- drivers/nvme/target/core.c | 16 ++++++++++++++++ drivers/nvme/target/fabrics-cmd-auth.c | 20 ++++++++++++++++---- drivers/nvme/target/nvmet.h | 7 +++++++ drivers/nvme/target/rdma.c | 9 ++++++++- drivers/nvme/target/tcp.c | 11 ++++++++++- 6 files changed, 63 insertions(+), 9 deletions(-)drivers/nvme/target/auth.c +6
-3@@ -261,10 +261,8 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)#endif } -bool nvmet_check_auth_status (struct nvmet_req *req) +bool nvmet_sq_auth_status (struct nvmet_sq *sq) { -struct nvmet_sq *sq = req->sq; -if (!sq->ctrl->host_key || sq->authenticated)return true;@@ -272,6 +270,11 @@ bool nvmet_check_auth_status(struct nvmet_req *req)return sq->qid &&nvmet_queue_tls_keyid (sq); } +bool nvmet_check_auth_status (struct nvmet_req *req) +{ +return nvmet_sq_auth_status (req->sq); +} +int nvmet_auth_host_hash (struct nvmet_req *req,u8 *response,unsigned int shash_len) {drivers/nvme/target/core.c +16
-0@@ -1828,6 +1828,22 @@ void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvmet_ctrl_fatal_error); +/* +* A fatal error on one queue. The Connect command of an I/O queue needs no +* secret, so until the queue has authenticated it may be anybody's: do not +* let it take down the controller of the host whose NQN it has named. +*/ +void nvmet_sq_fatal_error (struct nvmet_sq *sq) +{ +struct nvmet_ctrl *ctrl = sq->ctrl; + +if (sq->qid && ctrl->ops->disconnect_queue && !nvmet_sq_auth_status (sq)) + ctrl->ops->disconnect_queue (sq); +else +nvmet_ctrl_fatal_error (ctrl); +} +EXPORT_SYMBOL_GPL (nvmet_sq_fatal_error); +ssize_t nvmet_ctrl_host_traddr (struct nvmet_ctrl *ctrl,char *traddr,size_t traddr_len) {drivers/nvme/target/fabrics-cmd-auth.c +16
-4@@ -33,6 +33,14 @@ static void nvmet_auth_expired_work(struct work_struct *work)mutex_unlock (&sq->auth_lock); } +static void nvmet_auth_fatal_error (struct nvmet_sq *sq,bool passed) +{ +if (passed) +nvmet_ctrl_fatal_error (sq->ctrl); +else +nvmet_sq_fatal_error (sq); +} +void nvmet_auth_sq_init (struct nvmet_sq *sq) {/* Initialize in-band authentication */ @@ -253,11 +261,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)void *d;u32 tl;u16 status =0 ; +bool passed;u8 dhchap_status;mutex_lock (&req->sq->auth_lock);mutex_lock (&ctrl->auth_lock); - +/* A queue that fails to authenticate again is no stranger's. */ + passed =nvmet_sq_auth_status (req->sq);if (req->cmd->auth_send.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =@@ -430,7 +440,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)/* Final states, clear up variables */ nvmet_auth_sq_free (req->sq);if (req->sq->dhchap_step ==NVME_AUTH_DHCHAP_MESSAGE_FAILURE2 ) -nvmet_ctrl_fatal_error (ctrl); +nvmet_auth_fatal_error (req->sq, passed); complete:mutex_unlock (&ctrl->auth_lock);@@ -551,10 +561,12 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)void *d;u32 al;u16 status =0 ; +bool passed;mutex_lock (&req->sq->auth_lock);mutex_lock (&ctrl->auth_lock); - +/* A queue that fails to authenticate again is no stranger's. */ + passed =nvmet_sq_auth_status (req->sq);if (req->cmd->auth_receive.secp !=NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER ) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =@@ -641,7 +653,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)nvmet_auth_sq_free (req->sq); else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {nvmet_auth_sq_free (req->sq); -nvmet_ctrl_fatal_error (ctrl); +nvmet_auth_fatal_error (req->sq, passed); }mutex_unlock (&ctrl->auth_lock);mutex_unlock (&req->sq->auth_lock);drivers/nvme/target/nvmet.h +7
-0@@ -441,6 +441,7 @@ struct nvmet_fabrics_ops {int (*add_port)(struct nvmet_port *port);void (*remove_port)(struct nvmet_port *port);void (*delete_ctrl)(struct nvmet_ctrl *ctrl); +void (*disconnect_queue)(struct nvmet_sq *sq);void (*disc_traddr)(struct nvmet_req *req,struct nvmet_port *port,char *traddr); ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,@@ -622,6 +623,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);int nvmet_sq_init (struct nvmet_sq *sq,struct nvmet_cq *cq);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl); +void nvmet_sq_fatal_error (struct nvmet_sq *sq);void nvmet_update_cc (struct nvmet_ctrl *ctrl,u32 new);@@ -989,6 +991,7 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);void nvmet_auth_sq_free (struct nvmet_sq *sq);void nvmet_auth_sq_destroy (struct nvmet_sq *sq);int nvmet_setup_dhgroup (struct nvmet_ctrl *ctrl,u8 dhgroup_id); +bool nvmet_sq_auth_status (struct nvmet_sq *sq);bool nvmet_check_auth_status (struct nvmet_req *req);int nvmet_auth_host_hash (struct nvmet_req *req,u8 *response,unsigned int hash_len);@@ -1015,6 +1018,10 @@ static inline void nvmet_auth_sq_init(struct nvmet_sq *sq)static inline void nvmet_destroy_auth (struct nvmet_ctrl *ctrl) {};static inline void nvmet_auth_sq_free (struct nvmet_sq *sq) {};static inline void nvmet_auth_sq_destroy (struct nvmet_sq *sq) {}; +static inline bool nvmet_sq_auth_status (struct nvmet_sq *sq) +{ +return true; +}static inline bool nvmet_check_auth_status (struct nvmet_req *req) {return true;drivers/nvme/target/rdma.c +8
-1@@ -687,7 +687,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)static void nvmet_rdma_error_comp (struct nvmet_rdma_queue *queue) {if (queue->nvme_sq.ctrl) { -nvmet_ctrl_fatal_error (queue->nvme_sq.ctrl); +nvmet_sq_fatal_error (&queue->nvme_sq); }else {/* * we didn't setup the controller yet in case @@ -1848,6 +1848,12 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,return ret; } +static void nvmet_rdma_disconnect_queue (struct nvmet_sq *sq) +{ +nvmet_rdma_queue_disconnect (container_of(sq ,struct nvmet_rdma_queue , +nvme_sq )); +} +static void nvmet_rdma_delete_ctrl (struct nvmet_ctrl *ctrl) {struct nvmet_rdma_queue *queue, *n;@@ -2078,6 +2084,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = { .remove_port = nvmet_rdma_remove_port, .queue_response = nvmet_rdma_queue_response, .delete_ctrl = nvmet_rdma_delete_ctrl, + .disconnect_queue = nvmet_rdma_disconnect_queue, .disc_traddr = nvmet_rdma_disc_port_addr, .host_traddr = nvmet_rdma_host_port_addr, .get_mdts = nvmet_rdma_get_mdts,drivers/nvme/target/tcp.c +9
-1@@ -412,7 +412,7 @@ static void nvmet_tcp_socket_error(struct nvmet_tcp_queue *queue, int status)if (status == -EPIPE || status == -ECONNRESET || !queue->nvme_sq.ctrl)kernel_sock_shutdown (queue->sock,SHUT_RDWR );else -nvmet_ctrl_fatal_error (queue->nvme_sq.ctrl); +nvmet_sq_fatal_error (&queue->nvme_sq); }static int nvmet_tcp_map_data (struct nvmet_tcp_cmd *cmd)@@ -2223,6 +2223,14 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)kfree (port); } +static void nvmet_tcp_disconnect_queue (struct nvmet_sq *sq) +{ +struct nvmet_tcp_queue *queue = + container_of(sq, struct nvmet_tcp_queue, nvme_sq); + +kernel_sock_shutdown (queue->sock,SHUT_RDWR ); +} +static void nvmet_tcp_delete_ctrl (struct nvmet_ctrl *ctrl) {struct nvmet_tcp_queue *queue;
drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops = { · line 2314
Quoted 7 lines
drivers/nvme/target/tcp.c +1
-0@@ -2300,6 +2308,7 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = { .remove_port = nvmet_tcp_remove_port, .queue_response = nvmet_tcp_queue_response, .delete_ctrl = nvmet_tcp_delete_ctrl, + .disconnect_queue = nvmet_tcp_disconnect_queue, .install_queue = nvmet_tcp_install_queue, .disc_traddr = nvmet_tcp_disc_port_addr, .host_traddr = nvmet_tcp_host_port_addr,
No reason why we shouldn't to this for FC.
I'll update the patch.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 70/98 nvmet-tcp: don't wait for the host in the accept work
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
On a port that offers TLS but doesn't require it, nvmet_tcp_alloc_queue() peeks at the first bytes of a new connection to tell an ICReq from a TLS handshake. That receive blocks until the host sends something or closes, and it runs in the accept work, of which there is one per port. A connection that stays silent blocks every other connect on that port for as long as it's held open, no secret needed. And nvmet_tcp_remove_port() waits for the accept work with nvmet_config_sem held for writing, so an admin taking the port down hangs there and no port of the target answers a Connect anymore.
Install the socket callbacks first and peek without blocking from the io work, which data_ready() reruns. If it's a handshake, restore the default callbacks for the upcall as before, and make the upcall from one more run of the io work so no run is left that could steal data from the handshake. The state change happens under state_lock and only if the queue isn't already going away. The io work doesn't hold the port like the accept work did, so note the keyring serial, which is all the upcall needs from the port, at allocation.
A partially arrived header used to be taken for "no TLS", now the rest is waited for. A silent connection still holds its queue, as on a port without TLS.
--- drivers/nvme/target/tcp.c | 67 ++++++++++++++++++++++++++++++++++----- 1 file changed, 59 insertions(+), 8 deletions(-)
drivers/nvme/target/tcp.c +59 -8
@@ -140,6 +140,7 @@ struct nvmet_tcp_cmd {enum nvmet_tcp_queue_state {NVMET_TCP_Q_CONNECTING , +NVMET_TCP_Q_TLS_PEEK ,NVMET_TCP_Q_TLS_HANDSHAKE ,NVMET_TCP_Q_LIVE ,NVMET_TCP_Q_DISCONNECTING ,
@@ -176,6 +177,7 @@ struct nvmet_tcp_queue {/* TLS state */ key_serial_t tls_pskid; +key_serial_t tls_keyring;struct delayed_work tls_handshake_tmo_work;unsigned long poll_end;
@@ -1496,6 +1498,8 @@ static bool nvmet_tcp_check_queue_deadline(struct nvmet_tcp_queue *queue,return !time_after (jiffies, queue->poll_end); } +static bool nvmet_tcp_tls_peek (struct nvmet_tcp_queue *queue); +static void nvmet_tcp_io_work (struct work_struct *w) { struct nvmet_tcp_queue *queue =
@@ -1503,6 +1507,11 @@ static void nvmet_tcp_io_work(struct work_struct *w)bool pending;int ret, ops =0 ; +if ((queue->state ==NVMET_TCP_Q_TLS_PEEK || + queue->state ==NVMET_TCP_Q_TLS_HANDSHAKE ) && + !nvmet_tcp_tls_peek (queue)) +return ; + do { pending = false;
@@ -1828,12 +1837,9 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)struct msghdr msg = { .msg_control = cbuf, .msg_controllen =sizeof (cbuf), - .msg_flags =MSG_PEEK , + .msg_flags =MSG_PEEK |MSG_DONTWAIT , }; -if (nvmet_port_secure_channel_required (queue->port->nport)) -return 0 ; - len =kernel_recvmsg (queue->sock, &msg, &iov,1 , iov.iov_len, msg.msg_flags);if (unlikely (len <0 )) {
@@ -1955,7 +1961,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) args.ta_sock = queue->sock; args.ta_done = nvmet_tcp_tls_handshake_done; args.ta_data = queue; - args.ta_keyring =key_serial (queue->port->nport->keyring); + args.ta_keyring = queue->tls_keyring; args.ta_timeout_ms = tls_handshake_timeout *1000 ; ret =tls_server_hello_psk (&args,GFP_KERNEL );
@@ -1968,8 +1974,51 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) }return ret; } + +/* +* TLS is offered, but not required: the first bytes tell what the host wants. +* Returns true if the queue goes on without TLS. +*/ +static bool nvmet_tcp_tls_peek (struct nvmet_tcp_queue *queue) +{ +int ret; + +if (queue->state ==NVMET_TCP_Q_TLS_HANDSHAKE ) { +if (nvmet_tcp_tls_handshake (queue)) +nvmet_tcp_schedule_release_queue (queue); +return false; + } + +/* ->data_ready() brings us back */ + ret =nvmet_tcp_try_peek_pdu (queue); +if (ret == -EAGAIN ) +return false; + +/* The upcall wants the default callbacks */ +if (!ret) +nvmet_tcp_restore_socket_callbacks (queue); + +spin_lock_bh (&queue->state_lock); +if (queue->state !=NVMET_TCP_Q_TLS_PEEK ) { +spin_unlock_bh (&queue->state_lock); +return false; + } + queue->state = ret ?NVMET_TCP_Q_CONNECTING :NVMET_TCP_Q_TLS_HANDSHAKE ; +spin_unlock_bh (&queue->state_lock); +if (ret) +return true; + +/* +* ->data_ready() may have queued the io work meanwhile, nothing does +* from now on. The handshake must not start before that run is over, +* so leave it to that run. +*/ +queue_work_on (queue_cpu (queue), nvmet_tcp_wq, &queue->io_work); +return false; +}#else static void nvmet_tcp_tls_handshake_timeout (struct work_struct *w) {} +static bool nvmet_tcp_tls_peek (struct nvmet_tcp_queue *queue) {return true; }#endif static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
@@ -2039,14 +2088,16 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, sk->sk_user_data =NULL ; sk->sk_data_ready = port->data_ready;write_unlock_bh (&sk->sk_callback_lock); -if (!nvmet_tcp_try_peek_pdu (queue)) { +/* The port may be gone when the upcall is made */ + queue->tls_keyring =key_serial (port->nport->keyring); +if (nvmet_port_secure_channel_required (port->nport)) {if (!nvmet_tcp_tls_handshake (queue))return ;/* TLS handshake failed, terminate the connection */ goto out_destroy_sq; } -/* Not a TLS connection, continue with normal processing */ - queue->state = NVMET_TCP_Q_CONNECTING; +/* Don't wait for the host here: others want to connect, too */ + queue->state = NVMET_TCP_Q_TLS_PEEK; }#endif
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 71/98 nvmet: check the host NQN against the TLS PSK identity
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The identity of a TLS PSK names the host and subsystem the key was generated for. The target remembers which key a handshake ended with, but never compares it with the host NQN in the Connect that follows. So a host holding any PSK in the port's keyring can connect as another host, sees what that host may see, and isn't asked for DH-HMAC-CHAP since TLS queues skip it. It can add an I/O queue to that host's TLS controller the same way.
Refuse a Connect whose host NQN isn't the one in the PSK identity of the queue. The subsystem NQN isn't compared, a discovery subsystem goes by two names.
--- drivers/nvme/target/fabrics-cmd.c | 33 +++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
drivers/nvme/target/fabrics-cmd.c +33 -0
@@ -264,6 +264,30 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq) (needs_auth ?NVME_CONNECT_AUTHREQ_ATR :0 ); } +/* +* The identity of a TLS PSK names the host that the key was made for: +* "NVMe<version><type><hash> <host NQN> <subsystem NQN> ...". +* Whoever has proven to hold the key is that host, and no other one. +*/ +static u16 nvmet_check_tls_host (struct nvmet_sq *sq,const char *hostnqn) +{ +#ifdef CONFIG_NVME_TARGET_TCP_TLS +size_t len =strlen (hostnqn); +const char *id; + +if (!sq->tls_key) +return 0 ; + + id =strchr (sq->tls_key->description,' ' ); +if (!id ||strncmp (id +1 , hostnqn, len) || id[len +1 ] !=' ' ) { +pr_warn ("TLS PSK %08x does not belong to host %s\n" , +key_serial (sq->tls_key), hostnqn); +return NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; + } +#endif +return 0 ; +} +static void nvmet_execute_admin_connect (struct nvmet_req *req) {struct nvmf_connect_command *c = &req->cmd->connect;
@@ -309,6 +333,10 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req) d->subsysnqn[NVMF_NQN_FIELD_LEN -1 ] ='\0' ; d->hostnqn[NVMF_NQN_FIELD_LEN -1 ] ='\0' ; + args.status =nvmet_check_tls_host (req->sq, d->hostnqn); +if (args.status) + goto out; + args.subsysnqn = d->subsysnqn; args.hostnqn = d->hostnqn; args.hostid = &d->hostid;
@@ -363,6 +391,11 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) d->subsysnqn[NVMF_NQN_FIELD_LEN -1 ] ='\0' ; d->hostnqn[NVMF_NQN_FIELD_LEN -1 ] ='\0' ; + + status =nvmet_check_tls_host (req->sq, d->hostnqn); +if (status) + goto out; + ctrl =nvmet_ctrl_find_get (d->subsysnqn, d->hostnqn,le16_to_cpu (d->cntlid), req);if (!ctrl) {
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 72/98 nvmet: fix NULL dereference of PCI endpoint targets
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_pci_epf_create_ctrl() calls nvmet_alloc_ctrl() without a queue, the admin queue comes later from the host. nvmet_alloc_ctrl() passes args->sq to nvmet_queue_tls_keyid(), which with CONFIG_NVME_TARGET_TCP_TLS dereferences it, so every PCI endpoint controller creation crashes on such a kernel. The debugfs tls_key file passes ctrl->sqs[0] to the same function, which is NULL until the admin queue exists and again once a fabrics host has closed it while keeping an I/O queue.
A queue that isn't there has no TLS key.
--- drivers/nvme/target/nvmet.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/nvmet.h +2 -1
@@ -964,7 +964,8 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)#ifdef CONFIG_NVME_TARGET_TCP_TLS static inline key_serial_t nvmet_queue_tls_keyid (struct nvmet_sq *sq) { -return sq->tls_key ?key_serial (sq->tls_key) :0 ; +/* A PCI endpoint controller is created without a queue */ +return sq && sq->tls_key ?key_serial (sq->tls_key) :0 ; }static inline void nvmet_sq_put_tls_key (struct nvmet_sq *sq) {
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 73/98 nvmet-tcp: stop listening when a port is removed
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_tcp_accept_work() accepts until there's nothing left. A connection that is already dead when accepted is torn down right there, which takes a few ms, and more can arrive meanwhile. nvmet_tcp_remove_port() waits for the work with cancel_work_sync(), so someone opening and closing connections in a loop keeps the removal from finishing. It holds nvmet_config_sem for writing, so nothing can be configured and no port answers a Connect until the flood stops. No secret needed, just an admin removing the port at the wrong time.
Shut the listening socket down first. Pending connections are dropped, as they were on release, and the next kernel_accept() fails and ends the work.
--- drivers/nvme/target/tcp.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/target/tcp.c +2 -0
@@ -2263,6 +2263,8 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport) port->sock->sk->sk_data_ready = port->data_ready; port->sock->sk->sk_user_data =NULL ;write_unlock_bh (&port->sock->sk->sk_callback_lock); +/* The accept work goes on for as long as there is something to accept */ +kernel_sock_shutdown (port->sock,SHUT_RDWR );cancel_work_sync (&port->accept_work);/* * Destroy the remaining queues, which are not belong to any
Signature · 1 line
2.56.0
Re: [PATCH 73/98] nvmet-tcp: stop listening when a port is removed
Quoted 30 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_tcp_accept_work()accepts until there's nothing left. A connection that is already dead when accepted is torn down right there, which takes a few ms, and more can arrive meanwhile.nvmet_tcp_remove_port()waits for the work withcancel_work_sync(), so someone opening and closing connections in a loop keeps the removal from finishing. It holdsnvmet_config_semfor writing, so nothing can be configured and no port answers a Connect until the flood stops. No secret needed, just an admin removing the port at the wrong time.Shut the listening socket down first. Pending connections are dropped, as they were on release, and the next
kernel_accept()fails and ends the work.Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/tcp.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/target/tcp.c +0
-0
drivers/nvme/target/tcp.c static void nvmet_tcp_remove_port(struct nvmet_port *nport) · line 2270
Quoted 8 lines
drivers/nvme/target/tcp.c +2
-0@@ -2263,6 +2263,8 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport) port->sock->sk->sk_data_ready = port->data_ready; port->sock->sk->sk_user_data =NULL ;write_unlock_bh (&port->sock->sk->sk_callback_lock); +/* The accept work goes on for as long as there is something to accept */ +kernel_sock_shutdown (port->sock,SHUT_RDWR );cancel_work_sync (&port->accept_work);/* * Destroy the remaining queues, which are not belong to any
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 74/98 nvme-tcp: keep the page frag cache with the controller
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvme_tcp_init_request() allocates the PDU from the queue's page frag cache under pf_cache_lock, and it's called not just for the tag set but for the flush request of every hctx of a new namespace. The cache and the lock are torn down and reinitialized with the connection though, and a namespace scan isn't stopped by error recovery, so it allocates from a cache that's being drained or set up:
WARNING: kernel/locking/mutex.c:625 at __mutex_lock+0xa25/0x1a30 nvme_tcp_init_request+0x8c/0x1a0 blk_mq_alloc_and_init_hctx+0x28e/0x6c0
Without lock debugging a waiter can spin forever on a mutex that's being reinitialized, an allocation beside the drain can return a fragment of a freed page, and what's allocated after the drain leaks if the controller goes away. The target, or anyone who can break the connection, picks the moment.
The PDUs outlive the connection anyway. Initialize the lock with the controller and drain the caches when it's freed.
--- drivers/nvme/host/tcp.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)
drivers/nvme/host/tcp.c +13 -7
@@ -162,6 +162,7 @@ struct nvme_tcp_queue_setup_ctx {struct nvme_tcp_ctrl {/* read only in the hot path */ struct nvme_tcp_queue *queues; +int nr_queues;/* allocated, see queue_count */ struct blk_mq_tag_set tag_set;/* other member variables */
@@ -1668,8 +1669,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)if (!test_and_clear_bit (NVME_TCP_Q_ALLOCATED , &queue->flags))return ; -page_frag_cache_drain (&queue->pf_cache); -/** * Prevent memory reclaim from triggering block I/O during socket * teardown. The socket release path fput -> tcp_close ->
@@ -1692,7 +1691,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)kfree (queue->pdu);mutex_destroy (&queue->send_mutex); -mutex_destroy (&queue->pf_cache_lock); }static int nvme_tcp_init_connection (struct nvme_tcp_queue *queue)
@@ -2021,7 +2019,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,INIT_LIST_HEAD (&queue->send_list);mutex_init (&queue->send_mutex);INIT_WORK (&queue->io_work, nvme_tcp_io_work); -mutex_init (&queue->pf_cache_lock);if (qid >0 ) queue->cmnd_capsule_len = nctrl->ioccsz *16 ;
@@ -2153,7 +2150,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, queue->sock =NULL ; err_destroy_mutex:mutex_destroy (&queue->send_mutex); -mutex_destroy (&queue->pf_cache_lock);return ret; }
@@ -2840,6 +2836,10 @@ static void nvme_tcp_stop_ctrl(struct nvme_ctrl *ctrl)static void nvme_tcp_free_ctrl (struct nvme_ctrl *nctrl) {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl); +int i; + +for (i =0 ; i < ctrl->nr_queues; i++) +page_frag_cache_drain (&ctrl->queues[i].pf_cache);if (list_empty_careful (&ctrl->list)) goto free_ctrl;
@@ -3265,9 +3265,15 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, ret = -ENOMEM ; goto out_free_ctrl; } -/* A poller may try the lock of a queue that has been freed */ -for (i =0 ; i < ctrl->ctrl.queue_count; i++) + ctrl->nr_queues = ctrl->ctrl.queue_count; +/* +* A poller may try the lock of a queue that has been freed, and a +* request of a new namespace may be set up for one. +*/ +for (i =0 ; i < ctrl->nr_queues; i++) {mutex_init (&ctrl->queues[i].queue_lock); +mutex_init (&ctrl->queues[i].pf_cache_lock); + } ctrl->net =get_net (current->nsproxy->net_ns);
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 75/98 nvmet-tcp: hold the queue while the TLS timer is armed
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_tcp_tls_handshake() hands the queue to tls_server_hello_psk() and arms the timeout work afterwards. The reference it takes belongs to the handshake and is dropped by nvmet_tcp_tls_handshake_done(). If the handshake already ended by then, the callback found no timer to cancel, and if it failed because the host closed the connection, the queue may be gone as well. The timer is then armed in freed memory and fires there. Anyone who can reach a TLS port can start a handshake and close the connection, it just takes a quick tlshd.
Hold a second reference until the timer is armed, and arm it under state_lock only if the handshake is still in progress. The callback changes the state under that lock before it cancels the timer.
--- drivers/nvme/target/tcp.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +10 -2
@@ -1955,6 +1955,8 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)return -EINVAL ; } +/* One for the handshake, one to keep the queue until we are through */ +kref_get (&queue->kref);kref_get (&queue->kref);pr_debug ("queue %d: TLS ServerHello\n" , queue->idx);memset (&args,0 ,sizeof (args));
@@ -1969,9 +1971,15 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)kref_put (&queue->kref, nvmet_tcp_release_queue);pr_err ("failed to start TLS, err=%d\n" , ret); }else { -queue_delayed_work (nvmet_wq, &queue->tls_handshake_tmo_work, - tls_handshake_timeout *HZ ); +/* The handshake may be over, then nobody cancels the timer */ +spin_lock_bh (&queue->state_lock); +if (queue->state ==NVMET_TCP_Q_TLS_HANDSHAKE ) +queue_delayed_work (nvmet_wq, + &queue->tls_handshake_tmo_work, + tls_handshake_timeout *HZ ); +spin_unlock_bh (&queue->state_lock); } +kref_put (&queue->kref, nvmet_tcp_release_queue);return ret; }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 76/98 nvmet: fix NULL dereference in debugfs host_traddr
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
The host_traddr callbacks of tcp, rdma and fc take the host address from the transport queue around ctrl->sqs[0], which nvmet_sq_destroy() clears when the admin queue goes away while the controller lives on with its I/O queues. A host that closes the admin connection and keeps an I/O queue leaves such a controller behind, and reading its host_traddr in debugfs dereferences a near-NULL pointer:
BUG: kernel NULL pointer dereference, address: 00000000000002e0 RIP: 0010:nvmet_tcp_host_port_addr+0x19/0x50 [nvmet_tcp]
Don't ask the transport if there's no admin queue. Nothing holds the queue for debugfs, so a reader that's past the check when the queue goes away is still on its own.
--- drivers/nvme/target/core.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/nvme/target/core.c +3 -0
@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, {if (!ctrl->ops->host_traddr)return -EOPNOTSUPP ; +/* They all take the address from the admin queue */ +if (!ctrl->sqs[0 ]) +return -ENOTCONN ;return ctrl->ops->host_traddr (ctrl, traddr, traddr_len); }
Signature · 1 line
2.56.0
Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr
drivers/nvme/target/core.c ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, · line 1856
Quoted 9 lines · Jens Axboe
drivers/nvme/target/core.c +3
-0@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, {if (!ctrl->ops->host_traddr)return -EOPNOTSUPP ; +/* They all take the address from the admin queue */ +if (!ctrl->sqs[0 ]) +return -ENOTCONN ;return ctrl->ops->host_traddr (ctrl, traddr, traddr_len); }
As the commit message notes, nothing synchronizes this check with nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(), and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and dereference the container transport queue, leaving both a TOCTOU NULL dereference and a use-after-free.
Instead of reaching into ctrl->sqs[0] from debugfs, why not format and save the host transport address once in a char host_traddr[NVMF_TRADDR_SIZE] buffer on struct nvmet_ctrl when the admin queue connects, and have nvmet_ctrl_host_traddr() return that string? That avoids touching the admin queue after setup altogether.
Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr
In reply to Jens AxboeReview ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, · line 1856
Quoted 9 lines · Jens Axboe
(quoted diff) +3
-0@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, { if (!ctrl->ops->host_traddr) return -EOPNOTSUPP; + /* They all take the address from the admin queue */ + if (!ctrl->sqs[0]) + return -ENOTCONN; return ctrl->ops->host_traddr(ctrl, traddr, traddr_len); }
As the commit message notes, nothing synchronizes this check with nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(), and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and dereference the container transport queue, leaving a TOCTOU NULL dereference and use-after-free.
Since args->sq is already passed to nvmet_alloc_ctrl() when the admin queue creates the controller, we can pass struct nvmet_sq * directly to ops->host_traddr, format the address once into ctrl->host_traddr before nvmet_debugfs_ctrl_setup(ctrl), and have nvmet_ctrl_host_traddr() copy out ctrl->host_traddr. That avoids touching ctrl->sqs[0] or the admin transport queue after controller allocation altogether, and also keeps debugfs host_traddr working even if the admin connection has closed.
Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
drivers/nvme/target/core.c +7 -1
@@ -1729,6 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) goto init_pr_fail;list_add_tail (&ctrl->subsys_entry, &subsys->ctrls);nvmet_setup_p2p_ns_map (ctrl, args->p2p_client); +if (ctrl->ops->host_traddr && args->sq && + ctrl->ops->host_traddr (args->sq, ctrl->host_traddr, +sizeof (ctrl->host_traddr)) <0 ) + ctrl->host_traddr[0 ] ='\0' ;nvmet_debugfs_ctrl_setup (ctrl);mutex_unlock (&subsys->lock);up_read (&nvmet_config_sem);
@@ -1849,7 +1853,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl, {if (!ctrl->ops->host_traddr)return -EOPNOTSUPP ; -return ctrl->ops->host_traddr (ctrl, traddr, traddr_len); +if (!ctrl->host_traddr[0 ]) +return -EINVAL ; +return strscpy (traddr, ctrl->host_traddr, traddr_len); } static struct nvmet_subsys *nvmet_find_get_subsys(struct nvmet_port *port,
drivers/nvme/target/fc.c +1 -2
@@ -2960,10 +2960,9 @@ nvmet_fc_discovery_chg(struct nvmet_port *port) }static ssize_t -nvmet_fc_host_traddr (struct nvmet_ctrl *ctrl, +nvmet_fc_host_traddr (struct nvmet_sq *sq,char *traddr,size_t traddr_size) { -struct nvmet_sq *sq = ctrl->sqs[0 ];struct nvmet_fc_tgt_queue *queue = container_of(sq, struct nvmet_fc_tgt_queue, nvme_sq);struct nvmet_fc_tgtport *tgtport = queue->assoc ? queue->assoc->tgtport :NULL ;
drivers/nvme/target/nvmet.h +2 -1
@@ -297,6 +297,7 @@ struct nvmet_ctrl {u32 nr_changed_ns;char hostnqn[NVMF_NQN_FIELD_LEN ]; +char host_traddr[NVMF_TRADDR_SIZE ];struct device *p2p_client;struct radix_tree_root p2p_ns_map;
@@ -444,7 +445,7 @@ struct nvmet_fabrics_ops {void (*disconnect_queue)(struct nvmet_sq *sq);void (*disc_traddr)(struct nvmet_req *req,struct nvmet_port *port,char *traddr); -ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl, +ssize_t (*host_traddr)(struct nvmet_sq *sq,char *traddr,size_t traddr_len);u16 (*install_queue)(struct nvmet_sq *nvme_sq);void (*discovery_chg)(struct nvmet_port *port);
drivers/nvme/target/rdma.c +1 -2
@@ -2050,10 +2050,9 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req, } } -static ssize_t nvmet_rdma_host_port_addr (struct nvmet_ctrl *ctrl, +static ssize_t nvmet_rdma_host_port_addr (struct nvmet_sq *nvme_sq,char *traddr,size_t traddr_len) { -struct nvmet_sq *nvme_sq = ctrl->sqs[0 ];struct nvmet_rdma_queue *queue = container_of(nvme_sq, struct nvmet_rdma_queue, nvme_sq);
drivers/nvme/target/tcp.c +1 -2
@@ -2348,10 +2348,9 @@ static void nvmet_tcp_disc_port_addr(struct nvmet_req *req, } } -static ssize_t nvmet_tcp_host_port_addr (struct nvmet_ctrl *ctrl, +static ssize_t nvmet_tcp_host_port_addr (struct nvmet_sq *sq,char *traddr,size_t traddr_len) { -struct nvmet_sq *sq = ctrl->sqs[0 ];struct nvmet_tcp_queue *queue = container_of(sq, struct nvmet_tcp_queue, nvme_sq);
Jens Axboe
maintainerPATCH 77/98 nvmet-tcp: let the accept work come to an end
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_tcp_accept_work() loops until there's nothing left to accept, and anyone opening and closing connections fast enough keeps it running indefinitely, since tearing down a dead connection takes long enough for more to arrive. Whoever flushes nvmet_wq waits for it: removing any other port of the target does in nvmet_port_release(), and so does unloading the rdma and fc modules.
Accept a bounded number of connections per run and requeue the work. A flush doesn't wait for work queued behind it.
--- drivers/nvme/target/tcp.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +6 -2
@@ -85,6 +85,7 @@ MODULE_PARM_DESC(tls_handshake_timeout,#define NVMET_TCP_RECV_BUDGET 8#define NVMET_TCP_SEND_BUDGET 8#define NVMET_TCP_IO_WORK_BUDGET 64 +#define NVMET_TCP_ACCEPT_BUDGET 16enum nvmet_tcp_send_state {NVMET_TCP_SEND_DATA_PDU ,
@@ -2145,9 +2146,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w)struct nvmet_tcp_port *port = container_of(w, struct nvmet_tcp_port, accept_work);struct socket *newsock; -int ret; +int ret, i; -while (true) { +for (i =0 ; i <NVMET_TCP_ACCEPT_BUDGET ; i++) { ret =kernel_accept (port->sock, &newsock,O_NONBLOCK );if (ret <0 ) {if (ret != -EAGAIN )
@@ -2156,6 +2157,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w) }nvmet_tcp_alloc_queue (port, newsock); } + +/* There may be more. Come to an end for those who flush nvmet_wq. */ +queue_work (nvmet_wq, &port->accept_work); }static void nvmet_tcp_listen_data_ready (struct sock *sk)
Signature · 1 line
2.56.0
Re: [PATCH 77/98] nvmet-tcp: let the accept work come to an end
Quoted 18 lines · Jens Axboe
From: Josef Bacik <[email protected]>
nvmet_tcp_accept_work()loops until there's nothing left to accept, and anyone opening and closing connections fast enough keeps it running indefinitely, since tearing down a dead connection takes long enough for more to arrive. Whoever flushesnvmet_wqwaits for it: removing any other port of the target does innvmet_port_release(), and so does unloading the rdma and fc modules.Accept a bounded number of connections per run and requeue the work. A flush doesn't wait for work queued behind it.
Fixes 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 4 lines
--- drivers/nvme/target/tcp.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 78/98 nvmet-tcp: don't read a removed port at Connect time
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
nvmet_tcp_alloc_cmd() takes the nvmet port of a command from queue->port, and the commands of a queue are allocated from the io work when the Connect arrives. nvmet_tcp_remove_port() shuts down the queues that have no controller yet and frees the port without waiting for them, they're released later from a work. An io work handling a Connect at that moment reads the freed port for every command of the queue and hands the result to the core.
Note the nvmet port in the queue at allocation. It outlives the transport port.
--- drivers/nvme/target/tcp.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/tcp.c +4 -1
@@ -151,6 +151,7 @@ enum nvmet_tcp_queue_state {struct nvmet_tcp_queue {struct socket *sock;struct nvmet_tcp_port *port; +struct nvmet_port *nport;struct work_struct io_work;struct nvmet_cq nvme_cq;struct nvmet_sq nvme_sq;
@@ -1544,7 +1545,7 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,u8 hdgst =nvmet_tcp_hdgst_len (queue); c->queue = queue; - c->req.port = queue->port->nport; + c->req.port = queue->nport; c->cmd_pdu =page_frag_alloc (&queue->pf_cache,sizeof (*c->cmd_pdu) + hdgst,GFP_KERNEL | __GFP_ZERO);
@@ -2048,6 +2049,8 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,kref_init (&queue->kref); queue->sock = newsock; queue->port = port; +/* The port may be gone when the commands are allocated */ + queue->nport = port->nport; queue->nr_cmds =0 ;spin_lock_init (&queue->state_lock); if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 79/98 nvme-tcp: start error recovery when a send fails
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
When the target closes a connection, nvme_tcp_state_change() calls nvme_tcp_error_recovery(), which does nothing unless the controller is live. While connecting that's fine for the queue being set up, its Connect fails and the attempt with it. But if an already connected queue is closed at the end of the attempt, nothing fails, the notification is lost and the controller goes live on a dead socket. Every request on that queue then fails with "failed to send request -32" and is retried, forever with native multipathing, and if it's the admin queue the first keep-alive fails too. Nothing but a manual reset recovers it. The Linux target closes its connections at exactly that moment after a fatal error.
Start error recovery when a request can't be sent, as nvme-rdma does.
--- drivers/nvme/host/tcp.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +2 -0
@@ -1461,6 +1461,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) "failed to send request %d\n", ret);nvme_tcp_fail_request (queue->request);nvme_tcp_done_send_req (queue); +/* The close may have come when nobody could act on it */ +nvme_tcp_error_recovery (&queue->ctrl->ctrl); } out:memalloc_noreclaim_restore (noreclaim_flag);
Signature · 1 line
2.56.0
Re: [PATCH 79/98] nvme-tcp: start error recovery when a send fails
Quoted 31 lines · Jens Axboe
From: Josef Bacik <[email protected]>
When the target closes a connection,
nvme_tcp_state_change()callsnvme_tcp_error_recovery(), which does nothing unless the controller is live. While connecting that's fine for the queue being set up, its Connect fails and the attempt with it. But if an already connected queue is closed at the end of the attempt, nothing fails, the notification is lost and the controller goes live on a dead socket. Every request on that queue then fails with "failed to send request -32" and is retried, forever with native multipathing, and if it's the admin queue the first keep-alive fails too. Nothing but a manual reset recovers it. The Linux target closes its connections at exactly that moment after a fatal error.Start error recovery when a request can't be sent, as nvme-rdma does.
Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")Cc Keith Busch <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/host/tcp.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1468
Quoted 8 lines
drivers/nvme/host/tcp.c +2
-0@@ -1461,6 +1461,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) "failed to send request %d\n", ret);nvme_tcp_fail_request (queue->request);nvme_tcp_done_send_req (queue); +/* The close may have come when nobody could act on it */ +nvme_tcp_error_recovery (&queue->ctrl->ctrl); } out:memalloc_noreclaim_restore (noreclaim_flag);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 80/98 nvme-loop: don't BUG() if a scan runs beside a reset
In reply to Jens AxboeFrom: Josef Bacik <[email protected]>
A reset sets queue_count to 1 while it destroys the I/O queues and counts it back up when the new ones are made, and it doesn't stop the scan work. A new namespace found in that window trips the check in nvme_loop_init_hctx() for every hctx but the first:
kernel BUG at drivers/nvme/target/loop.c:224!
nvme_loop_init_hctx+0x7d/0x80 __blk_mq_realloc_hw_ctxs+0x3af/0xa80 nvme_alloc_ns+0x1a2/0x22f0
The check protects nothing. ctrl->queues has nr_io_queues + 1 entries for the life of the controller, the tag set never has more hardware queues than that, and nvme_loop_queue_rq() rejects requests until the controller is live again.
Remove it.
--- drivers/nvme/target/loop.c | 2 -- 1 file changed, 2 deletions(-)
drivers/nvme/target/loop.c +0 -2
@@ -221,8 +221,6 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,struct nvme_loop_ctrl *ctrl =to_loop_ctrl (data);struct nvme_loop_queue *queue = &ctrl->queues[hctx_idx +1 ]; -BUG_ON (hctx_idx >= ctrl->ctrl.queue_count); -/* * flush_end_io() can be called recursively for us, so use our own * lock class key for avoiding lockdep possible recursive locking,
Signature · 1 line
2.56.0
Re: [PATCH 80/98] nvme-loop: don't BUG() if a scan runs beside a reset
Quoted 34 lines · Jens Axboe
From: Josef Bacik <[email protected]>
A reset sets
queue_countto 1 while it destroys the I/O queues and counts it back up when the new ones are made, and it doesn't stop the scan work. A new namespace found in that window trips the check innvme_loop_init_hctx()for every hctx but the first:kernel BUG at drivers/nvme/target/loop.c:224! nvme_loop_init_hctx+0x7d/0x80 __blk_mq_realloc_hw_ctxs+0x3af/0xa80 nvme_alloc_ns+0x1a2/0x22f0The check protects nothing.
ctrl->queueshasnr_io_queues+ 1 entries for the life of the controller, the tag set never has more hardware queues than that, andnvme_loop_queue_rq()rejects requests until the controller is live again.Remove it.
Fixes 3a85a5de29ea ("nvme-loop: add a NVMe loopback host driver")Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Josef Bacik <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/loop.c | 2 -- 1 file changed, 2 deletions(-)drivers/nvme/target/loop.c +0
-0
drivers/nvme/target/loop.c static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, · line 226
Quoted 8 lines
drivers/nvme/target/loop.c +0
-2@@ -221,8 +221,6 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,struct nvme_loop_ctrl *ctrl =to_loop_ctrl (data);struct nvme_loop_queue *queue = &ctrl->queues[hctx_idx +1 ]; -BUG_ON (hctx_idx >= ctrl->ctrl.queue_count); -/* * flush_end_io() can be called recursively for us, so use our own * lock class key for avoiding lockdep possible recursive locking,
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 81/98 nvmet: fix pc_ref use-after-free in preempt and abort
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
With reservations enabled on a namespace, the target keeps a pc_ref per controller, a percpu_ref counting that controller's I/O on the namespace. "Preempt and abort" kills the pc_refs of the preempted controllers and waits for the kill to be confirmed in nvmet_pr_set_ctrl_to_abort(), then waits for the count to drop to zero and resurrects them in nvmet_pr_do_abort(). Neither holds a reference to the pc_ref, only pr_sem. nvmet_ctrl_destroy_pr() frees the pc_refs under subsys->lock when the controller goes away, so a host that registers a key, disconnects, and sends preempt and abort with that key over a second connection frees a pc_ref that the abort is sleeping on:
preempt and abort on C1 last put of C2
nvmet_pr_set_ctrl_to_abort()
percpu_ref_kill_and_confirm() wait_for_completion()
nvmet_ctrl_destroy_pr()
percpu_ref_exit() kfree(pc_ref)
percpu_ref_switch_to_atomic_rcu() /* use after free */
Put the pc_refs that an abort has killed on a per-namespace list. nvmet_ctrl_destroy_pr() only flags a pc_ref on that list, and nvmet_pr_do_abort() frees flagged ones once it's done with them. The list and the flag are written under the xarray lock.
--- drivers/nvme/target/nvmet.h | 5 ++ drivers/nvme/target/pr.c | 92 ++++++++++++++++++++++++++----------- 2 files changed, 69 insertions(+), 28 deletions(-)
drivers/nvme/target/nvmet.h +5 -0
@@ -88,6 +88,8 @@ struct nvmet_pr {*/ struct semaphore pr_sem;struct list_head registrant_list; +/* The pc_refs that a running "preempt and abort" has killed. */ +struct list_head abort_list; };struct nvmet_pr_per_ctrl_ref {
@@ -95,6 +97,9 @@ struct nvmet_pr_per_ctrl_ref {struct completion free_done;struct completion confirm_done;uuid_t hostid; +/* Both are written under the lock of ns->pr_per_ctrl_refs. */ +struct list_head abort_entry; +bool ctrl_gone; };struct nvmet_ns {
drivers/nvme/target/pr.c +64 -28
152 lines not on this page · raw message
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 82/98 nvmet: move preempt and abort work off nvmet-wq
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_pr_do_abort() runs on nvmet-wq, holds pr_sem while it waits for the preempted controllers' I/O, and releases it at the end. nvmet-wq also runs nvmet_tcp_release_queue_work(), which cancel_work_sync()s the io_work of a closed connection, and that io_work may be sleeping on pr_sem in a reservation command. So a host that keeps a write in flight on a preempted controller, sends a second preempt and abort, and then closes enough connections with a reservation command in flight fills nvmet-wq with release works that wait for pr_sem. When the first abort finishes, the second one is queued on a full workqueue behind them and never starts, pr_sem is never released, and the accept work doesn't run either, so nobody can connect.
Queue the abort work on its own workqueue so it starts regardless. The release works can still fill nvmet-wq while pr_sem is held, and a file-backed namespace's flush, DSM and write zeroes need nvmet-wq themselves, so this is a mitigation.
--- drivers/nvme/target/core.c | 11 ++++++++++- drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/pr.c | 6 +++++- 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/nvme/target/core.c +10 -1
@@ -29,6 +29,7 @@ struct workqueue_struct *nvmet_wq;EXPORT_SYMBOL_GPL (nvmet_wq);struct workqueue_struct *nvmet_aen_wq;EXPORT_SYMBOL_GPL (nvmet_aen_wq); +struct workqueue_struct *nvmet_pr_wq;/* * This read/write semaphore is used to synchronize access to configuration
@@ -2034,9 +2035,14 @@ static int __init nvmet_init(void)if (!nvmet_aen_wq) goto out_free_nvmet_work_queue; + nvmet_pr_wq =alloc_workqueue ("nvmet-pr-wq" , +WQ_MEM_RECLAIM |WQ_UNBOUND ,0 ); +if (!nvmet_pr_wq) + goto out_free_nvmet_aen_work_queue; + error =nvmet_init_debugfs ();if (error) - goto out_free_nvmet_aen_work_queue; + goto out_free_nvmet_pr_work_queue; error =nvmet_init_discovery ();if (error)
@@ -2052,6 +2058,8 @@ static int __init nvmet_init(void)nvmet_exit_discovery (); out_exit_debugfs:nvmet_exit_debugfs (); +out_free_nvmet_pr_work_queue: +destroy_workqueue (nvmet_pr_wq); out_free_nvmet_aen_work_queue:destroy_workqueue (nvmet_aen_wq); out_free_nvmet_work_queue:
@@ -2071,6 +2079,7 @@ static void __exit nvmet_exit(void)nvmet_exit_discovery ();nvmet_exit_debugfs ();ida_destroy (&cntlid_ida); +destroy_workqueue (nvmet_pr_wq);destroy_workqueue (nvmet_aen_wq);destroy_workqueue (nvmet_wq);destroy_workqueue (buffered_io_wq);
drivers/nvme/target/nvmet.h +1 -0
@@ -532,6 +532,7 @@ extern struct workqueue_struct *buffered_io_wq;extern struct workqueue_struct *zbd_wq;extern struct workqueue_struct *nvmet_wq;extern struct workqueue_struct *nvmet_aen_wq; +extern struct workqueue_struct *nvmet_pr_wq;static inline void nvmet_set_result (struct nvmet_req *req,u32 result) {
drivers/nvme/target/pr.c +5 -1
@@ -756,7 +756,11 @@ static void nvmet_execute_pr_acquire(struct nvmet_req *req)if (!status && acquire_act ==NVME_PR_ACQUIRE_ACT_PREEMPT_AND_ABORT ) {kfree (d);INIT_WORK (&req->r.abort_work, nvmet_pr_do_abort); -queue_work (nvmet_wq, &req->r.abort_work); +/* +* Not nvmet_wq: works on it can wait for pr_sem, which this +* work releases. +*/ +queue_work (nvmet_pr_wq, &req->r.abort_work);return ; }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 83/98 nvmet-rdma: treat a failed connect as a disconnect
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
Once a connect request has been accepted, a queue is only freed by its release_work, so that must be queued once. The disconnect paths take the queue off nvmet_rdma_queue_list first and queue the work only if it was still on it. nvmet_rdma_queue_connect_fail(), for REJECTED, UNREACHABLE and CONNECT_ERROR, queues it unconditionally and only warns if the queue isn't connecting. But the IB CM delivers a REJ from the peer while the connection is established and even after we've sent a DREQ, so a peer that causes a disconnect with a short capsule and then sends a REJ while the release work is running queues it a second time and the work runs on a freed queue. No credentials needed, just RoCE or IB reachability, reproduced over rxe with a UDP socket.
Route connect failures through nvmet_rdma_queue_disconnect(), so only the path that takes the queue off the list queues the work. A REJ on an established connection still frees the queue, the RDMA CM reports no disconnect after one.
--- drivers/nvme/target/rdma.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
drivers/nvme/target/rdma.c +7 -8
@@ -1738,15 +1738,14 @@ static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)static void nvmet_rdma_queue_connect_fail (struct rdma_cm_id *cm_id,struct nvmet_rdma_queue *queue) { -WARN_ON_ONCE (queue->state !=NVMET_RDMA_Q_CONNECTING ); - -mutex_lock (&nvmet_rdma_queue_mutex); -if (!list_empty (&queue->queue_list)) -list_del_init (&queue->queue_list); -mutex_unlock (&nvmet_rdma_queue_mutex); -pr_err ("failed to connect queue %d\n" , queue->idx); -queue_work (nvmet_wq, &queue->release_work); + +/* +* The peer decides when these events arrive: a REJ is also delivered +* for a connection that is established, or that we are disconnecting. +* Whoever takes the queue off the list queues release_work, once. +*/ +nvmet_rdma_queue_disconnect (queue); }/**
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 84/98 nvmet-rdma: disconnect only established queues
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
Over iWARP, rdma_disconnect() sleeps in iw_cm_disconnect() until the ESTABLISHED event of the connection has been handled, and iw_cm handles all events of all connections one at a time on iw_cm_wq. nvmet_rdma_destroy_port_queues(), nvmet_rdma_remove_one() and nvmet_rdma_delete_ctrl() walk the queue list with nvmet_rdma_queue_mutex held and call rdma_disconnect() for each, while nvmet_rdma_cm_handler() takes the same mutex. If the walk sleeps on a queue whose ESTABLISHED is queued behind another queue's event, both are stuck and iWARP connection handling is dead until reboot. A peer sets that up by connecting in a loop, or over siw by asking for MPA peer-to-peer mode and then staying silent, which delays ESTABLISHED indefinitely.
Only call rdma_disconnect() for a queue in NVMET_RDMA_Q_LIVE, which the handler sets when it gets ESTABLISHED. A queue that never got there is left to the release work: ib_drain_qp() moves the QP to the error state, which ends the connection, and rdma_destroy_id() tells the peer. Over IB and RoCE the skipped call did nothing more than that.
--- drivers/nvme/target/rdma.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +13 -2
@@ -1687,7 +1687,7 @@ static void nvmet_rdma_queue_established(struct nvmet_rdma_queue *queue)static void __nvmet_rdma_queue_disconnect (struct nvmet_rdma_queue *queue) { -bool disconnect = false; +bool disconnect = false, live = false;unsigned long flags;pr_debug ("cm_id= %p queue->state= %d\n" , queue->cm_id, queue->state);
@@ -1706,6 +1706,7 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue) } fallthrough;case NVMET_RDMA_Q_LIVE : + live = queue->state ==NVMET_RDMA_Q_LIVE ; queue->state =NVMET_RDMA_Q_DISCONNECTING ; disconnect = true;break ;
@@ -1715,7 +1716,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)spin_unlock_irqrestore (&queue->state_lock, flags);if (disconnect) { -rdma_disconnect (queue->cm_id); +/* +* Over iWARP rdma_disconnect() sleeps until the CM has handled +* the ESTABLISHED event of this connection. The CM handles the +* events of all connections one at a time, and our handler +* takes nvmet_rdma_queue_mutex, which callers of this function +* hold. So a queue that has not seen the event is left to the +* release work: ib_drain_qp() moves the queue pair to the error +* state, which ends the connection. +*/ +if (live) +rdma_disconnect (queue->cm_id);queue_work (nvmet_wq, &queue->release_work); } }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 85/98 nvmet-rdma: take a device off the list last
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_rdma_remove_one() disconnects the queues of a device and flushes nvmet-wq, but since commit a3dd7d0022c3 it returns right away if the device isn't on device_list. nvmet_rdma_free_dev(), run by the release work of the last queue, takes the device off that list first and frees the SRQs and the PD afterwards. A device removal in between finds nothing on the list, returns, and the RDMA core frees the device under ib_dealloc_pd().
Free the SRQs and the PD first and unlist the device last, so the remove callback finds it and its flush waits for the release work.
--- drivers/nvme/target/rdma.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/nvme/target/rdma.c +7 -3
@@ -1190,13 +1190,17 @@ static void nvmet_rdma_free_dev(struct kref *ref)struct nvmet_rdma_device *ndev = container_of(ref, struct nvmet_rdma_device, ref); +nvmet_rdma_destroy_srqs (ndev); +ib_dealloc_pd (ndev->pd); + +/* +* Not earlier: nvmet_rdma_remove_one() waits for the release work that +* got here only if it finds the device on the list. +*/ mutex_lock (&device_list_mutex);list_del (&ndev->entry);mutex_unlock (&device_list_mutex); -nvmet_rdma_destroy_srqs (ndev); -ib_dealloc_pd (ndev->pd); -kfree (ndev); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 86/98 nvmet-rdma: queue the release work under the mutex
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_rdma_remove_one() takes the queue mutex, unlists each queue of the device and queues its release work, then flushes nvmet-wq. So a queue that still holds objects on the device must either be on the list or have its release work queued. nvmet_rdma_queue_disconnect() breaks that: it unlists the queue under the mutex, drops it, and only then calls __nvmet_rdma_queue_disconnect() to queue the work. A device removal in that window finds nothing to wait for, and the release work then destroys the QP and CQ share on a freed device.
Queue the work before dropping the mutex, as nvmet_rdma_remove_one() itself does. That's safe now that rdma_disconnect() no longer sleeps on an iWARP event that needs the same mutex.
--- drivers/nvme/target/rdma.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/nvme/target/rdma.c +6 -6
@@ -1737,17 +1737,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)static void nvmet_rdma_queue_disconnect (struct nvmet_rdma_queue *queue) { -bool disconnect = false; - +/* +* The release work is queued with the mutex held, so that +* nvmet_rdma_remove_one() finds the queue on the list or its work on +* the workqueue. +*/ mutex_lock (&nvmet_rdma_queue_mutex);if (!list_empty (&queue->queue_list)) {list_del_init (&queue->queue_list); - disconnect = true; +__nvmet_rdma_queue_disconnect (queue); }mutex_unlock (&nvmet_rdma_queue_mutex); - -if (disconnect) -__nvmet_rdma_queue_disconnect (queue); } static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 87/98 nvmet-rdma: disconnect a queue on device removal
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
cma_remove_one() sends DEVICE_REMOVAL to the owner of every cm_id on the device and then waits, without timeout, until they're all destroyed. nvmet_rdma_device_removal() does nothing for a queue cm_id and leaves the queues to nvmet_rdma_remove_one(), which only works if that runs first. As a module it does, nvmet-rdma registers after rdma_cm. Built in, both register at the same initcall level and drivers/Makefile lists nvme/ ahead of infiniband/, so cma_remove_one() runs first and waits for cm_ids that only the later callback would destroy. Since the RDMA CM drops all later events for a cm_id once it has sent DEVICE_REMOVAL, a peer closing its connection doesn't help either. One connection from anyone who can reach the port hangs "rdma link delete" until something else ends the queue.
Call nvmet_rdma_queue_disconnect() for the queue. It unlists the queue and queues the release work, which destroys the cm_id. Don't wait for it, rdma_destroy_id() waits for the handler to return, which was the deadlock before commit f1d4ef7d8883.
--- drivers/nvme/target/rdma.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/nvme/target/rdma.c +8 -9
@@ -1773,10 +1773,13 @@ static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,* queue cm_id and/or a device bound listener cm_id (where in this * case queue will be null). * -* We registered an ib_client to handle device removal for queues, -* so we only need to handle the listening port cm_ids. In this case -* we nullify the priv to prevent double cm_id destruction and destroying -* the cm_id implicitly by returning a non-zero rc to the callout. +* The RDMA CM waits until each cm_id of the device is destroyed, so a queue +* has to go now. Our ib_client also removes the queues, but it cannot be +* relied on here: built in, it is called after the RDMA CM's client. +* +* For a listening port cm_id we nullify the priv to prevent double cm_id +* destruction and destroying the cm_id implicitly by returning a non-zero +* rc to the callout. */ static int nvmet_rdma_device_removal (struct rdma_cm_id *cm_id,struct nvmet_rdma_queue *queue)
@@ -1784,11 +1787,7 @@ static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,struct nvmet_rdma_port *port;if (queue) { -/* -* This is a queue cm_id. we have registered -* an ib_client to handle queues removal -* so don't interfere and just return. -*/ +nvmet_rdma_queue_disconnect (queue);return 0 ; }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 88/98 nvmet-rdma: flush nvmet-wq on module exit
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_rdma_exit() relies on ib_unregister_client() to flush nvmet-wq through nvmet_rdma_remove_one(), but since commit a3dd7d0022c3 that returns at once for a device no longer on device_list. The release work of the last queue takes the device off the list in nvmet_rdma_free_dev() and still has to drop the mutex, free the struct and return through nvmet_rdma_release_queue_work(), so an rmmod in that window frees the module code it's running.
Flush nvmet-wq after ib_unregister_client(). No release work can be queued after that point.
--- drivers/nvme/target/rdma.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/nvme/target/rdma.c +6 -0
@@ -2170,6 +2170,12 @@ static void __exit nvmet_rdma_exit(void) {nvmet_unregister_transport (&nvmet_rdma_ops);ib_unregister_client (&nvmet_rdma_ib_client); +/* +* nvmet_rdma_remove_one() does not flush for a device that has left +* device_list, and the release work that took it off still has to +* return from nvmet_rdma_free_dev(). +*/ +flush_workqueue (nvmet_wq);WARN_ON_ONCE (!list_empty (&nvmet_rdma_queue_list));ida_destroy (&nvmet_rdma_queue_ida); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 89/98 nvmet-rdma: reject queues larger than max_cqe
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
The host picks hrqsize and hsqsize in its RDMA CM connect request, and nvmet_rdma_create_queue_ib() asks ib_cq_pool_get() for
recv + 2 * send + 1 CQ entries, up to 196607 for an I/O queue. The pool limits each CQ it allocates to the device's max_cqe, so if the request exceeds that no CQ ever fits and ib_alloc_cqs() keeps adding CQs until an allocation fails. Those CQs stay until the device is removed. rxe has a max_cqe of 32767, and one connect request from an unprivileged process OOMs a 1 GiB target in under a second, with nothing to kill that frees the memory.
Refuse a queue that needs more than max_cqe entries before touching the pool. The connect is rejected with NVME_RDMA_CM_NO_RSC like any other queue creation failure.
--- drivers/nvme/target/rdma.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +11 -2
@@ -1286,8 +1286,17 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)*/ nr_cqe = queue->recv_queue_size +2 * queue->send_queue_size; - queue->cq =ib_cq_pool_get (ndev->device, nr_cqe +1 , - queue->comp_vector,IB_POLL_WORKQUEUE ); +/* +* The host picks both sizes. Never ask ib_cq_pool_get() for more +* entries than one CQ of the device can have: no CQ would ever fit, +* and it would allocate new CQs until an allocation fails. +*/ +if (nr_cqe +1 > ndev->device->attrs.max_cqe) + queue->cq =ERR_PTR (-EINVAL ); +else + queue->cq =ib_cq_pool_get (ndev->device, nr_cqe +1 , + queue->comp_vector, +IB_POLL_WORKQUEUE );if (IS_ERR (queue->cq)) { ret =PTR_ERR (queue->cq); pr_err("failed to create CQ cqe= %d ret= %d\n",
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 90/98 nvmet-rdma: disable interrupts under rsp_wr_wait_lock
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
rsp_wr_wait_lock protects the list of commands waiting for room in the send queue, and it's taken with plain spin_lock(). But nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() directly if ib_post_send() fails, and that processes the wait list. The call chain starts in nvmet_req_complete(), which for a block device namespace runs from the bio end_io callback, which can be in hard interrupt context. An interrupt that completes a command on a CPU holding the lock for the same queue spins forever. Lockdep over siw with a null_blk namespace:
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage
Use spin_lock_irqsave() for both takers. The processing loop still drops the lock, and restores the caller's interrupt state, around nvmet_rdma_execute_command().
--- drivers/nvme/target/rdma.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/nvme/target/rdma.c +9 -6
@@ -513,7 +513,9 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,static void nvmet_rdma_process_wr_wait_list (struct nvmet_rdma_queue *queue) { -spin_lock (&queue->rsp_wr_wait_lock); +unsigned long flags; + +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);while (!list_empty (&queue->rsp_wr_wait_list)) {struct nvmet_rdma_rsp *rsp;bool ret;
@@ -522,16 +524,16 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue) struct nvmet_rdma_rsp, wait_list);list_del (&rsp->wait_list); -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); ret =nvmet_rdma_execute_command (rsp); -spin_lock (&queue->rsp_wr_wait_lock); +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);if (!ret) {list_add (&rsp->wait_list, &queue->rsp_wr_wait_list);break ; } } -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); }static u16 nvmet_rdma_check_pi_status (struct ib_mr *sig_mr)
@@ -976,6 +978,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)static void nvmet_rdma_handle_command (struct nvmet_rdma_queue *queue,struct nvmet_rdma_rsp *cmd) { +unsigned long flags;u16 status; ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +996,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue, goto out_err;if (unlikely (!nvmet_rdma_execute_command (cmd))) { -spin_lock (&queue->rsp_wr_wait_lock); +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);list_add_tail (&cmd->wait_list, &queue->rsp_wr_wait_list); -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); }return ;
Signature · 1 line
2.56.0
Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock
drivers/nvme/target/rdma.c static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue) · line 529
Quoted 5 lines · Jens Axboe
drivers/nvme/target/rdma.c +2
-2-spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); ret =nvmet_rdma_execute_command (rsp); -spin_lock (&queue->rsp_wr_wait_lock); +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);
Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself, but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io callback in hardirq or softirq context (via nvmet_req_complete() -> nvmet_rdma_queue_response() -> failed ib_post_send() -> nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) -> rsp->req.execute(&rsp->req).
The nvmet ->execute() handlers assume process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls submit_bio() -> submit_bio_noacct(), which hits might_sleep() and manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations and take mutexes / rwsems.
Shouldn't rsp_wr_wait_list processing be deferred to a work_struct when !in_task() (or whenever nvmet_rdma_release_rsp() is called from atomic context)?
Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock
In reply to Jens AxboeReview
Quoted 5 lines · Jens Axboe
(quoted diff) +2
-2- spin_unlock(&queue->rsp_wr_wait_lock); + spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags); ret = nvmet_rdma_execute_command(rsp); - spin_lock(&queue->rsp_wr_wait_lock); + spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself, but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io callback in hardirq or softirq context (via nvmet_req_complete() -> nvmet_rdma_queue_response() -> failed ib_post_send() -> nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) -> rsp->req.execute(&rsp->req).
The nvmet ->execute() handlers require process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls submit_bio() -> submit_bio_noacct(), which hits might_sleep() and manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations and take mutexes / rwsems.
If nvmet_rdma_process_wr_wait_list() checks !in_task() under rsp_wr_wait_lock and schedules a per-queue wr_wait_work on system_dfl_wq when rsp_wr_wait_list is non-empty (cancelled after ib_drain_qp() in nvmet_rdma_destroy_queue_ib()), normal task/CQ-workqueue callers pay no extra cost while hardirq/softirq completions safely defer command execution to process context.
Note: patches 91/98 and 92/98 compose cleanly on top by placing the if (done && done->wait_released) check right above if (!in_task()) and passing NULL from nvmet_rdma_wr_wait_work().
Here is a replacement patch (checked with checkpatch.pl and syntax-checked with Clang; not runtime-tested):
---
drivers/nvme/target/rdma.c +26 -6
@@ -104,6 +104,7 @@ struct nvmet_rdma_queue {struct nvmet_rdma_cmd *cmds;struct work_struct release_work; +struct work_struct wr_wait_work;struct list_head rsp_wait_list;struct list_head rsp_wr_wait_list;spinlock_t rsp_wr_wait_lock;
@@ -513,7 +514,15 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,static void nvmet_rdma_process_wr_wait_list (struct nvmet_rdma_queue *queue) { -spin_lock (&queue->rsp_wr_wait_lock); +unsigned long flags; + +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags); +if (!in_task ()) { +if (!list_empty (&queue->rsp_wr_wait_list)) +schedule_work (&queue->wr_wait_work); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); +return ; + }while (!list_empty (&queue->rsp_wr_wait_list)) {struct nvmet_rdma_rsp *rsp;bool ret;
@@ -522,16 +531,24 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue) struct nvmet_rdma_rsp, wait_list);list_del (&rsp->wait_list); -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); ret =nvmet_rdma_execute_command (rsp); -spin_lock (&queue->rsp_wr_wait_lock); +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);if (!ret) {list_add (&rsp->wait_list, &queue->rsp_wr_wait_list);break ; } } -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); +} + +static void nvmet_rdma_wr_wait_work (struct work_struct *w) +{ +struct nvmet_rdma_queue *queue = + container_of(w, struct nvmet_rdma_queue, wr_wait_work); + +nvmet_rdma_process_wr_wait_list (queue); }static u16 nvmet_rdma_check_pi_status (struct ib_mr *sig_mr)
@@ -976,6 +993,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)static void nvmet_rdma_handle_command (struct nvmet_rdma_queue *queue,struct nvmet_rdma_rsp *cmd) { +unsigned long flags;u16 status; ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +1011,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue, goto out_err;if (unlikely (!nvmet_rdma_execute_command (cmd))) { -spin_lock (&queue->rsp_wr_wait_lock); +spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);list_add_tail (&cmd->wait_list, &queue->rsp_wr_wait_list); -spin_unlock (&queue->rsp_wr_wait_lock); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); }return ;
@@ -1374,6 +1392,7 @@ static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr,static void nvmet_rdma_destroy_queue_ib (struct nvmet_rdma_queue *queue) {ib_drain_qp (queue->qp); +cancel_work_sync (&queue->wr_wait_work);/* * Reclaim resources of a response that is still in-flight when the
@@ -1494,6 +1513,7 @@ nvmet_rdma_alloc_queue(struct nvmet_rdma_device *ndev,* inside a CM callback would trigger a deadlock. (great API design..) */ INIT_WORK (&queue->release_work, nvmet_rdma_release_queue_work); +INIT_WORK (&queue->wr_wait_work, nvmet_rdma_wr_wait_work); queue->dev = ndev; queue->cm_id = cm_id; queue->port = port->nport;
Jens Axboe
maintainerPATCH 91/98 nvmet-rdma: don't run the wait list recursively
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_rdma_release_rsp() returns the send queue slots of a response and then runs the wait list, passing each waiting command to nvmet_rdma_execute_command(). Many commands complete inside their execute callback, and if the response can't be posted, nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() right there, which starts another run of the wait list inside the first:
nvmet_rdma_process_wr_wait_list()
nvmet_rdma_execute_command()
nvmet_execute_admin_connect()
nvmet_req_complete()
nvmet_rdma_queue_response()
ib_post_send() fails
nvmet_rdma_release_rsp()
nvmet_rdma_process_wr_wait_list()
Each waiting command adds a level while posts keep failing, and the host decides how many commands wait: it sizes the receive queue, can fill the send queue with commands that nvmet_req_init() rejects without taking a slot, and over siw can make posts fail by holding back an RDMA READ. 400 waiting Connects overflow the stack of the CQ worker.
Before the loop executes a command, point the response's new wait_released at a flag on the loop's stack. A release that finds it set means the loop hasn't returned from that command yet: it returns the slots and releases the response, but sets the flag and leaves the list to the loop instead of running it. The flag tells the loop not to touch the response again.
--- drivers/nvme/target/rdma.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
drivers/nvme/target/rdma.c +25 -3
@@ -77,6 +77,8 @@ struct nvmet_rdma_rsp {u32 invalidate_rkey;struct list_head wait_list; +/* set under rsp_wr_wait_lock while the wait list's loop executes us */ +bool *wait_released;int tag; };
@@ -511,23 +513,42 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,return ret; } -static void nvmet_rdma_process_wr_wait_list (struct nvmet_rdma_queue *queue) +static void nvmet_rdma_process_wr_wait_list (struct nvmet_rdma_queue *queue, +struct nvmet_rdma_rsp *done) {unsigned long flags;spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags); +if (done->wait_released) { +/* +* The loop below has not come back from this command yet. It is +* further up this stack if the command completed in place and +* its response could not be posted. Otherwise it runs in +* another context. It goes on with the list, so leave the list +* to it. +*/ + *done->wait_released = true; +WRITE_ONCE (done->wait_released,NULL ); +spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); +return ; + }while (!list_empty (&queue->rsp_wr_wait_list)) {struct nvmet_rdma_rsp *rsp; +bool released = false;bool ret; rsp =list_entry (queue->rsp_wr_wait_list.next, struct nvmet_rdma_rsp, wait_list);list_del (&rsp->wait_list); + rsp->wait_released = &released;spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); ret =nvmet_rdma_execute_command (rsp);spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags); +/* A released response can be freed, or in use again. */ +if (!released) +WRITE_ONCE (rsp->wait_released,NULL );if (!ret) {list_add (&rsp->wait_list, &queue->rsp_wr_wait_list);break ;
@@ -680,8 +701,9 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)nvmet_rdma_free_rsp_resources (rsp); -if (unlikely (!list_empty_careful (&queue->rsp_wr_wait_list))) -nvmet_rdma_process_wr_wait_list (queue); +if (unlikely (READ_ONCE (rsp->wait_released) || + !list_empty_careful (&queue->rsp_wr_wait_list))) +nvmet_rdma_process_wr_wait_list (queue, rsp);nvmet_rdma_put_rsp (rsp); }
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 92/98 nvmet-rdma: fix a lost run of the wait list
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
A command is checked for free send queue slots before it's on the wait list, in nvmet_rdma_handle_command() and when the processing loop takes it off to retry, and nvmet_rdma_release_rsp() looks at the list without a lock after giving slots back. The CQ worker and the CM thread (via nvmet_rdma_queue_established()) can handle the commands of one queue concurrently, so a release can see an empty list while the other side is between the failed slot check and the list add:
handle a command release a response
too few free slots
give the slots back wait list empty: return
add the command to the list /* nobody runs the list */
If that was the last response in flight the command stays on the list forever. It holds sq->ref, so nvmet_sq_destroy() hangs at disconnect and the queue's memory is never freed. A host hits it by sending more Connects at once than the send queue has slots.
Look at the slots again after a command is on the list: the loop retries after a put-back with a full barrier in between, nvmet_rdma_release_rsp() has a matching barrier between returning the slots and looking at the list, and nvmet_rdma_handle_command() runs the list once after adding its command.
--- drivers/nvme/target/rdma.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +17 -1
@@ -519,7 +519,7 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,unsigned long flags;spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags); -if (done->wait_released) { +if (done && done->wait_released) {/* * The loop below has not come back from this command yet. It is * further up this stack if the command completed in place and
@@ -551,6 +551,15 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,WRITE_ONCE (rsp->wait_released,NULL );if (!ret) {list_add (&rsp->wait_list, &queue->rsp_wr_wait_list); +/* +* A release that came while the command was off the +* list can have seen an empty list, so look at the +* room again. +* Pairs with the barrier in nvmet_rdma_release_rsp(). +*/ +smp_mb (); +if (atomic_read (&queue->sq_wr_avail) > rsp->n_rdma) +continue ;break ; } }
@@ -698,6 +707,11 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)struct nvmet_rdma_queue *queue = rsp->queue;atomic_add (1 + rsp->n_rdma, &queue->sq_wr_avail); +/* +* Give the room back ahead of the look at the wait list below. +* Pairs with the smp_mb() in nvmet_rdma_process_wr_wait_list(). +*/ +smp_mb__after_atomic ();nvmet_rdma_free_rsp_resources (rsp);
@@ -1021,6 +1035,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,spin_lock_irqsave (&queue->rsp_wr_wait_lock, flags);list_add_tail (&cmd->wait_list, &queue->rsp_wr_wait_list);spin_unlock_irqrestore (&queue->rsp_wr_wait_lock, flags); +/* The last response may have been released in between. */ +nvmet_rdma_process_wr_wait_list (queue,NULL ); }return ;
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 93/98 nvmet-fc: only send read data for successful commands
In reply to Jens AxboeFrom: Andrea Parri <[email protected]>
__nvmet_fc_fcp_nvme_cmd_done() pushes the whole data buffer of a read to the host even when the command failed. The buffer comes from sgl_alloc() without __GFP_ZERO, and a command rejected by nvmet_check_transfer_len() never touches it, so the host receives stale target memory, as much as it asked for.
The tcp and rdma targets only send data when cqe->status is zero. Do the same for fc. The ERSP then reports fod->offset, which is 0, as the transferred length, matching what the host checks.
--- drivers/nvme/target/fc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/fc.c +4 -4
@@ -2487,11 +2487,11 @@ __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport, }else {/* -* try to push the data even if the SQE status is non-zero. -* There may be a status where data still was intended to -* be moved +* A failed command may not have filled its read buffer, so +* only push data for success, as the tcp and rdma targets do. */ -if ((fod->io_dir ==NVMET_FCP_READ ) && (fod->data_sg_cnt)) { +if (fod->io_dir ==NVMET_FCP_READ && fod->data_sg_cnt && + !cqe->status) {/* push the data over before sending rsp */ nvmet_fc_transfer_fcp_data (tgtport, fod,NVMET_FCOP_READDATA );
Signature · 1 line
2.56.0
Re: [PATCH 93/98] nvmet-fc: only send read data for successful commands
Quoted 37 lines · Jens Axboe
From: Andrea Parri <[email protected]>
__nvmet_fc_fcp_nvme_cmd_done()pushes the whole data buffer of a read to the host even when the command failed. The buffer comes fromsgl_alloc()without__GFP_ZERO, and a command rejected bynvmet_check_transfer_len()never touches it, so the host receives stale target memory, as much as it asked for.The tcp and rdma targets only send data when
cqe->statusis zero. Do the same for fc. The ERSP then reportsfod->offset, which is 0, as the transferred length, matching what the host checks.Fixes c53432030d86 ("nvme-fabrics: Add target support for FC transport")Cc Justin Tee <[email protected]>Cc Naresh Gottumukkala <[email protected]>Cc Paul Ely <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Andrea Parri <[email protected]>Signed-off-by Chris Mason <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- drivers/nvme/target/fc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)drivers/nvme/target/fc.c +0
-2@@ -2487,11 +2487,11 @@ __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport, }else {/* -* try to push the data even if the SQE status is non-zero. -* There may be a status where data still was intended to
drivers/nvme/target/fc.c __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport, · line 2497
Quoted 10 lines
drivers/nvme/target/fc.c +4
-2-* be moved +* A failed command may not have filled its read buffer, so +* only push data for success, as the tcp and rdma targets do. */ -if ((fod->io_dir ==NVMET_FCP_READ ) && (fod->data_sg_cnt)) { +if (fod->io_dir ==NVMET_FCP_READ && fod->data_sg_cnt && + !cqe->status) {/* push the data over before sending rsp */ nvmet_fc_transfer_fcp_data (tgtport, fod,NVMET_FCOP_READDATA );
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Jens Axboe
maintainerPATCH 94/98 nvmet: wait only for executing I/O in preempt and abort
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_parse_io_cmd() takes the pc_ref for an I/O command as soon as it's parsed, and "preempt and abort" waits for all pc_refs of the preempted controllers before it releases pr_sem. But a write that still needs its data can't complete without the host sending it and without the transport context receiving it, and that same context is where Register, Acquire and Release sleep on pr_sem. So a host that sends a write and holds the data back, then preempt and abort with its own key, then Register on the same queue, deadlocks io_work, the abort work and the write with each other. Two well-behaved hosts get there by accident when one fences the other while it re-registers under I/O. The connection can't be torn down, the namespace can't be disabled, and with nvmet_config_sem held across that, nobody can connect.
Take the pc_ref when the transport calls ->execute(), once all the data is there: nvmet_parse_io_cmd() stashes the real execute method and installs nvmet_pr_execute_io(), which gets the reference, rechecks the reservation access since the host may have lost it since parse time, and calls through. Preempt and abort then only waits for commands that have started executing, which need neither the host nor the transport to finish. A write whose data arrives after the abort now fails with Reservation Conflict where it was previously executed, that's the input the hang needs.
--- drivers/nvme/target/core.c | 2 +- drivers/nvme/target/nvmet.h | 4 +++- drivers/nvme/target/pr.c | 42 ++++++++++++++++++++++++++++++++++++- 3 files changed, 45 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +1 -1
@@ -1171,7 +1171,7 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)if (ret)return ret; - ret =nvmet_pr_get_ns_pc_ref (req); +nvmet_pr_set_execute (req); }return ret; }
drivers/nvme/target/nvmet.h +3 -1
@@ -517,6 +517,8 @@ struct nvmet_req {struct nvmet_port *port;void (*execute)(struct nvmet_req *req); +/* what nvmet_pr_execute_io() calls once it has the pc_ref */ +void (*pr_execute)(struct nvmet_req *req);const struct nvmet_fabrics_ops *ops;struct pci_dev *p2p_dev;
@@ -1051,7 +1053,7 @@ void nvmet_pr_exit_ns(struct nvmet_ns *ns);void nvmet_execute_get_log_page_resv (struct nvmet_req *req);u16 nvmet_set_feat_resv_notif_mask (struct nvmet_req *req,u32 mask);u16 nvmet_get_feat_resv_notif_mask (struct nvmet_req *req); -u16 nvmet_pr_get_ns_pc_ref (struct nvmet_req *req); +void nvmet_pr_set_execute (struct nvmet_req *req);static inline void nvmet_pr_put_ns_pc_ref (struct nvmet_pr_per_ctrl_ref *pc_ref) {percpu_ref_put (&pc_ref->ref);
drivers/nvme/target/pr.c +41 -1
54 lines not on this page · raw message
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 95/98 nvmet-rdma: fix double puts after a failed RDMA WRITE
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
For a read with T10-PI the RDMA WRITE is posted alone and nvmet_rdma_write_data_done() posts the SEND once the metadata check result is known. By then the request has been completed and its references put: __nvmet_req_complete() put ns->ref and the pc_ref, and nvmet_req_complete() sq->ref, right after queuing the response. The error branch of nvmet_rdma_write_data_done() copies the one from nvmet_rdma_read_data_done() and calls nvmet_req_uninit(), which puts all three again. A WRITE fails whenever the connection ends with reads in flight, which is up to the host. Depending on timing the sq->ref count goes to zero early and the queue is freed under the remaining completions, or never reaches zero and nvmet_sq_destroy() sleeps forever, or percpu_ref_put() dereferences NULL after percpu_ref_exit(). ns->ref stays too low for good, so disabling the namespace hangs too. Reproduced with KASAN over rxe with a test patch forcing the PI path.
Drop the nvmet_req_uninit(). The branch still releases the RDMA context and the response, which is all the request owns at that point.
--- drivers/nvme/target/rdma.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +4 -1
@@ -839,7 +839,10 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)if (unlikely (wc->status !=IB_WC_SUCCESS )) {nvmet_rdma_rw_ctx_destroy (rsp); -nvmet_req_uninit (&rsp->req); +/* +* Unlike in nvmet_rdma_read_data_done(), the request has been +* completed and its references are put: no nvmet_req_uninit(). +*/ nvmet_rdma_release_rsp (rsp);if (wc->status !=IB_WC_WR_FLUSH_ERR ) { pr_info("RDMA WRITE for CQE failed with status %s (%d).\n",
Signature · 1 line
2.56.0
Jens Axboe
maintainerPATCH 96/98 nvmet: take sq->ref in nvmet_sq_fatal_error()
In reply to Jens AxboeFrom: Chris Mason <[email protected]>
nvmet_sq_destroy() kills sq->ref, waits for the commands to drop it, then puts the controller and clears sq->ctrl. So sq->ctrl may only be used while holding sq->ref. nvmet_sq_fatal_error() reads it without, and its transport callers hold nothing either: in nvmet-rdma the completion handlers run until the QP is drained, which nvmet_rdma_release_queue_work() does only after nvmet_sq_destroy(), and in nvmet-tcp io_work can run between the two cancels of the release work. A short capsule or a failed completion that races the disconnect then dereferences a NULL sq->ctrl, or locks ctrl->lock and queues fatal_err_work in a freed controller. The host needs nothing but a Connect, reproduced over rxe with a sleep added to widen the window.
Take sq->ref with percpu_ref_tryget_live() around the body. If the ref is already dead the queue is going away and the error is dropped, which is the mitigation: nothing can tell a call that would have finished in time from one that wouldn't. tryget_live after percpu_ref_exit() is fine, it sees __PERCPU_REF_DEAD and touches nothing freed, and nvmet_req_init() already relies on that. nvmet_auth_fatal_error() runs in a command that holds the ref, so it calls the body directly.
--- drivers/nvme/target/core.c | 21 ++++++++++++++++++++- drivers/nvme/target/fabrics-cmd-auth.c | 3 ++- drivers/nvme/target/nvmet.h | 1 + 3 files changed, 23 insertions(+), 2 deletions(-)
drivers/nvme/target/core.c +20 -1
@@ -1833,8 +1833,11 @@ EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);* A fatal error on one queue. The Connect command of an I/O queue needs no * secret, so until the queue has authenticated it may be anybody's: do not * let it take down the controller of the host whose NQN it has named. +* +* The caller holds sq->ref, as a command that is being executed does: until it +* is put, nvmet_sq_destroy() waits, and does not clear sq->ctrl. */ -void nvmet_sq_fatal_error (struct nvmet_sq *sq) +void __nvmet_sq_fatal_error (struct nvmet_sq *sq) {struct nvmet_ctrl *ctrl = sq->ctrl;
@@ -1843,6 +1846,22 @@ void nvmet_sq_fatal_error(struct nvmet_sq *sq)else nvmet_ctrl_fatal_error (ctrl); } + +/* +* A transport calls this with no reference of its own, concurrently with +* nvmet_sq_destroy(), which puts the controller and clears sq->ctrl. As in +* nvmet_req_init(), the call can come after percpu_ref_exit(): +* percpu_ref_tryget_live() then sees that the ref is dead and touches nothing +* that is freed. +*/ +void nvmet_sq_fatal_error (struct nvmet_sq *sq) +{ +if (!percpu_ref_tryget_live (&sq->ref)) +return ; + +__nvmet_sq_fatal_error (sq); +percpu_ref_put (&sq->ref); +}EXPORT_SYMBOL_GPL (nvmet_sq_fatal_error); ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
drivers/nvme/target/fabrics-cmd-auth.c +2 -1
@@ -33,12 +33,13 @@ static void nvmet_auth_expired_work(struct work_struct *work)mutex_unlock (&sq->auth_lock); } +/* Called from a command that is being executed, which holds sq->ref. */ static void nvmet_auth_fatal_error (struct nvmet_sq *sq,bool passed) {if (passed)nvmet_ctrl_fatal_error (sq->ctrl);else -nvmet_sq_fatal_error (sq); +__nvmet_sq_fatal_error (sq); }void nvmet_auth_sq_init (struct nvmet_sq *sq)
drivers/nvme/target/nvmet.h +1 -0
@@ -631,6 +631,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);int nvmet_sq_init (struct nvmet_sq *sq,struct nvmet_cq *cq);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl); +void __nvmet_sq_fatal_error (struct nvmet_sq *sq);void nvmet_sq_fatal_error (struct nvmet_sq *sq);void nvmet_update_cc (struct nvmet_ctrl *ctrl,u32 new);
Signature · 1 line
2.56.0