Skip to contentSkip to conversation

nvme: fabrics and target fixes

PATCH Applied Guixin Liu 4 participants 11 messages Nov 16 – Nov 20

Subsystems (3)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v3 cover nvme: fabrics and target fixes

Two remaining fixes from the previous rounds of this series:

- a use-after-free in nvme_fc_match_disconn_ls(): no iterator is sound across the rport lock drop inside the loop, so the loop is reworked to run entirely under the rport lock and to take a reference only for the match; reproduced under KASAN through fcloop
- a data race on the shared authentication transaction counter, reworked per review to a per-queue counter, dropping the shared ctrl->transaction field entirely

The other three patches from v2 are already resolved: the GFP_KERNEL persistent reservation change and the tbkas configfs locking are in nvme-7.4 now, and the reconnect_delay fix was superseded by an earlier identical patch posted by someone else.

Both patches are verified on an initiator/target VM pair: the fc loop is compile tested and exercised under KASAN through fcloop with six controllers on one remote port across tbkas force-reconnect cycles and concurrent disconnects with no reports; the DH-HMAC-CHAP authenticated session including I/O covers the auth path.

Changes since v2:

- rework the nvme-auth fix to a per-queue counter (Keith Busch), drop the now unused shared field, and add the Fixes tag (sashiko); the Reviewed-by on v2 covered the atomic_t variant and is dropped
- resend the unchanged nvme-fc patch, still awaiting review
- drop the reconnect_delay patch (superseded), the GFP_KERNEL patch (applied) and the tbkas patch (applied)

Guixin Liu (2):
nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls()
nvme-auth: fix data race on ctrl->transaction

 drivers/nvme/host/auth.c |  3 +--
 drivers/nvme/host/fc.c   | 22 ++++++++++++----------
 drivers/nvme/host/nvme.h |  1 -
 3 files changed, 13 insertions(+), 13 deletions(-)
Signature · 1 line
2.43.7
author
#1 lore raw

PATCH v3 1/2 nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls()

Commit b71cbcf7d170e ("nvme-fc: don't hold rport lock when putting ctrl") dropped the rport lock inside the loop to put the reference of a non-matching controller, because nvme_fc_ctrl_free() takes the rport lock to unlink the controller. But no iterator is sound across that
drop: with list_for_each_entry_safe() the prefetched next entry can be freed in the window, and with a plain iterator the loop's own nvme_fc_ctrl_put() can be the final reference, freeing the current entry before the iterator reads its ->next pointer.

The reference dance is unnecessary in the first place. An entry cannot leave ctrl_list while rport->lock is held, as the final put unlinks it under that very lock, so association_id can be inspected under ctrl->lock without holding a reference. Take a reference only for the matching controller, still under the rport lock, and hand it to the caller as before. The loop then no longer needs to drop the lock at all, which also removes the reason the lock was dropped for in the first place.

Fixes b71cbcf7d170e ("nvme-fc: don't hold rport lock when putting ctrl")

---

Reproduced under KASAN in a VM through fcloop with several controllers on one remote port and a tbkas force-reconnect racing concurrent controller disconnects; the lock-drop window was widened with a test-only mdelay(200) to make the microsecond window deterministic:

  BUG: KASAN: slab-use-after-free in nvme_fc_match_disconn_ls
  Read of size 8 by task kworker
  Workqueue: events nvme_fc_handle_ls_rqst_work
  Call Trace:
   nvme_fc_match_disconn_ls
   nvme_fc_ls_disconnect_assoc
   nvme_fc_handle_ls_rqst_work
   process_one_work

The freed object was the prefetched next entry, allocated by nvme_fc_alloc_ctrl() on connect and freed by nvme_free_ctrl() on the concurrent teardown. The loop increment dereferences it before nvme_fc_ctrl_get() gets a chance to reject it, which is why the get-unless-zero check in the loop body does not help. The reworked loop does not drop the lock, so no such window is left to widen.

---
 drivers/nvme/host/fc.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
drivers/nvme/host/fc.c +12 -10
index 2cbd744ffd22..ce4598dd227f 100644
@@ -1469,29 +1469,31 @@ nvme_fc_match_disconn_ls(struct nvme_fc_rport *rport,
 {
 	struct fcnvme_ls_disconnect_assoc_rqst *rqst =
 					&lsop->rqstbuf->rq_dis_assoc;
-	struct nvme_fc_ctrl *ctrl, *tmp, *ret = NULL;
+	struct nvme_fc_ctrl *ctrl, *ret = NULL;
 	struct nvmefc_ls_rcv_op *oldls = NULL;
 	u64 association_id = be64_to_cpu(rqst->associd.association_id);
 	unsigned long flags;
 
 	spin_lock_irqsave(&rport->lock, flags);
 
-	list_for_each_entry_safe(ctrl, tmp, &rport->ctrl_list, ctrl_list) {
-		if (!nvme_fc_ctrl_get(ctrl))
-			continue;
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
 		spin_lock(&ctrl->lock);
 		if (association_id == ctrl->association_id) {
-			oldls = ctrl->rcv_disconn;
-			ctrl->rcv_disconn = lsop;
-			ret = ctrl;
+			/*
+			 * Entries cannot leave the list while rport->lock is
+			 * held, so they can be inspected without a reference.
+			 * Only the match needs one, for the caller.
+			 */
+			if (nvme_fc_ctrl_get(ctrl)) {
+				oldls = ctrl->rcv_disconn;
+				ctrl->rcv_disconn = lsop;
+				ret = ctrl;
+			}
 		}
 		spin_unlock(&ctrl->lock);
 		if (ret)
 			/* leave the ctrl get reference */
 			break;
-		spin_unlock_irqrestore(&rport->lock, flags);
-		nvme_fc_ctrl_put(ctrl);
-		spin_lock_irqsave(&rport->lock, flags);
 	}
 
 	spin_unlock_irqrestore(&rport->lock, flags);
Signature · 1 line
2.43.7
author
#3 lore raw

PATCH v3 2/2 nvme-auth: fix data race on ctrl->transaction

In reply to Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

Fixes f50fff73d620 ("nvme: implement In-Band authentication")

---

Changes since v2: reworked per review to use a per-queue counter and drop the shared field entirely (Keith Busch), and the Fixes tag was added (sashiko). The Reviewed-by on v2 covered the atomic_t version of this change and is dropped.

---
 drivers/nvme/host/auth.c | 3 +--
 drivers/nvme/host/nvme.h | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/auth.c +1 -2
index 9a3da74dec5a..54f3354ecae9 100644
@@ -644,7 +644,6 @@ static void nvme_auth_reset_dhchap(struct nvme_dhchap_queue_context *chap)
 	chap->s1 = 0;
 	chap->s2 = 0;
 	chap->bi_directional = false;
-	chap->transaction = 0;
 	memset(chap->c1, 0, sizeof(chap->c1));
 	memset(chap->c2, 0, sizeof(chap->c2));
 	mempool_free(chap->buf, nvme_chap_buf_pool);
@@ -764,7 +763,7 @@ static void nvme_queue_auth_work(struct work_struct *work)
 		return;
 	}
 
-	chap->transaction = ctrl->transaction++;
+	chap->transaction++;
 
 	/* DH-HMAC-CHAP Step 1: send negotiate */
 	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
drivers/nvme/host/nvme.h +0 -1
index 2543e52d6358..e015d3a3e29c 100644
@@ -463,7 +463,6 @@ struct nvme_ctrl {
 	struct nvme_dhchap_queue_context *dhchap_ctxs;
 	struct nvme_dhchap_key *host_key;
 	struct nvme_dhchap_key *ctrl_key;
-	u16 transaction;
 #endif
 	key_serial_t tls_pskid;
 
Signature · 1 line
2.43.7
maintainer
#4 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

Quoted 7 lines · Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

Quoted 6 lines

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

I disagree. The authentication transaction ID identifies the transaction itself, and is used to differentiate between different transactions. As such it needs to be unique within each queue. But as it carries controller-specific data we should also ensure the ID is unique per controller, ensuring we catch any possible misdirected authentication packet.

It should be possible to move the controller value to an atomic value, and then we could do

chap->transaction = atomic_inc_return(&ctrl->transaction);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer
#5 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

Quoted 14 lines · Hannes Reinecke
Quoted 7 lines · Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

Quoted 6 lines

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

Quoted 3 lines

I disagree. The authentication transaction ID identifies the transaction itself, and is used to differentiate between different transactions. As such it needs to be unique within each queue.

But the proposal *is* unique within the queue. It's just not unique at the controller level, and it sounds like the protocol doesn't need to be.

Quoted 3 lines

But as it carries controller-specific data we should also ensure the ID is unique per controller, ensuring we catch any possible misdirected authentication packet.

I don't understand. The qid:T_ID tuple already identifies the transaction. If a controller mixes up transactions across queues, it's already broken in ways we can't fix, so we shouldn't design around it.

Quoted 2 lines

It should be possible to move the controller value to an atomic value, and then we could do

If there's no protocol reason to require this type of uniqueness, then I'm not seeing why the driver is putting this constraint on itself.

maintainer
#6 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

Quoted 31 lines · Keith Busch
Quoted 14 lines · Hannes Reinecke
Quoted 7 lines · Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

Quoted 6 lines

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

Quoted 3 lines

I disagree. The authentication transaction ID identifies the transaction itself, and is used to differentiate between different transactions. As such it needs to be unique within each queue.

But the proposal *is* unique within the queue. It's just not unique at the controller level, and it sounds like the protocol doesn't need to be.

Quoted 3 lines

But as it carries controller-specific data we should also ensure the ID is unique per controller, ensuring we catch any possible misdirected authentication packet.

I don't understand. The qid:T_ID tuple already identifies the transaction. If a controller mixes up transactions across queues, it's already broken in ways we can't fix, so we shouldn't design around it.

Quoted 5 lines
Quoted 2 lines

It should be possible to move the controller value to an atomic value, and then we could do

If there's no protocol reason to require this type of uniqueness, then I'm not seeing why the driver is putting this constraint on itself.

It's somewhat ambiguous.
Spec says 'carrying the same transaction identifier value for the entire authentication transaction' (for AUTH_negotiate), and later on 'carrying the same transaction identifier' (for any other AUTH message). So actually we just have to make sure they are identical. (The random number is '4' ...)

But the overall idea here is to distinguish transactions, so the implication is that each transaction should carry different identifiers. And my issue here is precisely 'each transaction'. Surely within each
queue, but I'd really would want to distinguish _across_ queues, too.
(And possibly with a random starting value to boot).
Just to avoid any fuzzers trying to be clever with guessing the authentication ID (or bugs setting the transaction ID to 0 for every queue ... same reason why we introduced the 'genctr' for the command
ID).

But if you don't feel that's an issue, fine, I'll be shutting up.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer
#7 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

Quoted 19 lines · Hannes Reinecke

On 11/16/26 4:37 PM, Keith Busch wrote:

It's somewhat ambiguous.
Spec says 'carrying the same transaction identifier value for the entire authentication transaction' (for AUTH_negotiate), and later on 'carrying the same transaction identifier' (for any other AUTH message). So actually we just have to make sure they are identical. (The random number is '4' ...)

But the overall idea here is to distinguish transactions, so the implication is that each transaction should carry different identifiers.
And my issue here is precisely 'each transaction'. Surely within each queue, but I'd really would want to distinguish _across_ queues, too. (And possibly with a random starting value to boot).
Just to avoid any fuzzers trying to be clever with guessing the authentication ID (or bugs setting the transaction ID to 0 for every queue ... same reason why we introduced the 'genctr' for the command ID).

Quoted 1 line

But if you don't feel that's an issue, fine, I'll be shutting up.

I'd normally defer to you on this feature since you've more hands-on experience here. I just think the qid:t_id pair already distinguishes transactions on the wire; each queue is its own connection after all.

Regarding genctr, I thought that was because some intermediary switch was replaying packets, but the commit message doesn't appear to support that recollection. Regardless, the value is per-request, so it's on an even more finer granularity than per-queue.

author
#8 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

In reply to Hannes Reinecke
Quoted 14 lines · Hannes Reinecke
Quoted 7 lines · Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

Quoted 6 lines

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

Quoted 6 lines

I disagree. The authentication transaction ID identifies the transaction itself, and is used to differentiate between different transactions. As such it needs to be unique within each queue. But as it carries controller-specific data we should also ensure the ID is unique per controller, ensuring we catch any possible misdirected authentication packet.

A misdirected packet is caught regardless of the transaction id. Each queue's exchange carries its own random challenge:
nvmet_auth_challenge() fills req->sq->dhcahp_c1 with get_random_bytes() per exchange, and the response digest covers that challenge. A response computed for queue A fails the digest check on queue B even when the transaction id and the step happen to match, because the expected digest is different.

So controller-level uniqueness would only detect a case the crypto layer already rejects, while within a queue the id is unique either way.

Best Regards,
Guixin Liu

Quoted 9 lines

It should be possible to move the controller value to an atomic value, and then we could do

chap->transaction = atomic_inc_return(&ctrl->transaction);

Cheers,

Hannes

maintainer
#9 lore raw

Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction

Quoted 31 lines · Guixin Liu
Quoted 14 lines · Hannes Reinecke
Quoted 7 lines · Guixin Liu

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue on nvme-auth-wq without waiting for them, so multiple nvme_queue_auth_work() instances run concurrently and all increment ctrl->transaction in nvme_queue_auth_work(). The plain increment is a data race, and the lost updates hand the same transaction id to concurrent queues.

Quoted 6 lines

The counter does not need to be shared at all: the controller only echoes the transaction id within a queue, so a per-queue counter is sufficient. Increment chap->transaction instead, drop the reset in nvme_auth_reset_dhchap() so the id stays monotonic across re-authentication, and remove the now unused ctrl->transaction field.

Quoted 6 lines

I disagree. The authentication transaction ID identifies the transaction itself, and is used to differentiate between different transactions. As such it needs to be unique within each queue. But as it carries controller-specific data we should also ensure the ID is unique per controller, ensuring we catch any possible misdirected authentication packet.

A misdirected packet is caught regardless of the transaction id. Each queue's exchange carries its own random challenge:
nvmet_auth_challenge() fills req->sq->dhcahp_c1 with get_random_bytes() per exchange, and the response digest covers that challenge. A response computed for queue A fails the digest check on queue B even when the transaction id and the step happen to match, because the expected digest is different.

Quoted 4 lines

So controller-level uniqueness would only detect a case the crypto layer already rejects, while within a queue the id is unique either way.

True, but that's valid only for challenge and response. Success (and, more importantly, Failure) are not protected in a similar manner. So spoofing any of these will be quite easy if we only have per-queue transaction ids.

But as mentioned in my response to Keith: these are primarily security concerns, and really are just stop-gap measures to cover the most obvious issues.
In the end it's still a number and as such
not _that_ hard to guess.
So if the consensus is that the protocol provisions are good enough it's fine with me.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer Applied
#10 lore raw

Re: (subset) [PATCH v3 0/2] nvme: fabrics and target fixes

In reply to Guixin Liu
Quoted 11 lines · Guixin Liu

Two remaining fixes from the previous rounds of this series:

- a use-after-free in nvme_fc_match_disconn_ls(): no iterator is sound across the rport lock drop inside the loop, so the loop is reworked to run entirely under the rport lock and to take a reference only for the match; reproduced under KASAN through fcloop
- a data race on the shared authentication transaction counter, reworked per review to a per-queue counter, dropping the shared ctrl->transaction field entirely

[...]

Thanks, applied to nvme-7.4.

[1/2] nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls()

      commit: 505384850afd4c85f6d58cb3eb078a71f83d849d

--
Keith Busch <[email protected]>