Skip to contentSkip to conversation

nvmet: Check the admin queue before reading its TLS key in debugfs

PATCH In review Guixin Liu 3 participants 6 messages Oct 28 – Oct 29

Subsystems (2)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH 1/1 nvmet: Check the admin queue before reading its TLS key in debugfs

nvmet_ctrl_tls_key_show() dereferences ctrl->sqs[0] unconditionally, but the admin queue is not installed yet while the controller is being created, and sqs[0] is set to NULL again when it is destroyed. Reading the tls_key debugfs entry in either window crashes the kernel.

Only query the queue's TLS key when the admin queue is present and report 0 otherwise.

Fixes 316dabe6089fe ("nvmet: add tls_concat and tls_key debugfs entries")
---
 drivers/nvme/target/debugfs.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/target/debugfs.c +3 -1
index e85fe1d4c9f8..a27a501fe61a 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
 static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)
 {
 	struct nvmet_ctrl *ctrl = m->private;
-	key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+	key_serial_t keyid = 0;
 
+	if (ctrl->sqs[0])
+		keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
 	seq_printf(m, "%08x\n", keyid);
 	return 0;
 }
Signature · 1 line
2.43.7
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] nvmet: unsynchronized ctrl->sqs[0] access in nvmet_ctrl_tls_key_show() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] nvmet: unsynchronized ctrl->sqs[0] access in nvmet_ctrl_tls_key_show() --

commit 3cf90339b7ea83703b0eb52aef8d8c4af5c44e19
Author: Guixin Liu <[email protected]>

nvmet: Check the admin queue before reading its TLS key in debugfs

This commit checks whether ctrl->sqs[0] is non-NULL in nvmet_ctrl_tls_key_show() before querying the admin queue's TLS key ID for debugfs output.

drivers/nvme/target/debugfs.c NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr); · line 142

Quoted 13 lines
drivers/nvme/target/debugfs.c +3 -1
index e85fe1d4c9f8d..a27a501fe61a4 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
 static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)
 {
 	struct nvmet_ctrl *ctrl = m->private;
-	key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+	key_serial_t keyid = 0;
 
+	if (ctrl->sqs[0])
+		keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);

[Severity: High]
Can this unlocked check and reload of ctrl->sqs[0] race with admin queue teardown in nvmet_tcp_release_queue_work() and nvmet_sq_destroy(), or with the error path in nvmet_install_queue()?

First, ctrl->sqs[0] is checked and then reloaded without READ_ONCE() or any lock synchronizing against nvmet_sq_destroy() in drivers/nvme/target/core.c:

    if (sq->qid == 0) {
        ctrl->reset_tbkas = true;
        sq->ctrl->sqs[sq->qid] = NULL;
        nvmet_ctrl_put(ctrl);
    }

If ctrl->sqs[0] is cleared to NULL between the if (ctrl->sqs[0]) check and the argument evaluation for nvmet_queue_tls_keyid(ctrl->sqs[0]), NULL is passed into nvmet_queue_tls_keyid() in drivers/nvme/target/nvmet.h:

    static inline key_serial_t nvmet_queue_tls_keyid(struct nvmet_sq *sq)
    {
        return sq->tls_key ? key_serial(sq->tls_key) : 0;
    }

which dereferences sq->tls_key on a NULL pointer.

Second, even when ctrl->sqs[0] is read as non-NULL, nothing pins the queue or sq->tls_key while nvmet_ctrl_tls_key_show() runs. During queue teardown in drivers/nvme/target/tcp.c, nvmet_tcp_release_queue_work() executes:

    nvmet_sq_put_tls_key(&queue->nvme_sq);
    nvmet_tcp_uninit_data_in_cmds(queue);
    nvmet_sq_destroy(&queue->nvme_sq);
    ...
    kfree(queue);

where nvmet_sq_put_tls_key() in drivers/nvme/target/nvmet.h drops the key reference before ctrl->sqs[0] is cleared:

    static inline void nvmet_sq_put_tls_key(struct nvmet_sq *sq)
    {
        if (sq->tls_key) {
            key_put(sq->tls_key);
            sq->tls_key = NULL;
        }
    }

Because the controller debugfs directory stays alive until the final ctrl->ref is dropped in nvmet_ctrl_free() -> nvmet_debugfs_ctrl_free(ctrl) (for example, while queue->nvme_cq or I/O queues still hold controller references), a concurrent reader of the tls_key debugfs file can dereference sq->tls_key after key_put() or dereference sq after kfree(queue).

Could this be avoided by reading the key directly from ctrl->tls_key (or caching the key serial on struct nvmet_ctrl), or by properly serializing access to the queue and its key?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Guixin Liu
Quoted 21 lines · Guixin Liu

nvmet_ctrl_tls_key_show() dereferences ctrl->sqs[0] unconditionally, but the admin queue is not installed yet while the controller is being created, and sqs[0] is set to NULL again when it is destroyed. Reading the tls_key debugfs entry in either window crashes the kernel.

Only query the queue's TLS key when the admin queue is present and report 0 otherwise.

Fixes 316dabe6089fe ("nvmet: add tls_concat and tls_key debugfs entries")
---
  drivers/nvme/target/debugfs.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/target/debugfs.c +0 -0
index e85fe1d4c9f8..a27a501fe61a 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
  static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)

drivers/nvme/target/debugfs.c NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr); · line 145

Quoted 10 lines
drivers/nvme/target/debugfs.c +3 -1
  {
  	struct nvmet_ctrl *ctrl = m->private;
-	key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+	key_serial_t keyid = 0;
 
+	if (ctrl->sqs[0])
+		keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
  	seq_printf(m, "%08x\n", keyid);
  	return 0;
  }

I think it might be easier if 'nvmet_queue_tls_keyid()' would accept a NULL argument.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author
#3 lore raw
Quoted 26 lines · Hannes Reinecke
Quoted 25 lines · Guixin Liu

nvmet_ctrl_tls_key_show() dereferences ctrl->sqs[0] unconditionally, but the admin queue is not installed yet while the controller is being created, and sqs[0] is set to NULL again when it is destroyed. Reading the tls_key debugfs entry in either window crashes the kernel.

Only query the queue's TLS key when the admin queue is present and report 0 otherwise.

Fixes 316dabe6089fe ("nvmet: add tls_concat and tls_key debugfs entries")

---
  drivers/nvme/target/debugfs.c | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/target/debugfs.c +1 -1
index e85fe1d4c9f8..a27a501fe61a 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
   static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)
   {
       struct nvmet_ctrl *ctrl = m->private;
-    key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+    key_serial_t keyid = 0;
Quoted 8 lines

drivers/nvme/target/debugfs.c NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr); · line 145

Quoted 5 lines
drivers/nvme/target/debugfs.c +1 -0
   +    if (ctrl->sqs[0])
+        keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
       seq_printf(m, "%08x\n", keyid);
       return 0;
   }

I think it might be easier if 'nvmet_queue_tls_keyid()' would accept a NULL argument.

This is simpler, however Sashiko's review points out that after the NULL check, sq->tls_key clould be freed by nvmet_sq_put_tls_key(), or sq itself could be destroyed.

My idea is store admin queue's tls_key to nvmet_ctrl to prevent access sq and sq->tls_key in debugfs interface, what do you think?

Best Regards,
Guixin Liu

Quoted 4 lines

Cheers,

Hannes

#4 lore raw
Quoted 37 lines · Guixin Liu
Quoted 26 lines · Hannes Reinecke
Quoted 25 lines · Guixin Liu

nvmet_ctrl_tls_key_show() dereferences ctrl->sqs[0] unconditionally, but the admin queue is not installed yet while the controller is being created, and sqs[0] is set to NULL again when it is destroyed. Reading the tls_key debugfs entry in either window crashes the kernel.

Only query the queue's TLS key when the admin queue is present and report 0 otherwise.

Fixes 316dabe6089fe ("nvmet: add tls_concat and tls_key debugfs entries")
---
   drivers/nvme/target/debugfs.c | 4 +++-
   1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/target/debugfs.c +1 -1
index e85fe1d4c9f8..a27a501fe61a 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
   static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)
   {
       struct nvmet_ctrl *ctrl = m->private;
-    key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+    key_serial_t keyid = 0;
Quoted 8 lines

drivers/nvme/target/debugfs.c NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr); · line 145

Quoted 5 lines
drivers/nvme/target/debugfs.c +1 -0
   +    if (ctrl->sqs[0])
+        keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
       seq_printf(m, "%08x\n", keyid);
       return 0;
   }

I think it might be easier if 'nvmet_queue_tls_keyid()' would accept a NULL argument.

Quoted 7 lines

This is simpler, however Sashiko's review points out that after the NULL check, sq->tls_key clould be freed by nvmet_sq_put_tls_key(), or sq itself could be destroyed.

My idea is store admin queue's tls_key to nvmet_ctrl to prevent access sq and sq->tls_key in debugfs interface, what do you think?

Argl. Don't.

I would do something like

key_serial_t keyid = ctrl->sqs ? nvmet_queue_tls_keyid(ctrl->sqs[0]):0;

to make it clear that ctrl->sqs might be NULL here.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author
#5 lore raw
Quoted 49 lines · Hannes Reinecke
Quoted 37 lines · Guixin Liu
Quoted 26 lines · Hannes Reinecke
Quoted 25 lines · Guixin Liu

nvmet_ctrl_tls_key_show() dereferences ctrl->sqs[0] unconditionally, but the admin queue is not installed yet while the controller is being created, and sqs[0] is set to NULL again when it is destroyed. Reading the tls_key debugfs entry in either window crashes the kernel.

Only query the queue's TLS key when the admin queue is present and report 0 otherwise.

Fixes 316dabe6089fe ("nvmet: add tls_concat and tls_key debugfs entries")

---
   drivers/nvme/target/debugfs.c | 4 +++-

   1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/target/debugfs.c +1 -1
index e85fe1d4c9f8..a27a501fe61a 100644
@@ -136,8 +136,10 @@ NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr);
    static int nvmet_ctrl_tls_key_show(struct seq_file *m, void *p)
    {
        struct nvmet_ctrl *ctrl = m->private;
-    key_serial_t keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
+    key_serial_t keyid = 0;
Quoted 8 lines

drivers/nvme/target/debugfs.c NVMET_DEBUGFS_ATTR(nvmet_ctrl_host_traddr); · line 145

Quoted 5 lines
drivers/nvme/target/debugfs.c +1 -0
    +    if (ctrl->sqs[0])
+        keyid = nvmet_queue_tls_keyid(ctrl->sqs[0]);
        seq_printf(m, "%08x\n", keyid);
        return 0;
    }

I think it might be easier if 'nvmet_queue_tls_keyid()' would accept a NULL argument.

Quoted 7 lines

This is simpler, however Sashiko's review points out that after the NULL check, sq->tls_key clould be freed by nvmet_sq_put_tls_key(), or sq itself could be destroyed.

My idea is store admin queue's tls_key to nvmet_ctrl to prevent access sq and sq->tls_key in debugfs interface, what do you think?

Argl. Don't.

I would do something like

Quoted 7 lines

key_serial_t keyid = ctrl->sqs ? nvmet_queue_tls_keyid(ctrl->sqs[0]):0;

to make it clear that ctrl->sqs might be NULL here.

Cheers,

Hannes

OK, I will send a v2 patch.

Best Regards,
Guixin Liu