Skip to contentSkip to conversation

nvme: check for valid data from from nvme_identify_ns() before using it

PATCH Superseded Ewan D. Milne 3 participants 8 messages Nov 10, 2023 – Nov 27, 2023 NVMe

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: check for valid data from from nvme_identify_ns() before using it

When scanning namespaces, it is possible to get valid data from the first call to nvme_identify_ns() in nvme_alloc_ns(), but not from the second call in nvme_update_ns_info_block(). In particular, if the NSID becomes inactive between the two commands, a storage device may return a buffer filled with zero as per 4.1.5.1. In this case, we can get a kernel crash due to a divide-by-zero in blk_stack_limits() because ns->lba_shift will be set to zero.

PID: 326 TASK: ffff95fec3cd8000 CPU: 29 COMMAND: "kworker/u98:10"
#0 [ffffad8f8702f9e0] machine_kexec at ffffffff91c76ec7 #1 [ffffad8f8702fa38] __crash_kexec at ffffffff91dea4fa #2 [ffffad8f8702faf8] crash_kexec at ffffffff91deb788 #3 [ffffad8f8702fb00] oops_end at ffffffff91c2e4bb #4 [ffffad8f8702fb20] do_trap at ffffffff91c2a4ce
#5 [ffffad8f8702fb70] do_error_trap at ffffffff91c2a595 #6 [ffffad8f8702fbb0] exc_divide_error at ffffffff928506e6 #7 [ffffad8f8702fbd0] asm_exc_divide_error at ffffffff92a00926

    [exception RIP: blk_stack_limits+434]
    RIP: ffffffff92191872  RSP: ffffad8f8702fc80  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff95efa0c91800  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: 0000000000000001
    RBP: 00000000ffffffff   R8: ffff95fec7df35a8   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000001  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff95fed33c09a8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

#8 [ffffad8f8702fce0] nvme_update_ns_info_block at ffffffffc06d3533 [nvme_core] #9 [ffffad8f8702fd18] nvme_scan_ns at ffffffffc06d6fa7 [nvme_core]

This happened when the check for valid data was moved out of nvme_identify_ns() into one of the callers. Fix this by checking in both callers.

Fixes 0dd6fff2aad4 ("nvme: bring back auto-removal of deleted namespaces during sequential scan")
---
 drivers/nvme/host/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/host/core.c +6 -0
index 88b54cdcbd68..ae35b19cd562 100644
@@ -2027,6 +2027,12 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
 	if (ret)
 		return ret;
 
+	if (id->ncap == 0) {
+		/* namespace not allocated or attached */
+		info->is_removed = true;
+		return -ENODEV;
+	}
+
 	blk_mq_freeze_queue(ns->disk->queue);
 	lbaf = nvme_lbaf_index(id->flbas);
 	ns->lba_shift = id->lbaf[lbaf].ds;
Signature · 1 line
2.20.1
author
#1 lore raw

Keith / Christoph-

Can you take a look at this fix please? I've got someone getting this crash fairly reliably, their customers are vulnerable.

-Ewan

Quoted 60 lines · Ewan D. Milne

When scanning namespaces, it is possible to get valid data from the first call to nvme_identify_ns() in nvme_alloc_ns(), but not from the second call in nvme_update_ns_info_block(). In particular, if the NSID becomes inactive between the two commands, a storage device may return a buffer filled with zero as per 4.1.5.1. In this case, we can get a kernel crash due to a divide-by-zero in blk_stack_limits() because ns->lba_shift will be set to zero.

PID: 326 TASK: ffff95fec3cd8000 CPU: 29 COMMAND: "kworker/u98:10"
#0 [ffffad8f8702f9e0] machine_kexec at ffffffff91c76ec7 #1 [ffffad8f8702fa38] __crash_kexec at ffffffff91dea4fa #2 [ffffad8f8702faf8] crash_kexec at ffffffff91deb788 #3 [ffffad8f8702fb00] oops_end at ffffffff91c2e4bb #4 [ffffad8f8702fb20] do_trap at ffffffff91c2a4ce
#5 [ffffad8f8702fb70] do_error_trap at ffffffff91c2a595 #6 [ffffad8f8702fbb0] exc_divide_error at ffffffff928506e6 #7 [ffffad8f8702fbd0] asm_exc_divide_error at ffffffff92a00926

    [exception RIP: blk_stack_limits+434]
    RIP: ffffffff92191872  RSP: ffffad8f8702fc80  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff95efa0c91800  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: 0000000000000001
    RBP: 00000000ffffffff   R8: ffff95fec7df35a8   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000001  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff95fed33c09a8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

#8 [ffffad8f8702fce0] nvme_update_ns_info_block at ffffffffc06d3533 [nvme_core] #9 [ffffad8f8702fd18] nvme_scan_ns at ffffffffc06d6fa7 [nvme_core]

This happened when the check for valid data was moved out of nvme_identify_ns() into one of the callers. Fix this by checking in both callers.

Fixes 0dd6fff2aad4 ("nvme: bring back auto-removal of deleted namespaces during sequential scan")
---
 drivers/nvme/host/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/host/core.c +6 -1
index 88b54cdcbd68..ae35b19cd562 100644
@@ -2027,6 +2027,12 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
        if (ret)
                return ret;
 
+       if (id->ncap == 0) {
+               /* namespace not allocated or attached */
+               info->is_removed = true;
+               return -ENODEV;
+       }
+
        blk_mq_freeze_queue(ns->disk->queue);
        lbaf = nvme_lbaf_index(id->flbas);
        ns->lba_shift = id->lbaf[lbaf].ds;
--

2.20.1

author
#2 lore raw

Oh, never mind, I see it. Just didn't see anything on the list. Sorry.

-Ewan

Quoted 69 lines · Ewan Milne

Keith / Christoph-

Can you take a look at this fix please? I've got someone getting this crash fairly reliably, their customers are vulnerable.

-Ewan

Quoted 60 lines · Ewan D. Milne

When scanning namespaces, it is possible to get valid data from the first call to nvme_identify_ns() in nvme_alloc_ns(), but not from the second call in nvme_update_ns_info_block(). In particular, if the NSID becomes inactive between the two commands, a storage device may return a buffer filled with zero as per 4.1.5.1. In this case, we can get a kernel crash due to a divide-by-zero in blk_stack_limits() because ns->lba_shift will be set to zero.

PID: 326 TASK: ffff95fec3cd8000 CPU: 29 COMMAND: "kworker/u98:10"
#0 [ffffad8f8702f9e0] machine_kexec at ffffffff91c76ec7 #1 [ffffad8f8702fa38] __crash_kexec at ffffffff91dea4fa #2 [ffffad8f8702faf8] crash_kexec at ffffffff91deb788 #3 [ffffad8f8702fb00] oops_end at ffffffff91c2e4bb #4 [ffffad8f8702fb20] do_trap at ffffffff91c2a4ce
#5 [ffffad8f8702fb70] do_error_trap at ffffffff91c2a595 #6 [ffffad8f8702fbb0] exc_divide_error at ffffffff928506e6 #7 [ffffad8f8702fbd0] asm_exc_divide_error at ffffffff92a00926

    [exception RIP: blk_stack_limits+434]
    RIP: ffffffff92191872  RSP: ffffad8f8702fc80  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff95efa0c91800  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: 0000000000000001
    RBP: 00000000ffffffff   R8: ffff95fec7df35a8   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000001  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff95fed33c09a8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

#8 [ffffad8f8702fce0] nvme_update_ns_info_block at ffffffffc06d3533 [nvme_core] #9 [ffffad8f8702fd18] nvme_scan_ns at ffffffffc06d6fa7 [nvme_core]

This happened when the check for valid data was moved out of nvme_identify_ns() into one of the callers. Fix this by checking in both callers.

Fixes 0dd6fff2aad4 ("nvme: bring back auto-removal of deleted namespaces during sequential scan")
---
 drivers/nvme/host/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/host/core.c +6 -1
index 88b54cdcbd68..ae35b19cd562 100644
@@ -2027,6 +2027,12 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
        if (ret)
                return ret;
 
+       if (id->ncap == 0) {
+               /* namespace not allocated or attached */
+               info->is_removed = true;
+               return -ENODEV;
+       }
+
        blk_mq_freeze_queue(ns->disk->queue);
        lbaf = nvme_lbaf_index(id->flbas);
        ns->lba_shift = id->lbaf[lbaf].ds;
--

2.20.1

maintainer
#3 lore raw
Quoted 1 line · Ewan Milne

Oh, never mind, I see it. Just didn't see anything on the list. Sorry.

Is it? I don't see it in any of the staging or upstream trees, but thank you for pinging me on the patch.

Quoted 8 lines
Quoted 8 lines

drivers/nvme/host/core.c static int nvme_update_ns_info_block(struct nvme_ns *ns, · line 2033

Quoted 8 lines
drivers/nvme/host/core.c +4 -0
@@ -2027,6 +2027,12 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
        if (ret)
                return ret;
 
+       if (id->ncap == 0) {
+               /* namespace not allocated or attached */
+               info->is_removed = true;
+               return -ENODEV;

The concept sounds good, though you are leaking 'id' in the new error case.

#4 lore raw

út 21. 11. 2023 v 19:14 odesílatel Keith Busch <[email protected]> napsal:

Quoted 2 lines

The concept sounds good, though you are leaking 'id' in the new error case.

We have the same problem in the nvme_ns_info_from_identify() function.

Maurizio

maintainer
#5 lore raw
Quoted 5 lines · Maurizio Lombardi

'ut 21. 11. 2023 v 19:14 odes'ilatel Keith Busch <[email protected]> napsal:

Quoted 2 lines

The concept sounds good, though you are leaking 'id' in the new error case.

We have the same problem in the nvme_ns_info_from_identify() function.

Indeed you are correct. Would you like to send the patch for it? This patch will also need a v2 to incorporate the same.

#6 lore raw

st 22. 11. 2023 v 17:31 odesílatel Keith Busch <[email protected]> napsal:

Quoted 7 lines
Quoted 3 lines

We have the same problem in the nvme_ns_info_from_identify() function.

Indeed you are correct. Would you like to send the patch for it? This patch will also need a v2 to incorporate the same.

Ok, I will send it later today.

Maurizio

author
#7 lore raw
In reply to Keith Busch
Quoted 6 lines · Keith Busch
Quoted 1 line · Ewan Milne

Oh, never mind, I see it. Just didn't see anything on the list. Sorry.

Is it? I don't see it in any of the staging or upstream trees, but thank you for pinging me on the patch.

Sorry, I wasn't clear. I hadn't seen any replies to my original patch on the list, but
I saw a rather long time of receiving nothing from linux-nvme, so I wasn't sure if my original mail went anywhere. But I found it on the archive.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/nvme/host/core.c +0 -0
@@ -2027,6 +2027,12 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
        if (ret)
                return ret;
 
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

drivers/nvme/host/core.c static int nvme_update_ns_info_block(struct nvme_ns *ns, · line 2033

Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+       if (id->ncap == 0) {
+               /* namespace not allocated or attached */
+               info->is_removed = true;
+               return -ENODEV;

The concept sounds good, though you are leaking 'id' in the new error case.

I will send a v2 to correct this, Maurizio has fixed the other caller.

-Ewan