Skip to contentSkip to conversation

lpfc: fixup out-of-bounds access during CPU hotplug

PATCH In review Hannes Reinecke 4 participants 11 messages Nov 18, 2019 – Nov 20, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lpfc: fixup out-of-bounds access during CPU hotplug

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
 drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
index ba26df90a36a..2380452a8efd 100644
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,
 	int tag;
 	struct fcp_cmd_rsp_buf *tmp = NULL;
 
-	cpu = raw_smp_processor_id();
+	cpu = min_t(u32, raw_smp_processor_id(),
+		    phba->sli4_hba.num_possible_cpu);
 	if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) {
 		tag = blk_mq_unique_tag(cmnd->request);
 		idx = blk_mq_unique_tag_to_hwq(tag);
Signature · 1 line
2.16.4
#1 lore raw

Hi Hannes,

Quoted 7 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

There are a few more places I think the check is needed:

lpfc_nvme_fcp_io_submit(), lpfc_nvmet_ctxbuf_post(), lpfc_nvmet_xmt_fcp_op(), lpfc_nvmet_rcv_unsol_abort(), lpfc_nvme_io_cmd_wqe_cmpl(), lpfc_nvmet_unsol_fcp_buffer(), lpfc_scsi_cmd_iocb_cmpl()

At least all of them seem to use the CPU id to access some array. I suggest we review all those places as well.

Thanks,
Daniel

#2 lore raw
In reply to Hannes Reinecke

Hi Hannes,

Quoted 16 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
 drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 646

Quoted 6 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,
 	int tag;
 	struct fcp_cmd_rsp_buf *tmp = NULL;
 
-	cpu = raw_smp_processor_id();
+	cpu = min_t(u32, raw_smp_processor_id(),
+		    phba->sli4_hba.num_possible_cpu);

The index is limited by phba->cfg_hdw_queue and not the number of CPUs.

Thanks,
Daniel

#3 lore raw
Quoted 22 lines · Daniel Wagner

Hi Hannes,

Quoted 19 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
 drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,
 	int tag;
 	struct fcp_cmd_rsp_buf *tmp = NULL;
 
Quoted 6 lines

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 646

Quoted 3 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
-	cpu = raw_smp_processor_id();
+	cpu = min_t(u32, raw_smp_processor_id(),
+		    phba->sli4_hba.num_possible_cpu);

The index is limited by phba->cfg_hdw_queue and not the number of CPUs.

Nope.

phba->sli4_hba.cpu_map = kcalloc(phba->sli4_hba.num_possible_cpu,

				sizeof(struct lpfc_vector_map_info),
				GFP_KERNEL);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      Teamlead Storage & Networking
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 247165 (AG München), GF: Felix Imendörffer
#4 lore raw
Quoted 4 lines

phba->sli4_hba.cpu_map = kcalloc(phba->sli4_hba.num_possible_cpu,

				sizeof(struct lpfc_vector_map_info),
				GFP_KERNEL);

Indeed, I didn't realize this.

Thanks,
Daniel

#5 lore raw
In reply to Hannes Reinecke
Quoted 16 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
  drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 649

Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,
  	int tag;
  	struct fcp_cmd_rsp_buf *tmp = NULL;
 
-	cpu = raw_smp_processor_id();
+	cpu = min_t(u32, raw_smp_processor_id(),
+		    phba->sli4_hba.num_possible_cpu);
  	if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) {
  		tag = blk_mq_unique_tag(cmnd->request);
  		idx = blk_mq_unique_tag_to_hwq(tag);

This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.

I am curious, how if a cpu is hot removed - how num_possible dynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number than num_possible ?

-- james

#6 lore raw
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
  drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct · line 649

Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct 
 lpfc_nodelist *ndlp,
       int tag;
       struct fcp_cmd_rsp_buf *tmp = NULL;
-    cpu = raw_smp_processor_id();
+    cpu = min_t(u32, raw_smp_processor_id(),
+            phba->sli4_hba.num_possible_cpu);
       if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) {
           tag = blk_mq_unique_tag(cmnd->request);
           idx = blk_mq_unique_tag_to_hwq(tag);

This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.

Quoted 2 lines

I am curious, how if a cpu is hot removed - how num_possible dynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number than num_possible ?

Actually, the behaviour of num_possible_cpus() under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
[email protected]                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
#7 lore raw
Quoted 38 lines · Hannes Reinecke
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
  drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, · line 649

Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, 
 struct lpfc_nodelist *ndlp,
       int tag;
       struct fcp_cmd_rsp_buf *tmp = NULL;
-    cpu = raw_smp_processor_id();
+    cpu = min_t(u32, raw_smp_processor_id(),
+            phba->sli4_hba.num_possible_cpu);
       if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) {
           tag = blk_mq_unique_tag(cmnd->request);
           idx = blk_mq_unique_tag_to_hwq(tag);

This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.

Quoted 2 lines

I am curious, how if a cpu is hot removed - how num_possible dynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number than num_possible ?

Actually, the behaviour of num_possible_cpus() under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.

Quoted 3 lines

Cheers,

Hannes

That sure seems to be at odds with this comment from include/linux/cpumask.h:

 *  The cpu_possible_mask is fixed at boot time, as the set of CPU id's *  that it is possible might ever be plugged in at anytime during the *  life of that system boot.  The cpu_present_mask is dynamic(*), *  representing which CPUs are currently plugged in.  And
 *  cpu_online_mask is the dynamic subset of cpu_present_mask, *  indicating those CPUs available for scheduling.

and
#define num_possible_cpus()     cpumask_weight(cpu_possible_mask)

-- james

#8 lore raw
Quoted 54 lines · James Smart
Quoted 38 lines · Hannes Reinecke
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke

The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.

---
  drivers/scsi/lpfc/lpfc_scsi.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +0 -0
index ba26df90a36a..2380452a8efd 100644

drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, · line 649

Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, 
 struct lpfc_nodelist *ndlp,
       int tag;
       struct fcp_cmd_rsp_buf *tmp = NULL;
-    cpu = raw_smp_processor_id();
+    cpu = min_t(u32, raw_smp_processor_id(),
+            phba->sli4_hba.num_possible_cpu);
       if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) {
           tag = blk_mq_unique_tag(cmnd->request);
           idx = blk_mq_unique_tag_to_hwq(tag);

This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.

Quoted 2 lines

I am curious, how if a cpu is hot removed - how num_possible dynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number than num_possible ?

Actually, the behaviour of num_possible_cpus() under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.

Quoted 3 lines

Cheers,

Hannes

That sure seems to be at odds with this comment from include/linux/cpumask.h:

 *  The cpu_possible_mask is fixed at boot time, as the set of CPU id's *  that it is possible might ever be plugged in at anytime during the *  life of that system boot.  The cpu_present_mask is dynamic(*), *  representing which CPUs are currently plugged in.  And
 *  cpu_online_mask is the dynamic subset of cpu_present_mask, *  indicating those CPUs available for scheduling.

Quoted 6 lines

and
#define num_possible_cpus()     cpumask_weight(cpu_possible_mask)

-- james

Although... num_possible_cpus() shouldn't change, but I guess a cpu id can be greater than the possible cpu count.

But there is this as well - also from include/linux/cpumask.h:
 *  If HOTPLUG is enabled, then cpu_possible_mask is forced to have *  all NR_CPUS bits set, otherwise it is just the set of CPUs that *  ACPI reports present at boot.

-- james

#9 lore raw
Quoted 8 lines · James Smart

Although... num_possible_cpus() shouldn't change, but I guess a cpu id
can be greater than the possible cpu count.

But there is this as well - also from include/linux/cpumask.h:

  *  If HOTPLUG is enabled, then cpu_possible_mask is forced to have
  *  all NR_CPUS bits set, otherwise it is just the set of CPUs that
  *  ACPI reports present at boot.

That's probably an x86'ism... on powerpc the mask is set based on what's really possible and it may not be all bits

Cheers,
Ben.

#10 lore raw
In reply to James Smart
Quoted 28 lines · James Smart
Quoted 14 lines

Actually, the behaviour of num_possible_cpus() under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number.
Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug. Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.

Quoted 3 lines

Cheers,

Hannes

That sure seems to be at odds with this comment from include/linux/cpumask.h:

  *  The cpu_possible_mask is fixed at boot time, as the set of CPU id's
  *  that it is possible might ever be plugged in at anytime during the
  *  life of that system boot.  The cpu_present_mask is dynamic(*),
  *  representing which CPUs are currently plugged in.  And
  *  cpu_online_mask is the dynamic subset of cpu_present_mask,
  *  indicating those CPUs available for scheduling.
Quoted 2 lines

and
#define num_possible_cpus() cpumask_weight(cpu_possible_mask)

Right, it should be fixed at boot. Where do you see powerpc change it ?

Cheers,
Ben.