lpfc: fixup out-of-bounds access during CPU hotplug
Subsystems (2)
PATCH 1/1 lpfc: fixup out-of-bounds access during CPU hotplug
The lpfc driver allocates a cpu_map based on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.
--- drivers/scsi/lpfc/lpfc_scsi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_scsi.c +2 -1
@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,int tag;struct fcp_cmd_rsp_buf *tmp =NULL ; - cpu =raw_smp_processor_id (); + cpu =min_t (u32,raw_smp_processor_id (), + phba->sli4_hba.num_possible_cpu);if (cmnd && phba->cfg_fcp_io_sched ==LPFC_FCP_SCHED_BY_HDWQ ) { tag =blk_mq_unique_tag (cmnd->request); idx =blk_mq_unique_tag_to_hwq (tag);
Signature · 1 line
2.16.4
Hi Hannes,
Quoted 7 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>
There are a few more places I think the check is needed:
lpfc_nvme_fcp_io_submit(), lpfc_nvmet_ctxbuf_post(), lpfc_nvmet_xmt_fcp_op(), lpfc_nvmet_rcv_unsol_abort(), lpfc_nvme_io_cmd_wqe_cmpl(), lpfc_nvmet_unsol_fcp_buffer(), lpfc_scsi_cmd_iocb_cmpl()
At least all of them seem to use the CPU id to access some array. I suggest we review all those places as well.
Thanks,
Daniel
Hi Hannes,
Quoted 16 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/scsi/lpfc/lpfc_scsi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0
drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 646
Quoted 6 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,int tag;struct fcp_cmd_rsp_buf *tmp =NULL ; - cpu =raw_smp_processor_id (); + cpu =min_t (u32,raw_smp_processor_id (), + phba->sli4_hba.num_possible_cpu);
The index is limited by phba->cfg_hdw_queue and not the number of CPUs.
Thanks,
Daniel
Quoted 22 lines · Daniel Wagner
Hi Hannes,
Quoted 19 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/scsi/lpfc/lpfc_scsi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,int tag;struct fcp_cmd_rsp_buf *tmp =NULL ;
Quoted 6 lines
drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 646
Quoted 3 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1- cpu =raw_smp_processor_id (); + cpu =min_t (u32,raw_smp_processor_id (), + phba->sli4_hba.num_possible_cpu);The index is limited by
phba->cfg_hdw_queueand not the number of CPUs.
Nope.
phba->sli4_hba.cpu_map = kcalloc(phba->sli4_hba.num_possible_cpu,
sizeof(struct lpfc_vector_map_info), GFP_KERNEL);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 247165 (AG München), GF: Felix Imendörffer
Quoted 4 lines
phba->sli4_hba.cpu_map= kcalloc(phba->sli4_hba.num_possible_cpu,sizeof(struct lpfc_vector_map_info), GFP_KERNEL);
Indeed, I didn't realize this.
Thanks,
Daniel
Quoted 16 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/scsi/lpfc/lpfc_scsi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0
drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp, · line 649
Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp,int tag;struct fcp_cmd_rsp_buf *tmp =NULL ; - cpu =raw_smp_processor_id (); + cpu =min_t (u32,raw_smp_processor_id (), + phba->sli4_hba.num_possible_cpu);if (cmnd && phba->cfg_fcp_io_sched ==LPFC_FCP_SCHED_BY_HDWQ ) { tag =blk_mq_unique_tag (cmnd->request); idx =blk_mq_unique_tag_to_hwq (tag);
This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.
I am curious, how if a cpu is hot removed - how num_possible dynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number than num_possible ?
-- james
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/scsi/lpfc/lpfc_scsi.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, struct · line 649
Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, structlpfc_nodelist *ndlp, int tag; struct fcp_cmd_rsp_buf *tmp = NULL; - cpu = raw_smp_processor_id(); + cpu = min_t(u32, raw_smp_processor_id(), + phba->sli4_hba.num_possible_cpu); if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) { tag = blk_mq_unique_tag(cmnd->request); idx = blk_mq_unique_tag_to_hwq(tag);This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.
Quoted 2 lines
I am curious, how if a cpu is hot removed - how
num_possibledynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number thannum_possible?
Actually, the behaviour of num_possible_cpus() under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Quoted 38 lines · Hannes Reinecke
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/scsi/lpfc/lpfc_scsi.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, · line 649
Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba,struct lpfc_nodelist *ndlp, int tag; struct fcp_cmd_rsp_buf *tmp = NULL; - cpu = raw_smp_processor_id(); + cpu = min_t(u32, raw_smp_processor_id(), + phba->sli4_hba.num_possible_cpu); if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) { tag = blk_mq_unique_tag(cmnd->request); idx = blk_mq_unique_tag_to_hwq(tag);This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.
Quoted 2 lines
I am curious, how if a cpu is hot removed - how
num_possibledynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number thannum_possible?Actually, the behaviour of
num_possible_cpus()under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.
Quoted 3 lines
Cheers,
Hannes
That sure seems to be at odds with this comment from include/linux/cpumask.h:
* The cpu_possible_mask is fixed at boot time, as the set of CPU id's * that it is possible might ever be plugged in at anytime during the * life of that system boot. The cpu_present_mask is dynamic(*), * representing which CPUs are currently plugged in. And
* cpu_online_mask is the dynamic subset of cpu_present_mask, * indicating those CPUs available for scheduling.
and
#define num_possible_cpus() cpumask_weight(cpu_possible_mask)
-- james
Quoted 54 lines · James Smart
Quoted 38 lines · Hannes Reinecke
Quoted 29 lines · James Smart
Quoted 16 lines · Hannes Reinecke
The lpfc driver allocates a
cpu_mapbased on the number of possible cpus during startup. If a CPU hotplug occurs the number of CPUs might change, causing an out-of-bounds access when trying to lookup the hardware index for a given CPU.Suggested-by Daniel Wagner <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/scsi/lpfc/lpfc_scsi.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/scsi/lpfc/lpfc_scsi.c +0
-0drivers/scsi/lpfc/lpfc_scsi.c lpfc_get_scsi_buf_s4(struct lpfc_hba *phba, · line 649
Quoted 9 lines
drivers/scsi/lpfc/lpfc_scsi.c +2
-1@@ -642,7 +642,8 @@ lpfc_get_scsi_buf_s4(struct lpfc_hba *phba,struct lpfc_nodelist *ndlp, int tag; struct fcp_cmd_rsp_buf *tmp = NULL; - cpu = raw_smp_processor_id(); + cpu = min_t(u32, raw_smp_processor_id(), + phba->sli4_hba.num_possible_cpu); if (cmnd && phba->cfg_fcp_io_sched == LPFC_FCP_SCHED_BY_HDWQ) { tag = blk_mq_unique_tag(cmnd->request); idx = blk_mq_unique_tag_to_hwq(tag);This should be unnecessary with the lpfc 12.6.0.1 and 12.6.0.2 patches that tie into cpu onling/offlining and cpu hot add.
Quoted 2 lines
I am curious, how if a cpu is hot removed - how
num_possibledynamically changes (to a lower value ?) such that a thread can be running on a cpu that returns a higher cpu number thannum_possible?Actually, the behaviour of
num_possible_cpus()under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number. Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug.
Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.Quoted 3 lines
Cheers,
Hannes
That sure seems to be at odds with this comment from
include/linux/cpumask.h:* The
cpu_possible_maskis fixed at boot time, as the set of CPU id's * that it is possible might ever be plugged in at anytime during the * life of that system boot. Thecpu_present_maskisdynamic(*), * representing which CPUs are currently plugged in. And
*cpu_online_maskis the dynamic subset ofcpu_present_mask, * indicating those CPUs available for scheduling.
Quoted 6 lines
and
#define num_possible_cpus()cpumask_weight(cpu_possible_mask)-- james
Although... num_possible_cpus() shouldn't change, but I guess a cpu id can be greater than the possible cpu count.
But there is this as well - also from include/linux/cpumask.h:
* If HOTPLUG is enabled, then cpu_possible_mask is forced to have * all NR_CPUS bits set, otherwise it is just the set of CPUs that * ACPI reports present at boot.
-- james
Quoted 8 lines · James Smart
Although...
num_possible_cpus()shouldn't change, but I guess a cpu id
can be greater than the possible cpu count.But there is this as well - also from
include/linux/cpumask.h:* If HOTPLUG is enabled, then cpu_possible_mask is forced to have * all NR_CPUS bits set, otherwise it is just the set of CPUs that * ACPI reports present at boot.
That's probably an x86'ism... on powerpc the mask is set based on what's really possible and it may not be all bits
Cheers,
Ben.
Quoted 28 lines · James Smart
Quoted 14 lines
Actually, the behaviour of
num_possible_cpus()under cpu hotplug seems to be implementation-specific.
One might want to set it to the max value at all times, which has the drawback that you'd need to scale per-cpu values with that number, leading to a massive memory overhead as only very few installation will ever reach that number.
Others might want to set to the max value at the current configuration, implying that it might change under CPU hotplug. Which seems to be the case for PowerPC, which was the case which triggered the issue at hand.
But maybe it was a plain bug in the CPU hotplug implementation; one should be asking BenH for details here.Quoted 3 lines
Cheers,
Hannes
That sure seems to be at odds with this comment from
include/linux/cpumask.h:* The cpu_possible_mask is fixed at boot time, as the set of CPU id's * that it is possible might ever be plugged in at anytime during the * life of that system boot. The cpu_present_mask is dynamic(*), * representing which CPUs are currently plugged in. And * cpu_online_mask is the dynamic subset of cpu_present_mask, * indicating those CPUs available for scheduling.
Quoted 2 lines
and
#definenum_possible_cpus()cpumask_weight(cpu_possible_mask)
Right, it should be fixed at boot. Where do you see powerpc change it ?
Cheers,
Ben.