Fix usage of irq affinity_hint
Subsystems (5)
Haiyang Zhang
maintainercover Fix usage of irq affinity_hint
Fix 2 bugs of irq affinity_hint usage in the mana driver
Haiyang Zhang (2):
net: mana: Fix hint value before free irq
net: mana: Fix accessing freed irq affinity_hint
.../net/ethernet/microsoft/mana/gdma_main.c | 45 +++++++++++-------- include/net/mana/gdma.h | 1 + 2 files changed, 28 insertions(+), 18 deletions(-)
Signature · 1 line
2.25.1
Haiyang Zhang
maintainerPATCH 1/2 net: mana: Fix hint value before free irq
Need to clear affinity_hint before free_irq(), otherwise there is a one-time warning and stack trace during module unloading.
To fix this bug, set affinity_hint to NULL before free as required.
--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma_main.c +5 -0
@@ -1297,6 +1297,8 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j]; + +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }
@@ -1324,6 +1326,9 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)continue ; gic = &gc->irq_contexts[i]; + +/* Need to clear the hint before free_irq */ +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }
Signature · 1 line
2.25.1
Re: [PATCH net, 1/2] net: mana: Fix hint value before free irq
Quoted 2 lines · Haiyang Zhang
Need to clear
affinity_hintbeforefree_irq(), otherwise there is a one-time warning and stack trace during module unloading.
Please add this warning and stack trace to the commit message.
Thanks
Quoted 36 lines
To fix this bug, set
affinity_hintto NULL before free as required.Fixes 71fa6887eeca ("net: mana: Assign interrupts to CPUs based on NUMA nodes")Signed-off-by Haiyang Zhang <[email protected]>--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/net/ethernet/microsoft/mana/gdma_main.c +5
-0@@ -1297,6 +1297,8 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j]; + +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }@@ -1324,6 +1326,9 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)continue ; gic = &gc->irq_contexts[i]; + +/* Need to clear the hint before free_irq */ +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }Signature · 1 line
2.25.1
Haiyang Zhang
maintainerRE: [PATCH net, 1/2] net: mana: Fix hint value before free irq
Quoted 12 lines
-----Original Message-----
From: Leon Romanovsky <[email protected]>
Sent: Sunday, January 29, 2023 4:28 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Paul Rosswurm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net, 1/2] net: mana: Fix hint value before free irq
Quoted 5 lines
Quoted 2 lines · Haiyang Zhang
Need to clear
affinity_hintbeforefree_irq(), otherwise there is a one-time warning and stack trace during module unloading.Please add this warning and stack trace to the commit message.
Will do.
Thanks,
- Haiyang
RE: [PATCH net, 1/2] net: mana: Fix hint value before free irq
In reply to Haiyang ZhangFrom: LKML haiyangz <[email protected]> On Behalf Of Haiyang Zhang Sent: Thursday, January 26, 2023 1:05 PM
Quoted 38 lines
Need to clear
affinity_hintbeforefree_irq(), otherwise there is a one-time warning and stack trace during module unloading.To fix this bug, set
affinity_hintto NULL before free as required.Fixes 71fa6887eeca ("net: mana: Assign interrupts to CPUs based on NUMA nodes")Signed-off-by Haiyang Zhang <[email protected]>--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/net/ethernet/microsoft/mana/gdma_main.c +5
-1@@ -1297,6 +1297,8 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j]; + +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }@@ -1324,6 +1326,9 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)continue ; gic = &gc->irq_contexts[i]; + +/* Need to clear the hint before free_irq */ +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); } --
Quoted 1 line
2.25.1
I think this patch should be folded into the second patch of this series. While this patch makes the warning go away, it doesn't really solve any problems by itself. It just papers over the warning. My first reaction on seeing this patch is that the warning exists because the memory for the mask likely had been incorrectly managed, which is exactly the case. Since this patch doesn't really fix a problem on its own, I'd say merge it into the second patch.
That's just my $.02. If you really want to keep it separate, that's not a blocker for me.
Michael
Haiyang Zhang
maintainerRE: [PATCH net, 1/2] net: mana: Fix hint value before free irq
Quoted 65 lines
-----Original Message-----
From: Michael Kelley (LINUX) <[email protected]>
Sent: Sunday, January 29, 2023 9:26 AM
To: Haiyang Zhang <[email protected]>; [email protected]; [email protected]
Cc: Haiyang Zhang <[email protected]>; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Paul Rosswurm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH net, 1/2] net: mana: Fix hint value before free irqFrom: LKML haiyangz <[email protected]> On Behalf Of Haiyang Zhang
Sent: Thursday, January 26, 2023 1:05 PMQuoted 8 lines
Need to clear
affinity_hintbeforefree_irq(), otherwise there is a one-time warning and stack trace during module unloading.To fix this bug, set
affinity_hintto NULL before free as required.Fixes 71fa6887eeca ("net: mana: Assign interrupts to CPUs based on NUMAnodes")
drivers/net/ethernet/microsoft/mana/gdma_main.c static int mana_gd_setup_irqs(struct pci_dev
Quoted 11 lines
Signed-off-by Haiyang Zhang <[email protected]>--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0@@ -1297,6 +1297,8 @@ static int mana_gd_setup_irqs(struct pci_dev*pdev)
drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_remove_irqs(struct pci_dev
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-0for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j]; + +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); }@@ -1324,6 +1326,9 @@ static void mana_gd_remove_irqs(struct pci_dev*pdev)
Quoted 11 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-1continue ; gic = &gc->irq_contexts[i]; + +/* Need to clear the hint before free_irq */ +irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); } --2.25.1
I think this patch should be folded into the second patch of this series. While this patch makes the warning go away, it doesn't really solve any problems by itself. It just papers over the warning. My first reaction on seeing this patch is that the warning exists because the memory for the mask likely had been incorrectly managed, which is exactly the case. Since this patch doesn't really fix a problem on its own, I'd say merge it into the second patch.
Quoted 2 lines
That's just my $.02. If you really want to keep it separate, that's not a blocker for me.
Will do.
Thanks,
- Haiyang
Haiyang Zhang
maintainerPATCH 2/2 net: mana: Fix accessing freed irq affinity_hint
In reply to Haiyang ZhangAfter calling irq_set_affinity_and_hint(), the cpumask pointer is saved in desc->affinity_hint, and will be used later when reading /proc/irq/<num>/affinity_hint. So the cpumask variable needs to be allocated per irq, and available until freeing the irq. Otherwise, we are accessing freed memory when reading the affinity_hint file.
To fix the bug, allocate the cpumask per irq, and free it just before freeing the irq.
--- .../net/ethernet/microsoft/mana/gdma_main.c | 40 ++++++++++--------- include/net/mana/gdma.h | 1 + 2 files changed, 23 insertions(+), 18 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +22 -18
@@ -1219,7 +1219,6 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)struct gdma_irq_context *gic;unsigned int max_irqs;u16 *cpus; -cpumask_var_t req_mask;int nvec, irq;int err, i =0 , j;
@@ -1240,25 +1239,26 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) goto free_irq_vector; } -if (!zalloc_cpumask_var (&req_mask,GFP_KERNEL )) { - err = -ENOMEM ; - goto free_irq; - } - cpus =kcalloc (nvec,sizeof (*cpus),GFP_KERNEL );if (!cpus) { err = -ENOMEM ; - goto free_mask; + goto free_gic; }for (i =0 ; i < nvec; i++) cpus[i] =cpumask_local_spread (i, gc->numa_node);for (i =0 ; i < nvec; i++) { -cpumask_set_cpu (cpus[i], req_mask); gic = &gc->irq_contexts[i]; gic->handler =NULL ; gic->arg =NULL ; +if (!zalloc_cpumask_var (&gic->cpu_hint,GFP_KERNEL )) { + err = -ENOMEM ; + goto free_irq; + } + +cpumask_set_cpu (cpus[i], gic->cpu_hint); +if (!i)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_hwc@pci:%s" ,pci_name (pdev));
@@ -1269,17 +1269,18 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) irq =pci_irq_vector (pdev, i);if (irq <0 ) { err = irq; - goto free_mask; +free_cpumask_var (gic->cpu_hint); + goto free_irq; } err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_mask; -irq_set_affinity_and_hint (irq, req_mask); -cpumask_clear (req_mask); +if (err) { +free_cpumask_var (gic->cpu_hint); + goto free_irq; + } + +irq_set_affinity_and_hint (irq, gic->cpu_hint); } -free_cpumask_var (req_mask); -kfree (cpus); err =mana_gd_alloc_res_map (nvec, &gc->msix_resource);if (err)
@@ -1288,20 +1289,22 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) gc->max_num_msix = nvec; gc->num_msix_usable = nvec; +kfree (cpus);return 0 ; -free_mask: -free_cpumask_var (req_mask); -kfree (cpus); free_irq:for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j];irq_update_affinity_hint (irq,NULL ); +free_cpumask_var (gic->cpu_hint);free_irq (irq, gic); } +kfree (cpus); + +free_gic:kfree (gc->irq_contexts); gc->irq_contexts =NULL ; free_irq_vector:
@@ -1329,6 +1332,7 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)/* Need to clear the hint before free_irq */ irq_update_affinity_hint (irq,NULL ); +free_cpumask_var (gic->cpu_hint);free_irq (irq, gic); }
include/net/mana/gdma.h +1 -0
@@ -342,6 +342,7 @@ struct gdma_irq_context {void (*handler)(void *arg);void *arg;char name[MANA_IRQ_NAME_SZ ]; +cpumask_var_t cpu_hint; };struct gdma_context {
Signature · 1 line
2.25.1
Re: [PATCH net, 2/2] net: mana: Fix accessing freed irq affinity_hint
Quoted 9 lines · Haiyang Zhang
After calling
irq_set_affinity_and_hint(), the cpumask pointer is saved indesc->affinity_hint, and will be used later when reading /proc/irq/<num>/affinity_hint. So the cpumask variable needs to be allocated per irq, and available until freeing the irq. Otherwise, we are accessing freed memory when reading theaffinity_hintfile.To fix the bug, allocate the cpumask per irq, and free it just before freeing the irq.
Quoted 8 lines
Fixes 71fa6887eeca ("net: mana: Assign interrupts to CPUs based on NUMA nodes")Signed-off-by Haiyang Zhang <[email protected]>--- .../net/ethernet/microsoft/mana/gdma_main.c | 40 ++++++++++--------- include/net/mana/gdma.h | 1 + 2 files changed, 23 insertions(+), 18 deletions(-)
Thanks,
RE: [PATCH net, 2/2] net: mana: Fix accessing freed irq affinity_hint
In reply to Haiyang ZhangFrom: LKML haiyangz <[email protected]> On Behalf Of Haiyang Zhang Sent: Thursday, January 26, 2023 1:05 PM
Quoted 9 lines
After calling
irq_set_affinity_and_hint(), the cpumask pointer is saved indesc->affinity_hint, and will be used later when reading /proc/irq/<num>/affinity_hint. So the cpumask variable needs to be allocated per irq, and available until freeing the irq. Otherwise, we are accessing freed memory when reading theaffinity_hintfile.To fix the bug, allocate the cpumask per irq, and free it just before freeing the irq.
Since the cpumask being passed to irq_set_affinity_and_hint() always contains exactly one CPU, the code can be considerably simplified by using the pre-calculated and persistent masks available as cpumask_of(cpu). All allocation of cpumasks in this code goes away, and you can set the affinity_hint to NULL in the cleanup and remove paths without having to free any masks.
Michael
Quoted 131 lines
Fixes 71fa6887eeca ("net: mana: Assign interrupts to CPUs based on NUMA nodes")Signed-off-by Haiyang Zhang <[email protected]>--- .../net/ethernet/microsoft/mana/gdma_main.c | 40 ++++++++++--------- include/net/mana/gdma.h | 1 + 2 files changed, 23 insertions(+), 18 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +8
-8@@ -1219,7 +1219,6 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)struct gdma_irq_context *gic;unsigned int max_irqs;u16 *cpus; -cpumask_var_t req_mask;int nvec, irq;int err, i =0 , j;@@ -1240,25 +1239,26 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) goto free_irq_vector; } -if (!zalloc_cpumask_var (&req_mask,GFP_KERNEL )) { - err = -ENOMEM ; - goto free_irq; - } - cpus =kcalloc (nvec,sizeof (*cpus),GFP_KERNEL );if (!cpus) { err = -ENOMEM ; - goto free_mask; + goto free_gic; }for (i =0 ; i < nvec; i++) cpus[i] =cpumask_local_spread (i, gc->numa_node);for (i =0 ; i < nvec; i++) { -cpumask_set_cpu (cpus[i], req_mask); gic = &gc->irq_contexts[i]; gic->handler =NULL ; gic->arg =NULL ; +if (!zalloc_cpumask_var (&gic->cpu_hint,GFP_KERNEL )) { + err = -ENOMEM ; + goto free_irq; + } + +cpumask_set_cpu (cpus[i], gic->cpu_hint); +if (!i) snprintf(gic->name, MANA_IRQ_NAME_SZ, "mana_hwc@pci:%s",drivers/net/ethernet/microsoft/mana/gdma_main.c +14
-10pci_name(pdev));@@ -1269,17 +1269,18 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) irq =pci_irq_vector (pdev, i);if (irq <0 ) { err = irq; - goto free_mask; +free_cpumask_var (gic->cpu_hint); + goto free_irq; } err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_mask; -irq_set_affinity_and_hint (irq, req_mask); -cpumask_clear (req_mask); +if (err) { +free_cpumask_var (gic->cpu_hint); + goto free_irq; + } + +irq_set_affinity_and_hint (irq, gic->cpu_hint); } -free_cpumask_var (req_mask); -kfree (cpus); err =mana_gd_alloc_res_map (nvec, &gc->msix_resource);if (err)@@ -1288,20 +1289,22 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) gc->max_num_msix = nvec; gc->num_msix_usable = nvec; +kfree (cpus);return 0 ; -free_mask: -free_cpumask_var (req_mask); -kfree (cpus); free_irq:for (j = i -1 ; j >=0 ; j--) { irq =pci_irq_vector (pdev, j); gic = &gc->irq_contexts[j];irq_update_affinity_hint (irq,NULL ); +free_cpumask_var (gic->cpu_hint);free_irq (irq, gic); } +kfree (cpus); + +free_gic:kfree (gc->irq_contexts); gc->irq_contexts =NULL ; free_irq_vector:@@ -1329,6 +1332,7 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)/* Need to clear the hint before free_irq */ irq_update_affinity_hint (irq,NULL ); +free_cpumask_var (gic->cpu_hint);free_irq (irq, gic); }include/net/mana/gdma.h +1
-1@@ -342,6 +342,7 @@ struct gdma_irq_context {void (*handler)(void *arg);void *arg;char name[MANA_IRQ_NAME_SZ ]; +cpumask_var_t cpu_hint; };struct gdma_context { --2.25.1
Haiyang Zhang
maintainerRE: [PATCH net, 2/2] net: mana: Fix accessing freed irq affinity_hint
Quoted 25 lines
-----Original Message-----
From: Michael Kelley (LINUX) <[email protected]>
Sent: Sunday, January 29, 2023 9:27 AM
To: Haiyang Zhang <[email protected]>; [email protected]; [email protected]
Cc: Haiyang Zhang <[email protected]>; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Paul Rosswurm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH net, 2/2] net: mana: Fix accessing freed irqaffinity_hintFrom: LKML haiyangz <[email protected]> On Behalf Of Haiyang Zhang
Sent: Thursday, January 26, 2023 1:05 PMQuoted 9 lines
After calling
irq_set_affinity_and_hint(), the cpumask pointer is saved indesc->affinity_hint, and will be used later when reading /proc/irq/<num>/affinity_hint. So the cpumask variable needs to be allocated per irq, and available until freeing the irq. Otherwise, we are accessing freed memory when reading theaffinity_hintfile.To fix the bug, allocate the cpumask per irq, and free it just before freeing the irq.
Quoted 7 lines
Since the cpumask being passed to
irq_set_affinity_and_hint()always contains exactly one CPU, the code can be considerably simplified by using the pre-calculated and persistent masks available ascpumask_of(cpu). All allocation of cpumasks in this code goes away, and you can set theaffinity_hintto NULL in the cleanup and remove paths without having to free any masks.
Great idea!
Will update the patch accordingly.
- Haiyang
Haiyang Zhang
maintainerRE: [PATCH net, 2/2] net: mana: Fix accessing freed irq affinity_hint
Quoted 49 lines
-----Original Message-----
From: Haiyang Zhang <[email protected]>
Sent: Sunday, January 29, 2023 1:51 PM
To: Michael Kelley (LINUX) <[email protected]>; linux- [email protected]; [email protected]
Cc: Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Paul Rosswurm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH net, 2/2] net: mana: Fix accessing freed irqaffinity_hintQuoted 4 lines
-----Original Message-----
From: Michael Kelley (LINUX) <[email protected]>
Sent: Sunday, January 29, 2023 9:27 AM
To: Haiyang Zhang <[email protected]>; linux-Quoted 3 lines
[email protected]
Cc: Haiyang Zhang <[email protected]>; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; PaulRosswurm
Quoted 2 lines
<[email protected]>; [email protected]; [email protected]; [email protected]; [email protected];
Quoted 3 lines
Subject: RE: [PATCH net, 2/2] net: mana: Fix accessing freed irq
affinity_hintFrom: LKML haiyangz <[email protected]> On Behalf Of Haiyang
Zhang
Quoted 11 lines
Sent: Thursday, January 26, 2023 1:05 PM
Quoted 9 lines
After calling
irq_set_affinity_and_hint(), the cpumask pointer is saved indesc->affinity_hint, and will be used later when reading /proc/irq/<num>/affinity_hint. So the cpumask variable needs to be allocated per irq, and available until freeing the irq. Otherwise, we are accessing freed memory when reading theaffinity_hintfile.To fix the bug, allocate the cpumask per irq, and free it just before freeing the irq.
Quoted 7 lines
Since the cpumask being passed to
irq_set_affinity_and_hint()always contains exactly one CPU, the code can be considerably simplified by using the pre-calculated and persistent masks available ascpumask_of(cpu). All allocation of cpumasks in this code goes away, and you can set theaffinity_hintto NULL in the cleanup and remove paths without having to free any masks.
Quoted 2 lines
Great idea!
Will update the patch accordingly.
Also, I saw this alloc isn't necessary either:
cpus = kcalloc(nvec, sizeof(*cpus), GFP_KERNEL);
We can simply use the return from cpumask_local_spread() without saving all cpu numbers in a tmp array.
I will clean this up too :)
Thanks,
- Haiyang