Allow dyn MSI-X vector allocation of MANA
Subsystems (7)
v4 cover Allow dyn MSI-X vector allocation of MANA
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports pci_msix_prepare_desc() in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.
The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the MSI_FLAG_PCI_MSIX_ALLOC_DYN flag and using the pci_msix_prepare_desc() exported in first patch.
The third patch adds a detailed description of the irq_setup(), to help understand the function design better.
The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in irq_setup() to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.
The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
---
Change in v4
* add a patch describing the functionality of irq_setup() through a comment
* In irq_setup(), avoid using a label next_cpumask:
* modify the changes in MANA patch about restructuring the error handling path in mana_gd_setup_dyn_irqs()
* modify the mana_gd_setup_irqs() to simplify handling around start_irq_index
* add warning if an invalid gic is returned
* place the xa_destroy() cleanup in mana_gd_remove
---
Changes in v3
* split the 3rd patch into preparation patch around irq_setup() and changes in mana driver to allow dynamic IRQ allocation
* Add arm64 support for dynamic MSI-X allocation in pci_hyperv controller
---
Changes in v2
* split the first patch into two(exporting the preapre_desc func and using the function and flag in pci-hyperv)
* replace 'pci vectors' by 'MSI-X vectors'
* Change the cover letter description to align with changes made
---
Shradha Gupta (5):
PCI/MSI: Export pci_msix_prepare_desc() for dynamic MSI-X allocations
PCI: hv: Allow dynamic MSI-X vector allocation
net: mana: explain irq_setup() algorithm
net: mana: Allow irq_setup() to skip cpus for affinity
net: mana: Allocate MSI-X vectors dynamically
.../net/ethernet/microsoft/mana/gdma_main.c | 356 ++++++++++++++---- drivers/pci/controller/pci-hyperv.c | 5 +- drivers/pci/msi/irqdomain.c | 5 +- include/linux/msi.h | 2 + include/net/mana/gdma.h | 8 +- 5 files changed, 293 insertions(+), 83 deletions(-)
Signature · 1 line
2.34.1
PATCH v4 1/5 PCI/MSI: Export pci_msix_prepare_desc() for dynamic MSI-X allocations
For supporting dynamic MSI-X vector allocation by PCI controllers, enabling the flag MSI_FLAG_PCI_MSIX_ALLOC_DYN is not enough, msix_prepare_msi_desc() to prepare the MSI descriptor is also needed.
Export pci_msix_prepare_desc() to allow PCI controllers to support dynamic MSI-X vector allocation.
---
Changes in v3
* Improved the patch description by removing abbreviations
--- drivers/pci/msi/irqdomain.c | 5 +++-- include/linux/msi.h | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-)
drivers/pci/msi/irqdomain.c +3 -2
@@ -222,13 +222,14 @@ static void pci_irq_unmask_msix(struct irq_data *data)pci_msix_unmask (irq_data_get_msi_desc (data)); } -static void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, -struct msi_desc *desc) +void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, +struct msi_desc *desc) {/* Don't fiddle with preallocated MSI descriptors */ if (!desc->pci.mask_base)msix_prepare_msi_desc (to_pci_dev (desc->dev), desc); } +EXPORT_SYMBOL_GPL (pci_msix_prepare_desc);static const struct msi_domain_template pci_msix_template = { .chip = {
include/linux/msi.h +2 -0
@@ -691,6 +691,8 @@ struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode, struct irq_domain *parent);u32 pci_msi_domain_get_msi_rid (struct irq_domain *domain,struct pci_dev *pdev);struct irq_domain *pci_msi_get_device_domain (struct pci_dev *pdev); +void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, +struct msi_desc *desc);#else /* CONFIG_PCI_MSI */ static inline struct irq_domain *pci_msi_get_device_domain (struct pci_dev *pdev) {
Signature · 1 line
2.34.1
Re: [PATCH v4 1/5] PCI/MSI: Export pci_msix_prepare_desc() for dynamic MSI-X allocations
Quoted 55 lines · Shradha Gupta
For supporting dynamic MSI-X vector allocation by PCI controllers, enabling the flag
MSI_FLAG_PCI_MSIX_ALLOC_DYNis not enough,msix_prepare_msi_desc()to prepare the MSI descriptor is also needed.Export
pci_msix_prepare_desc()to allow PCI controllers to support dynamic MSI-X vector allocation.Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>Reviewed-by Thomas Gleixner <[email protected]>---
Changes in v3
* Improved the patch description by removing abbreviations--- drivers/pci/msi/irqdomain.c | 5 +++-- include/linux/msi.h | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-)drivers/pci/msi/irqdomain.c +3
-2@@ -222,13 +222,14 @@ static void pci_irq_unmask_msix(struct irq_data *data)pci_msix_unmask (irq_data_get_msi_desc (data)); } -static void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, -struct msi_desc *desc) +void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, +struct msi_desc *desc) {/* Don't fiddle with preallocated MSI descriptors */ if (!desc->pci.mask_base)msix_prepare_msi_desc (to_pci_dev (desc->dev), desc); } +EXPORT_SYMBOL_GPL (pci_msix_prepare_desc);static const struct msi_domain_template pci_msix_template = { .chip = {include/linux/msi.h +2
-0@@ -691,6 +691,8 @@ struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode, struct irq_domain *parent);u32 pci_msi_domain_get_msi_rid (struct irq_domain *domain,struct pci_dev *pdev);struct irq_domain *pci_msi_get_device_domain (struct pci_dev *pdev); +void pci_msix_prepare_desc (struct irq_domain *domain,msi_alloc_info_t *arg, +struct msi_desc *desc);#else /* CONFIG_PCI_MSI */ static inline struct irq_domain *pci_msi_get_device_domain (struct pci_dev *pdev) {Signature · 1 line
2.34.1
PATCH v4 2/5 PCI: hv: Allow dynamic MSI-X vector allocation
In reply to Shradha GuptaAllow dynamic MSI-X vector allocation for pci_hyperv PCI controller by adding support for the flag MSI_FLAG_PCI_MSIX_ALLOC_DYN and using pci_msix_prepare_desc() to prepare the MSI-X descriptors.
Feature support added for both x86 and ARM64
---
Changes in v4:
* use the same prepare_desc() callback for arm and x86
---
Changes in v3:
* Add arm64 support
---
Changes in v2:
* split the patch to keep changes in PCI and pci_hyperv controller seperate
* replace strings "pci vectors" by "MSI-X vectors"
--- drivers/pci/controller/pci-hyperv.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletions(-)
drivers/pci/controller/pci-hyperv.c +2 -1
@@ -2063,6 +2063,7 @@ static struct irq_chip hv_msi_irq_chip = {static struct msi_domain_ops hv_msi_ops = { .msi_prepare = hv_msi_prepare, .msi_free = hv_msi_free, + .prepare_desc = pci_msix_prepare_desc, };/**
@@ -2084,7 +2085,7 @@ static int hv_pcie_init_irq_domain(struct hv_pcibus_device *hbus) hbus->msi_info.ops = &hv_msi_ops; hbus->msi_info.flags = (MSI_FLAG_USE_DEF_DOM_OPS |MSI_FLAG_USE_DEF_CHIP_OPS |MSI_FLAG_MULTI_PCI_MSI | -MSI_FLAG_PCI_MSIX ); +MSI_FLAG_PCI_MSIX |MSI_FLAG_PCI_MSIX_ALLOC_DYN ); hbus->msi_info.handler =FLOW_HANDLER ; hbus->msi_info.handler_name =FLOW_NAME ; hbus->msi_info.data = hbus;
Signature · 1 line
2.34.1
Re: [PATCH v4 2/5] PCI: hv: Allow dynamic MSI-X vector allocation
Quoted 47 lines · Shradha Gupta
Allow dynamic MSI-X vector allocation for
pci_hypervPCI controller by adding support for the flagMSI_FLAG_PCI_MSIX_ALLOC_DYNand usingpci_msix_prepare_desc()to prepare the MSI-X descriptors.Feature support added for both x86 and ARM64
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>---
Changes in v4:
* use the sameprepare_desc()callback for arm and x86
---
Changes in v3:
* Add arm64 support
---
Changes in v2:
* split the patch to keep changes in PCI andpci_hypervcontroller seperate
* replace strings "pci vectors" by "MSI-X vectors"--- drivers/pci/controller/pci-hyperv.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletions(-)drivers/pci/controller/pci-hyperv.c +2
-1@@ -2063,6 +2063,7 @@ static struct irq_chip hv_msi_irq_chip = {static struct msi_domain_ops hv_msi_ops = { .msi_prepare = hv_msi_prepare, .msi_free = hv_msi_free, + .prepare_desc = pci_msix_prepare_desc, };/** @@ -2084,7 +2085,7 @@ static int hv_pcie_init_irq_domain(struct hv_pcibus_device *hbus) hbus->msi_info.ops = &hv_msi_ops; hbus->msi_info.flags = (MSI_FLAG_USE_DEF_DOM_OPS |MSI_FLAG_USE_DEF_CHIP_OPS |MSI_FLAG_MULTI_PCI_MSI | -MSI_FLAG_PCI_MSIX ); +MSI_FLAG_PCI_MSIX |MSI_FLAG_PCI_MSIX_ALLOC_DYN ); hbus->msi_info.handler =FLOW_HANDLER ; hbus->msi_info.handler_name =FLOW_NAME ; hbus->msi_info.data = hbus;Signature · 1 line
2.34.1
PATCH v4 3/5 net: mana: explain irq_setup() algorithm
In reply to Shradha GuptaCommit 91bfe210e196 ("net: mana: add a function to spread IRQs per CPUs") added the irq_setup() function that distributes IRQs on CPUs according to a tricky heuristic. The corresponding commit message explains the heuristic.
Duplicate it in the source code to make available for readers without digging git in history. Also, add more detailed explanation about how the heuristics is implemented.
--- .../net/ethernet/microsoft/mana/gdma_main.c | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma_main.c +41 -0
@@ -1288,6 +1288,47 @@ void mana_gd_free_res_map(struct gdma_resource *r) r->size =0 ; } +/* +* Spread on CPUs with the following heuristics: +* +* 1. No more than one IRQ per CPU, if possible; +* 2. NUMA locality is the second priority; +* 3. Sibling dislocality is the last priority. +* +* Let's consider this topology: +* +* Node 0 1 +* Core 0 1 2 3 +* CPU 0 1 2 3 4 5 6 7 +* +* The most performant IRQ distribution based on the above topology +* and heuristics may look like this: +* +* IRQ Nodes Cores CPUs +* 0 1 0 0-1 +* 1 1 1 2-3 +* 2 1 0 0-1 +* 3 1 1 2-3 +* 4 2 2 4-5 +* 5 2 3 6-7 +* 6 2 2 4-5 +* 7 2 3 6-7 +* +* The heuristics is implemented as follows. +* +* The outer for_each() loop resets the 'weight' to the actual number +* of CPUs in the hop. Then inner for_each() loop decrements it by the +* number of sibling groups (cores) while assigning first set of IRQs +* to each group. IRQs 0 and 1 above are distributed this way. +* +* Now, because NUMA locality is more important, we should walk the +* same set of siblings and assign 2nd set of IRQs (2 and 3), and it's +* implemented by the medium while() loop. We do like this unless the +* number of IRQs assigned on this hop will not become equal to number +* of CPUs in the hop (weight == 0). Then we switch to the next hop and +* do the same thing. +*/ +static int irq_setup (unsigned int *irqs,unsigned int len,int node) {const struct cpumask *next, *prev = cpu_none_mask;
Signature · 1 line
2.34.1
Re: [PATCH v4 3/5] net: mana: explain irq_setup() algorithm
So now git will think that you're the author of the patch.
If author and sender are different people, the first line in commit message body should state that. In this case, it should be:
From: Yury Norov <[email protected]>
Please consider this one example
https://patchew.org/linux/[email protected]/[email protected]/
Thanks,
Yury
Quoted 69 lines · Shradha Gupta
Commit
91bfe210e196("net: mana: add a function to spread IRQs per CPUs") added theirq_setup()function that distributes IRQs on CPUs according to a tricky heuristic. The corresponding commit message explains the heuristic.Duplicate it in the source code to make available for readers without digging git in history. Also, add more detailed explanation about how the heuristics is implemented.
Signed-off-by Yury Norov [NVIDIA] <[email protected]>Signed-off-by Shradha Gupta <[email protected]>--- .../net/ethernet/microsoft/mana/gdma_main.c | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+)drivers/net/ethernet/microsoft/mana/gdma_main.c +41
-0@@ -1288,6 +1288,47 @@ void mana_gd_free_res_map(struct gdma_resource *r) r->size =0 ; } +/* +* Spread on CPUs with the following heuristics: +* +* 1. No more than one IRQ per CPU, if possible; +* 2. NUMA locality is the second priority; +* 3. Sibling dislocality is the last priority. +* +* Let's consider this topology: +* +* Node 0 1 +* Core 0 1 2 3 +* CPU 0 1 2 3 4 5 6 7 +* +* The most performant IRQ distribution based on the above topology +* and heuristics may look like this: +* +* IRQ Nodes Cores CPUs +* 0 1 0 0-1 +* 1 1 1 2-3 +* 2 1 0 0-1 +* 3 1 1 2-3 +* 4 2 2 4-5 +* 5 2 3 6-7 +* 6 2 2 4-5 +* 7 2 3 6-7 +* +* The heuristics is implemented as follows. +* +* The outer for_each() loop resets the 'weight' to the actual number +* of CPUs in the hop. Then inner for_each() loop decrements it by the +* number of sibling groups (cores) while assigning first set of IRQs +* to each group. IRQs 0 and 1 above are distributed this way. +* +* Now, because NUMA locality is more important, we should walk the +* same set of siblings and assign 2nd set of IRQs (2 and 3), and it's +* implemented by the medium while() loop. We do like this unless the +* number of IRQs assigned on this hop will not become equal to number +* of CPUs in the hop (weight == 0). Then we switch to the next hop and +* do the same thing. +*/ +static int irq_setup (unsigned int *irqs,unsigned int len,int node) {const struct cpumask *next, *prev = cpu_none_mask;Signature · 1 line
2.34.1
Re: [PATCH v4 3/5] net: mana: explain irq_setup() algorithm
Quoted 5 lines · Yury Norov
So now git will think that you're the author of the patch.
If author and sender are different people, the first line in commit message body should state that. In this case, it should be:
Quoted 9 lines
From: Yury Norov <[email protected]>
Please consider this one example
https://patchew.org/linux/[email protected]/[email protected]/
Thanks,
Yury
Understood, Thank you Yury. I'll make this change in the next version
Regards,
Shradha.
Quoted 70 lines · Shradha Gupta
Quoted 69 lines
Commit
91bfe210e196("net: mana: add a function to spread IRQs per CPUs") added theirq_setup()function that distributes IRQs on CPUs according to a tricky heuristic. The corresponding commit message explains the heuristic.Duplicate it in the source code to make available for readers without digging git in history. Also, add more detailed explanation about how the heuristics is implemented.
Signed-off-by Yury Norov [NVIDIA] <[email protected]>Signed-off-by Shradha Gupta <[email protected]>--- .../net/ethernet/microsoft/mana/gdma_main.c | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+)drivers/net/ethernet/microsoft/mana/gdma_main.c +41
-0@@ -1288,6 +1288,47 @@ void mana_gd_free_res_map(struct gdma_resource *r) r->size =0 ; } +/* +* Spread on CPUs with the following heuristics: +* +* 1. No more than one IRQ per CPU, if possible; +* 2. NUMA locality is the second priority; +* 3. Sibling dislocality is the last priority. +* +* Let's consider this topology: +* +* Node 0 1 +* Core 0 1 2 3 +* CPU 0 1 2 3 4 5 6 7 +* +* The most performant IRQ distribution based on the above topology +* and heuristics may look like this: +* +* IRQ Nodes Cores CPUs +* 0 1 0 0-1 +* 1 1 1 2-3 +* 2 1 0 0-1 +* 3 1 1 2-3 +* 4 2 2 4-5 +* 5 2 3 6-7 +* 6 2 2 4-5 +* 7 2 3 6-7 +* +* The heuristics is implemented as follows. +* +* The outer for_each() loop resets the 'weight' to the actual number +* of CPUs in the hop. Then inner for_each() loop decrements it by the +* number of sibling groups (cores) while assigning first set of IRQs +* to each group. IRQs 0 and 1 above are distributed this way. +* +* Now, because NUMA locality is more important, we should walk the +* same set of siblings and assign 2nd set of IRQs (2 and 3), and it's +* implemented by the medium while() loop. We do like this unless the +* number of IRQs assigned on this hop will not become equal to number +* of CPUs in the hop (weight == 0). Then we switch to the next hop and +* do the same thing. +*/ +static int irq_setup (unsigned int *irqs,unsigned int len,int node) {const struct cpumask *next, *prev = cpu_none_mask;Signature · 1 line
2.34.1
PATCH v4 4/5 net: mana: Allow irq_setup() to skip cpus for affinity
In reply to Shradha GuptaIn order to prepare the MANA driver to allocate the MSI-X IRQs dynamically, we need to enhance irq_setup() to allow skipping affinitizing IRQs to the first CPU sibling group.
This would be for cases when the number of IRQs is less than or equal to the number of online CPUs. In such cases for dynamically added IRQs the first CPU sibling group would already be affinitized with HWC IRQ.
---
Changes in v4
* fix commit description
* avoided using next_cpumask: label in the irq_setup()
--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +12 -4
@@ -1329,7 +1329,8 @@ void mana_gd_free_res_map(struct gdma_resource *r)* do the same thing. */ -static int irq_setup (unsigned int *irqs,unsigned int len,int node) +static int irq_setup (unsigned int *irqs,unsigned int len,int node, +bool skip_first_cpu) {const struct cpumask *next, *prev = cpu_none_mask; cpumask_var_t cpus __free(free_cpumask_var);
@@ -1344,11 +1345,18 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node)while (weight >0 ) {cpumask_andnot (cpus, next, prev);for_each_cpu (cpu, cpus) { +cpumask_andnot (cpus, cpus,topology_sibling_cpumask (cpu)); + --weight; + +if (unlikely (skip_first_cpu)) { + skip_first_cpu = false; +continue ; + } +if (len-- ==0 ) goto done; +irq_set_affinity_and_hint (*irqs++,topology_sibling_cpumask (cpu)); -cpumask_andnot (cpus, cpus,topology_sibling_cpumask (cpu)); - --weight; } } prev = next;
@@ -1444,7 +1452,7 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev) } } - err =irq_setup (irqs, (nvec - start_irq_index), gc->numa_node); + err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false);if (err) goto free_irq;
Signature · 1 line
2.34.1
PATCH v4 5/5 net: mana: Allocate MSI-X vectors dynamically
In reply to Shradha GuptaCurrently, the MANA driver allocates MSI-X vectors statically based on MANA_MAX_NUM_QUEUES and num_online_cpus() values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.
To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
---
Changes in v4:
* added BUG_ON at appropriate places
* moved xa_destroy to mana_gd_remove()
* rearragned the cleanup logic in mana_gd_setup_dyn_irqs()
* simplified processing around start_irq_index in mana_gd_setup_irqs()
* return 0 instead of return err as appropriate
---
Changes in v3:
* implemented irq_contexts as xarrays rather than list
* split the patch to create a perparation patch around irq_setup()
* add log when IRQ allocation/setup for remaining IRQs fails
---
Changes in v2:
* Use string 'MSI-X vectors' instead of 'pci vectors'
* make skip-cpu a bool instead of int
* rearrange the comment arout skip_cpu variable appropriately
* update the capability bit for driver indicating dynamic IRQ allocation
* enforced max line length to 80
* enforced RCT convention
* initialized gic to NULL, for when there is a possibility of gic not being populated correctly
--- .../net/ethernet/microsoft/mana/gdma_main.c | 306 +++++++++++++----- include/net/mana/gdma.h | 8 +- 2 files changed, 235 insertions(+), 79 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +229 -77
@@ -6,6 +6,8 @@#include <linux/pci.h> #include <linux/utsname.h> #include <linux/version.h> +#include <linux/msi.h> +#include <linux/irqdomain.h> #include <net/mana/mana.h>
@@ -80,8 +82,15 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)return err ? err : -EPROTO ; } -if (gc->num_msix_usable > resp.max_msix) - gc->num_msix_usable = resp.max_msix; +if (!pci_msix_can_alloc_dyn (pdev)) { +if (gc->num_msix_usable > resp.max_msix) + gc->num_msix_usable = resp.max_msix; + }else { +/* If dynamic allocation is enabled we have already allocated +* hwc msi +*/ + gc->num_msix_usable =min (resp.max_msix,num_online_cpus () +1 ); + }if (gc->num_msix_usable <=1 )return -ENOSPC ;
@@ -482,7 +491,9 @@ static int mana_gd_register_irq(struct gdma_queue *queue, } queue->eq.msix_index = msi_index; - gic = &gc->irq_contexts[msi_index]; + gic =xa_load (&gc->irq_contexts, msi_index); +if (WARN_ON (!gic)) +return -EINVAL ;spin_lock_irqsave (&gic->lock, flags);list_add_rcu (&queue->entry, &gic->eq_list);
@@ -507,7 +518,10 @@ static void mana_gd_deregiser_irq(struct gdma_queue *queue)if (WARN_ON (msix_index >= gc->num_msix_usable))return ; - gic = &gc->irq_contexts[msix_index]; + gic =xa_load (&gc->irq_contexts, msix_index); +if (WARN_ON (!gic)) +return ; +spin_lock_irqsave (&gic->lock, flags);list_for_each_entry_rcu (eq, &gic->eq_list, entry) {if (queue == eq) {
@@ -1366,47 +1380,113 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node,return 0 ; } -static int mana_gd_setup_irqs (struct pci_dev *pdev) +static int mana_gd_setup_dyn_irqs (struct pci_dev *pdev,int nvec) {struct gdma_context *gc =pci_get_drvdata (pdev); -unsigned int max_queues_per_port;struct gdma_irq_context *gic; -unsigned int max_irqs, cpu; -int start_irq_index =1 ; -int nvec, *irqs, irq; -int err, i =0 , j; +bool skip_first_cpu = false; +int *irqs, irq, err, i;cpus_read_lock (); - max_queues_per_port =num_online_cpus (); -if (max_queues_per_port >MANA_MAX_NUM_QUEUES ) - max_queues_per_port =MANA_MAX_NUM_QUEUES ; -/* Need 1 interrupt for the Hardware communication Channel (HWC) */ - max_irqs = max_queues_per_port +1 ; - - nvec =pci_alloc_irq_vectors (pdev,2 , max_irqs,PCI_IRQ_MSIX ); -if (nvec <0 ) { -cpus_read_unlock (); -return nvec; - } -if (nvec <=num_online_cpus ()) - start_irq_index =0 ; - - irqs =kmalloc_array ((nvec - start_irq_index),sizeof (int ),GFP_KERNEL ); + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL );if (!irqs) { err = -ENOMEM ; goto free_irq_vector; } - gc->irq_contexts =kcalloc (nvec,sizeof (struct gdma_irq_context ), -GFP_KERNEL ); -if (!gc->irq_contexts) { +/* +* While processing the next pci irq vector, we start with index 1, +* as IRQ vector at index 0 is already processed for HWC. +* However, the population of irqs array starts with index 0, to be +* further used in irq_setup() +*/ +for (i =1 ; i <= nvec; i++) { + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events; +INIT_LIST_HEAD (&gic->eq_list); +spin_lock_init (&gic->lock); + +snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , + i -1 ,pci_name (pdev)); + +/* one pci vector is already allocated for HWC */ + irqs[i -1 ] =pci_irq_vector (pdev, i); +if (irqs[i -1 ] <0 ) { + err = irqs[i -1 ]; + goto free_current_gic; + } + + err =request_irq (irqs[i -1 ], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); + } + +/* +* When calling irq_setup() for dynamically added IRQs, if number of +* CPUs is more than or equal to allocated MSI-X, we need to skip the +* first CPU sibling group since they are already affinitized to HWC IRQ +*/ +if (gc->num_msix_usable <=num_online_cpus ()) + skip_first_cpu = true; + + err =irq_setup (irqs, nvec, gc->numa_node, skip_first_cpu); +if (err) + goto free_irq; + +cpus_read_unlock (); +kfree (irqs); +return 0 ; + +free_current_gic: +kfree (gic); +free_irq: +for (i -=1 ; i >0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ; + +irq_update_affinity_hint (irq,NULL ); +free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); + } +kfree (irqs); +free_irq_vector: +cpus_read_unlock (); +return err; +} + +static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock (); + + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);
@@ -1418,69 +1498,128 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , i -1 ,pci_name (pdev)); - irq =pci_irq_vector (pdev, i); -if (irq <0 ) { - err = irq; - goto free_irq; + irqs[i] =pci_irq_vector (pdev, i); +if (irqs[i] <0 ) { + err = irqs[i]; + goto free_current_gic; } -if (!i) { - err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_irq; - -/* If number of IRQ is one extra than number of online CPUs, -* then we need to assign IRQ0 (hwc irq) and IRQ1 to -* same CPU. -* Else we will use different CPUs for IRQ0 and IRQ1. -* Also we are using cpumask_local_spread instead of -* cpumask_first for the node, because the node can be -* mem only. -*/ -if (start_irq_index) { - cpu =cpumask_local_spread (i, gc->numa_node); -irq_set_affinity_and_hint (irq,cpumask_of (cpu)); - }else { - irqs[start_irq_index] = irq; - } - }else { - irqs[i - start_irq_index] = irq; - err =request_irq (irqs[i - start_irq_index], mana_gd_intr,0 , - gic->name, gic); -if (err) - goto free_irq; - } + err =request_irq (irqs[i], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); } - err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false); +/* If number of IRQ is one extra than number of online CPUs, +* then we need to assign IRQ0 (hwc irq) and IRQ1 to +* same CPU. +* Else we will use different CPUs for IRQ0 and IRQ1. +* Also we are using cpumask_local_spread instead of +* cpumask_first for the node, because the node can be +* mem only. +*/ + start_irqs = irqs; +if (nvec >num_online_cpus ()) { + cpu =cpumask_local_spread (0 , gc->numa_node); +irq_set_affinity_and_hint (irqs[0 ],cpumask_of (cpu)); + irqs++; + nvec -=1 ; + } + + err =irq_setup (irqs, nvec, gc->numa_node, false);if (err) goto free_irq; - gc->max_num_msix = nvec; - gc->num_msix_usable = nvec;cpus_read_unlock (); -kfree (irqs); +kfree (start_irqs);return 0 ; +free_current_gic: +kfree (gic); free_irq: -for (j = i -1 ; j >=0 ; j--) { - irq =pci_irq_vector (pdev, j); - gic = &gc->irq_contexts[j]; +for (i -=1 ; i >=0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); } -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; -free_irq_array: -kfree (irqs); +kfree (start_irqs); free_irq_vector:cpus_read_unlock (); -pci_free_irq_vectors (pdev);return err; } +static int mana_gd_setup_hwc_irqs (struct pci_dev *pdev) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +unsigned int max_irqs, min_irqs; +int nvec, err; + +if (pci_msix_can_alloc_dyn (pdev)) { + max_irqs =1 ; + min_irqs =1 ; + }else { +/* Need 1 interrupt for HWC */ + max_irqs =min (num_online_cpus (),MANA_MAX_NUM_QUEUES ) +1 ; + min_irqs =2 ; + } + + nvec =pci_alloc_irq_vectors (pdev, min_irqs, max_irqs,PCI_IRQ_MSIX ); +if (nvec <0 ) +return nvec; + + err =mana_gd_setup_irqs (pdev, nvec); +if (err) { +pci_free_irq_vectors (pdev); +return err; + } + + gc->num_msix_usable = nvec; + gc->max_num_msix = nvec; + +return 0 ; +} + +static int mana_gd_setup_remaining_irqs (struct pci_dev *pdev) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct msi_map irq_map; +int max_irqs, i, err; + +if (!pci_msix_can_alloc_dyn (pdev)) +/* remain irqs are already allocated with HWC IRQ */ +return 0 ; + +/* allocate only remaining IRQs*/ + max_irqs = gc->num_msix_usable -1 ; + +for (i =1 ; i <= max_irqs; i++) { + irq_map =pci_msix_alloc_irq_at (pdev, i,NULL ); +if (!irq_map.virq) { + err = irq_map.index; +/* caller will handle cleaning up all allocated +* irqs, after HWC is destroyed +*/ +return err; + } + } + + err =mana_gd_setup_dyn_irqs (pdev, max_irqs); +if (err) +return err; + + gc->max_num_msix = gc->max_num_msix + max_irqs; + +return 0 ; +} +static void mana_gd_remove_irqs (struct pci_dev *pdev) {struct gdma_context *gc =pci_get_drvdata (pdev);
@@ -1495,19 +1634,21 @@ static void mana_gd_remove_irqs(struct pci_dev *pdev)if (irq <0 )continue ; - gic = &gc->irq_contexts[i]; + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;/* Need to clear the hint before free_irq */ irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); }pci_free_irq_vectors (pdev); gc->max_num_msix =0 ; gc->num_msix_usable =0 ; -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; }static int mana_gd_setup (struct pci_dev *pdev)
@@ -1518,9 +1659,10 @@ static int mana_gd_setup(struct pci_dev *pdev)mana_gd_init_registers (pdev);mana_smc_init (&gc->shm_channel, gc->dev, gc->shm_base); - err =mana_gd_setup_irqs (pdev); + err =mana_gd_setup_hwc_irqs (pdev);if (err) { -dev_err (gc->dev,"Failed to setup IRQs: %d\n" , err); +dev_err (gc->dev,"Failed to setup IRQs for HWC creation: %d\n" , + err);return err; }
@@ -1536,6 +1678,12 @@ static int mana_gd_setup(struct pci_dev *pdev)if (err) goto destroy_hwc; + err =mana_gd_setup_remaining_irqs (pdev); +if (err) { +dev_err (gc->dev,"Failed to setup remaining IRQs: %d" , err); + goto destroy_hwc; + } + err =mana_gd_detect_devices (pdev);if (err) goto destroy_hwc;
@@ -1612,6 +1760,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent) gc->is_pf =mana_is_pf (pdev->device); gc->bar0_va = bar0_va; gc->dev = &pdev->dev; +xa_init (&gc->irq_contexts);if (gc->is_pf) gc->mana_pci_debugfs =debugfs_create_dir ("0" , mana_debugfs_root);
@@ -1640,6 +1789,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)*/ debugfs_remove_recursive (gc->mana_pci_debugfs); gc->mana_pci_debugfs =NULL ; +xa_destroy (&gc->irq_contexts);pci_iounmap (pdev, bar0_va); free_gc:pci_set_drvdata (pdev,NULL );
@@ -1664,6 +1814,8 @@ static void mana_gd_remove(struct pci_dev *pdev) gc->mana_pci_debugfs =NULL ; +xa_destroy (&gc->irq_contexts); +pci_iounmap (pdev, gc->bar0_va);vfree (gc);
include/net/mana/gdma.h +6 -2
@@ -373,7 +373,7 @@ struct gdma_context {unsigned int max_num_queues;unsigned int max_num_msix;unsigned int num_msix_usable; -struct gdma_irq_context *irq_contexts; +struct xarray irq_contexts;/* L2 MTU */ u16 adapter_mtu;
@@ -558,12 +558,16 @@ enum {/* Driver can handle holes (zeros) in the device list */ #define GDMA_DRV_CAP_FLAG_1_DEV_LIST_HOLES_SUP BIT(11) +/* Driver supports dynamic MSI-X vector allocation */ +#define GDMA_DRV_CAP_FLAG_1_DYNAMIC_IRQ_ALLOC_SUPPORT BIT(13) +#define GDMA_DRV_CAP_FLAGS1 \ (GDMA_DRV_CAP_FLAG_1_EQ_SHARING_MULTI_VPORT | \ GDMA_DRV_CAP_FLAG_1_NAPI_WKDONE_FIX | \ GDMA_DRV_CAP_FLAG_1_HWC_TIMEOUT_RECONFIG | \ GDMA_DRV_CAP_FLAG_1_VARIABLE_INDIRECTION_TABLE_SUPPORT | \ - GDMA_DRV_CAP_FLAG_1_DEV_LIST_HOLES_SUP) + GDMA_DRV_CAP_FLAG_1_DEV_LIST_HOLES_SUP | \ + GDMA_DRV_CAP_FLAG_1_DYNAMIC_IRQ_ALLOC_SUPPORT)#define GDMA_DRV_CAP_FLAGS2 0
Signature · 1 line
2.34.1
Re: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
Quoted 321 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>---
Changes in v4:
* addedBUG_ONat appropriate places
* movedxa_destroytomana_gd_remove()
* rearragned the cleanup logic inmana_gd_setup_dyn_irqs()
* simplified processing aroundstart_irq_indexinmana_gd_setup_irqs()
* return 0 instead of return err as appropriate
---
Changes in v3:
* implementedirq_contextsas xarrays rather than list
* split the patch to create a perparation patch aroundirq_setup()
* add log when IRQ allocation/setup for remaining IRQs fails
---
Changes in v2:
* Use string 'MSI-X vectors' instead of 'pci vectors'
* make skip-cpu a bool instead of int
* rearrange the comment aroutskip_cpuvariable appropriately
* update the capability bit for driver indicating dynamic IRQ allocation
* enforced max line length to 80
* enforced RCT convention
* initialized gic to NULL, for when there is a possibility of gic not being populated correctly--- .../net/ethernet/microsoft/mana/gdma_main.c | 306 +++++++++++++----- include/net/mana/gdma.h | 8 +- 2 files changed, 235 insertions(+), 79 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +144
-67@@ -6,6 +6,8 @@#include <linux/pci.h> #include <linux/utsname.h> #include <linux/version.h> +#include <linux/msi.h> +#include <linux/irqdomain.h> #include <net/mana/mana.h> @@ -80,8 +82,15 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)return err ? err : -EPROTO ; } -if (gc->num_msix_usable > resp.max_msix) - gc->num_msix_usable = resp.max_msix; +if (!pci_msix_can_alloc_dyn (pdev)) { +if (gc->num_msix_usable > resp.max_msix) + gc->num_msix_usable = resp.max_msix; + }else { +/* If dynamic allocation is enabled we have already allocated +* hwc msi +*/ + gc->num_msix_usable =min (resp.max_msix,num_online_cpus () +1 ); + }if (gc->num_msix_usable <=1 )return -ENOSPC ;@@ -482,7 +491,9 @@ static int mana_gd_register_irq(struct gdma_queue *queue, } queue->eq.msix_index = msi_index; - gic = &gc->irq_contexts[msi_index]; + gic =xa_load (&gc->irq_contexts, msi_index); +if (WARN_ON (!gic)) +return -EINVAL ;spin_lock_irqsave (&gic->lock, flags);list_add_rcu (&queue->entry, &gic->eq_list);@@ -507,7 +518,10 @@ static void mana_gd_deregiser_irq(struct gdma_queue *queue)if (WARN_ON (msix_index >= gc->num_msix_usable))return ; - gic = &gc->irq_contexts[msix_index]; + gic =xa_load (&gc->irq_contexts, msix_index); +if (WARN_ON (!gic)) +return ; +spin_lock_irqsave (&gic->lock, flags);list_for_each_entry_rcu (eq, &gic->eq_list, entry) {if (queue == eq) {@@ -1366,47 +1380,113 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node,return 0 ; } -static int mana_gd_setup_irqs (struct pci_dev *pdev) +static int mana_gd_setup_dyn_irqs (struct pci_dev *pdev,int nvec) {struct gdma_context *gc =pci_get_drvdata (pdev); -unsigned int max_queues_per_port;struct gdma_irq_context *gic; -unsigned int max_irqs, cpu; -int start_irq_index =1 ; -int nvec, *irqs, irq; -int err, i =0 , j; +bool skip_first_cpu = false; +int *irqs, irq, err, i;cpus_read_lock (); - max_queues_per_port =num_online_cpus (); -if (max_queues_per_port >MANA_MAX_NUM_QUEUES ) - max_queues_per_port =MANA_MAX_NUM_QUEUES ; -/* Need 1 interrupt for the Hardware communication Channel (HWC) */ - max_irqs = max_queues_per_port +1 ; - - nvec =pci_alloc_irq_vectors (pdev,2 , max_irqs,PCI_IRQ_MSIX ); -if (nvec <0 ) { -cpus_read_unlock (); -return nvec; - } -if (nvec <=num_online_cpus ()) - start_irq_index =0 ; - - irqs =kmalloc_array ((nvec - start_irq_index),sizeof (int ),GFP_KERNEL ); + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL );if (!irqs) { err = -ENOMEM ; goto free_irq_vector; } - gc->irq_contexts =kcalloc (nvec,sizeof (struct gdma_irq_context ), -GFP_KERNEL ); -if (!gc->irq_contexts) { +/* +* While processing the next pci irq vector, we start with index 1, +* as IRQ vector at index 0 is already processed for HWC. +* However, the population of irqs array starts with index 0, to be +* further used in irq_setup() +*/ +for (i =1 ; i <= nvec; i++) { + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events; +INIT_LIST_HEAD (&gic->eq_list); +spin_lock_init (&gic->lock); + +snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , + i -1 ,pci_name (pdev)); + +/* one pci vector is already allocated for HWC */ + irqs[i -1 ] =pci_irq_vector (pdev, i); +if (irqs[i -1 ] <0 ) { + err = irqs[i -1 ]; + goto free_current_gic; + } + + err =request_irq (irqs[i -1 ], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); + } + +/* +* When calling irq_setup() for dynamically added IRQs, if number of +* CPUs is more than or equal to allocated MSI-X, we need to skip the +* first CPU sibling group since they are already affinitized to HWC IRQ +*/ +if (gc->num_msix_usable <=num_online_cpus ()) + skip_first_cpu = true; + + err =irq_setup (irqs, nvec, gc->numa_node, skip_first_cpu); +if (err) + goto free_irq; + +cpus_read_unlock (); +kfree (irqs); +return 0 ; + +free_current_gic: +kfree (gic); +free_irq: +for (i -=1 ; i >0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ; + +irq_update_affinity_hint (irq,NULL ); +free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); + } +kfree (irqs); +free_irq_vector: +cpus_read_unlock (); +return err; +} + +static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock (); + + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);@@ -1418,69 +1498,128 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , i -1 ,pci_name (pdev)); - irq =pci_irq_vector (pdev, i); -if (irq <0 ) { - err = irq; - goto free_irq; + irqs[i] =pci_irq_vector (pdev, i); +if (irqs[i] <0 ) { + err = irqs[i]; + goto free_current_gic; } -if (!i) { - err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_irq; - -/* If number of IRQ is one extra than number of online CPUs, -* then we need to assign IRQ0 (hwc irq) and IRQ1 to -* same CPU. -* Else we will use different CPUs for IRQ0 and IRQ1. -* Also we are using cpumask_local_spread instead of -* cpumask_first for the node, because the node can be -* mem only. -*/ -if (start_irq_index) { - cpu =cpumask_local_spread (i, gc->numa_node); -irq_set_affinity_and_hint (irq,cpumask_of (cpu)); - }else { - irqs[start_irq_index] = irq; - } - }else { - irqs[i - start_irq_index] = irq; - err =request_irq (irqs[i - start_irq_index], mana_gd_intr,0 , - gic->name, gic); -if (err) - goto free_irq; - } + err =request_irq (irqs[i], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); } - err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false); +/* If number of IRQ is one extra than number of online CPUs, +* then we need to assign IRQ0 (hwc irq) and IRQ1 to +* same CPU. +* Else we will use different CPUs for IRQ0 and IRQ1. +* Also we are using cpumask_local_spread instead of +* cpumask_first for the node, because the node can be +* mem only. +*/ + start_irqs = irqs; +if (nvec >num_online_cpus ()) { + cpu =cpumask_local_spread (0 , gc->numa_node); +irq_set_affinity_and_hint (irqs[0 ],cpumask_of (cpu)); + irqs++; + nvec -=1 ; + } + + err =irq_setup (irqs, nvec, gc->numa_node, false);if (err) goto free_irq; - gc->max_num_msix = nvec; - gc->num_msix_usable = nvec;cpus_read_unlock (); -kfree (irqs); +kfree (start_irqs);return 0 ; +free_current_gic: +kfree (gic); free_irq: -for (j = i -1 ; j >=0 ; j--) { - irq =pci_irq_vector (pdev, j); - gic = &gc->irq_contexts[j]; +for (i -=1 ; i >=0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;irq_update_affinity_hint (irq,NULL );
drivers/net/ethernet/microsoft/mana/gdma_main.c static int mana_gd_setup_irqs(struct pci_dev *pdev) · line 1553
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-4free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); } -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; -free_irq_array: -kfree (irqs); +kfree (start_irqs);
There is a case when start_irqs can be used here uninitialized.
- Saurabh
Re: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
Quoted 335 lines · Saurabh Singh Sengar
Quoted 321 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>---
Changes in v4:
* addedBUG_ONat appropriate places
* movedxa_destroytomana_gd_remove()
* rearragned the cleanup logic inmana_gd_setup_dyn_irqs()
* simplified processing aroundstart_irq_indexinmana_gd_setup_irqs()
* return 0 instead of return err as appropriate
---
Changes in v3:
* implementedirq_contextsas xarrays rather than list
* split the patch to create a perparation patch aroundirq_setup()
* add log when IRQ allocation/setup for remaining IRQs fails
---
Changes in v2:
* Use string 'MSI-X vectors' instead of 'pci vectors'
* make skip-cpu a bool instead of int
* rearrange the comment aroutskip_cpuvariable appropriately
* update the capability bit for driver indicating dynamic IRQ allocation
* enforced max line length to 80
* enforced RCT convention
* initialized gic to NULL, for when there is a possibility of gic not being populated correctly--- .../net/ethernet/microsoft/mana/gdma_main.c | 306 +++++++++++++----- include/net/mana/gdma.h | 8 +- 2 files changed, 235 insertions(+), 79 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +144
-67@@ -6,6 +6,8 @@#include <linux/pci.h> #include <linux/utsname.h> #include <linux/version.h> +#include <linux/msi.h> +#include <linux/irqdomain.h> #include <net/mana/mana.h> @@ -80,8 +82,15 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)return err ? err : -EPROTO ; } -if (gc->num_msix_usable > resp.max_msix) - gc->num_msix_usable = resp.max_msix; +if (!pci_msix_can_alloc_dyn (pdev)) { +if (gc->num_msix_usable > resp.max_msix) + gc->num_msix_usable = resp.max_msix; + }else { +/* If dynamic allocation is enabled we have already allocated +* hwc msi +*/ + gc->num_msix_usable =min (resp.max_msix,num_online_cpus () +1 ); + }if (gc->num_msix_usable <=1 )return -ENOSPC ;@@ -482,7 +491,9 @@ static int mana_gd_register_irq(struct gdma_queue *queue, } queue->eq.msix_index = msi_index; - gic = &gc->irq_contexts[msi_index]; + gic =xa_load (&gc->irq_contexts, msi_index); +if (WARN_ON (!gic)) +return -EINVAL ;spin_lock_irqsave (&gic->lock, flags);list_add_rcu (&queue->entry, &gic->eq_list);@@ -507,7 +518,10 @@ static void mana_gd_deregiser_irq(struct gdma_queue *queue)if (WARN_ON (msix_index >= gc->num_msix_usable))return ; - gic = &gc->irq_contexts[msix_index]; + gic =xa_load (&gc->irq_contexts, msix_index); +if (WARN_ON (!gic)) +return ; +spin_lock_irqsave (&gic->lock, flags);list_for_each_entry_rcu (eq, &gic->eq_list, entry) {if (queue == eq) {@@ -1366,47 +1380,113 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node,return 0 ; } -static int mana_gd_setup_irqs (struct pci_dev *pdev) +static int mana_gd_setup_dyn_irqs (struct pci_dev *pdev,int nvec) {struct gdma_context *gc =pci_get_drvdata (pdev); -unsigned int max_queues_per_port;struct gdma_irq_context *gic; -unsigned int max_irqs, cpu; -int start_irq_index =1 ; -int nvec, *irqs, irq; -int err, i =0 , j; +bool skip_first_cpu = false; +int *irqs, irq, err, i;cpus_read_lock (); - max_queues_per_port =num_online_cpus (); -if (max_queues_per_port >MANA_MAX_NUM_QUEUES ) - max_queues_per_port =MANA_MAX_NUM_QUEUES ; -/* Need 1 interrupt for the Hardware communication Channel (HWC) */ - max_irqs = max_queues_per_port +1 ; - - nvec =pci_alloc_irq_vectors (pdev,2 , max_irqs,PCI_IRQ_MSIX ); -if (nvec <0 ) { -cpus_read_unlock (); -return nvec; - } -if (nvec <=num_online_cpus ()) - start_irq_index =0 ; - - irqs =kmalloc_array ((nvec - start_irq_index),sizeof (int ),GFP_KERNEL ); + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL );if (!irqs) { err = -ENOMEM ; goto free_irq_vector; } - gc->irq_contexts =kcalloc (nvec,sizeof (struct gdma_irq_context ), -GFP_KERNEL ); -if (!gc->irq_contexts) { +/* +* While processing the next pci irq vector, we start with index 1, +* as IRQ vector at index 0 is already processed for HWC. +* However, the population of irqs array starts with index 0, to be +* further used in irq_setup() +*/ +for (i =1 ; i <= nvec; i++) { + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events; +INIT_LIST_HEAD (&gic->eq_list); +spin_lock_init (&gic->lock); + +snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , + i -1 ,pci_name (pdev)); + +/* one pci vector is already allocated for HWC */ + irqs[i -1 ] =pci_irq_vector (pdev, i); +if (irqs[i -1 ] <0 ) { + err = irqs[i -1 ]; + goto free_current_gic; + } + + err =request_irq (irqs[i -1 ], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); + } + +/* +* When calling irq_setup() for dynamically added IRQs, if number of +* CPUs is more than or equal to allocated MSI-X, we need to skip the +* first CPU sibling group since they are already affinitized to HWC IRQ +*/ +if (gc->num_msix_usable <=num_online_cpus ()) + skip_first_cpu = true; + + err =irq_setup (irqs, nvec, gc->numa_node, skip_first_cpu); +if (err) + goto free_irq; + +cpus_read_unlock (); +kfree (irqs); +return 0 ; + +free_current_gic: +kfree (gic); +free_irq: +for (i -=1 ; i >0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ; + +irq_update_affinity_hint (irq,NULL ); +free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); + } +kfree (irqs); +free_irq_vector: +cpus_read_unlock (); +return err; +} + +static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock (); + + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);@@ -1418,69 +1498,128 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , i -1 ,pci_name (pdev)); - irq =pci_irq_vector (pdev, i); -if (irq <0 ) { - err = irq; - goto free_irq; + irqs[i] =pci_irq_vector (pdev, i); +if (irqs[i] <0 ) { + err = irqs[i]; + goto free_current_gic; } -if (!i) { - err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_irq; - -/* If number of IRQ is one extra than number of online CPUs, -* then we need to assign IRQ0 (hwc irq) and IRQ1 to -* same CPU. -* Else we will use different CPUs for IRQ0 and IRQ1. -* Also we are using cpumask_local_spread instead of -* cpumask_first for the node, because the node can be -* mem only. -*/ -if (start_irq_index) { - cpu =cpumask_local_spread (i, gc->numa_node); -irq_set_affinity_and_hint (irq,cpumask_of (cpu)); - }else { - irqs[start_irq_index] = irq; - } - }else { - irqs[i - start_irq_index] = irq; - err =request_irq (irqs[i - start_irq_index], mana_gd_intr,0 , - gic->name, gic); -if (err) - goto free_irq; - } + err =request_irq (irqs[i], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); } - err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false); +/* If number of IRQ is one extra than number of online CPUs, +* then we need to assign IRQ0 (hwc irq) and IRQ1 to +* same CPU. +* Else we will use different CPUs for IRQ0 and IRQ1. +* Also we are using cpumask_local_spread instead of +* cpumask_first for the node, because the node can be +* mem only. +*/ + start_irqs = irqs; +if (nvec >num_online_cpus ()) { + cpu =cpumask_local_spread (0 , gc->numa_node); +irq_set_affinity_and_hint (irqs[0 ],cpumask_of (cpu)); + irqs++; + nvec -=1 ; + } + + err =irq_setup (irqs, nvec, gc->numa_node, false);if (err) goto free_irq; - gc->max_num_msix = nvec; - gc->num_msix_usable = nvec;cpus_read_unlock (); -kfree (irqs); +kfree (start_irqs);return 0 ; +free_current_gic: +kfree (gic); free_irq: -for (j = i -1 ; j >=0 ; j--) { - irq =pci_irq_vector (pdev, j); - gic = &gc->irq_contexts[j]; +for (i -=1 ; i >=0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;irq_update_affinity_hint (irq,NULL );drivers/net/ethernet/microsoft/mana/gdma_main.c static int mana_gd_setup_irqs(struct pci_dev *pdev) · line 1553
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-4free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); } -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; -free_irq_array: -kfree (irqs); +kfree (start_irqs);There is a case when
start_irqscan be used here uninitialized.
Quoted 1 line
- Saurabh
Thanks Saurabh. I'll get this in the next version
Simon Horman
reviewerRe: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
In reply to Shradha GuptaQuoted 12 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>
...
Quoted 61 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +23
-25+static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock (); + + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);@@ -1418,69 +1498,128 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , i -1 ,pci_name (pdev)); - irq =pci_irq_vector (pdev, i); -if (irq <0 ) { - err = irq; - goto free_irq; + irqs[i] =pci_irq_vector (pdev, i); +if (irqs[i] <0 ) { + err = irqs[i]; + goto free_current_gic; } -if (!i) { - err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_irq; - -/* If number of IRQ is one extra than number of online CPUs, -* then we need to assign IRQ0 (hwc irq) and IRQ1 to -* same CPU. -* Else we will use different CPUs for IRQ0 and IRQ1. -* Also we are using cpumask_local_spread instead of -* cpumask_first for the node, because the node can be -* mem only. -*/ -if (start_irq_index) { - cpu =cpumask_local_spread (i, gc->numa_node); -irq_set_affinity_and_hint (irq,cpumask_of (cpu)); - }else { - irqs[start_irq_index] = irq; - }
drivers/net/ethernet/microsoft/mana/gdma_main.c static int mana_gd_setup_irqs(struct pci_dev *pdev) · line 1509
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-7- }else { - irqs[i - start_irq_index] = irq; - err =request_irq (irqs[i - start_irq_index], mana_gd_intr,0 , - gic->name, gic); -if (err) - goto free_irq; - } + err =request_irq (irqs[i], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic;
Jumping to free_current_gic will free start_irqs. However, start_irqs isn't initialised until a few lines below.
Flagged by Smatch.
Quoted 60 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +30
-12+ +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); } - err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false); +/* If number of IRQ is one extra than number of online CPUs, +* then we need to assign IRQ0 (hwc irq) and IRQ1 to +* same CPU. +* Else we will use different CPUs for IRQ0 and IRQ1. +* Also we are using cpumask_local_spread instead of +* cpumask_first for the node, because the node can be +* mem only. +*/ + start_irqs = irqs; +if (nvec >num_online_cpus ()) { + cpu =cpumask_local_spread (0 , gc->numa_node); +irq_set_affinity_and_hint (irqs[0 ],cpumask_of (cpu)); + irqs++; + nvec -=1 ; + } + + err =irq_setup (irqs, nvec, gc->numa_node, false);if (err) goto free_irq; - gc->max_num_msix = nvec; - gc->num_msix_usable = nvec;cpus_read_unlock (); -kfree (irqs); +kfree (start_irqs);return 0 ; +free_current_gic: +kfree (gic); free_irq: -for (j = i -1 ; j >=0 ; j--) { - irq =pci_irq_vector (pdev, j); - gic = &gc->irq_contexts[j]; +for (i -=1 ; i >=0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); } -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; -free_irq_array: -kfree (irqs); +kfree (start_irqs); free_irq_vector:cpus_read_unlock (); -pci_free_irq_vectors (pdev);return err; }
...
Re: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
Quoted 91 lines · Simon Horman
Quoted 12 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>...
Quoted 61 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +23
-25+static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock (); + + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);@@ -1418,69 +1498,128 @@ static int mana_gd_setup_irqs(struct pci_dev *pdev)snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , i -1 ,pci_name (pdev)); - irq =pci_irq_vector (pdev, i); -if (irq <0 ) { - err = irq; - goto free_irq; + irqs[i] =pci_irq_vector (pdev, i); +if (irqs[i] <0 ) { + err = irqs[i]; + goto free_current_gic; } -if (!i) { - err =request_irq (irq, mana_gd_intr,0 , gic->name, gic); -if (err) - goto free_irq; - -/* If number of IRQ is one extra than number of online CPUs, -* then we need to assign IRQ0 (hwc irq) and IRQ1 to -* same CPU. -* Else we will use different CPUs for IRQ0 and IRQ1. -* Also we are using cpumask_local_spread instead of -* cpumask_first for the node, because the node can be -* mem only. -*/ -if (start_irq_index) { - cpu =cpumask_local_spread (i, gc->numa_node); -irq_set_affinity_and_hint (irq,cpumask_of (cpu)); - }else { - irqs[start_irq_index] = irq; - }drivers/net/ethernet/microsoft/mana/gdma_main.c static int mana_gd_setup_irqs(struct pci_dev *pdev) · line 1509
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-7- }else { - irqs[i - start_irq_index] = irq; - err =request_irq (irqs[i - start_irq_index], mana_gd_intr,0 , - gic->name, gic); -if (err) - goto free_irq; - } + err =request_irq (irqs[i], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic;Jumping to
free_current_gicwill freestart_irqs. However,start_irqsisn't initialised until a few lines below.
Quoted 2 lines
Flagged by Smatch.
Thanks Simon, I'll get this in next version
Quoted 62 lines
Quoted 60 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +30
-12+ +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); } - err =irq_setup (irqs, nvec - start_irq_index, gc->numa_node, false); +/* If number of IRQ is one extra than number of online CPUs, +* then we need to assign IRQ0 (hwc irq) and IRQ1 to +* same CPU. +* Else we will use different CPUs for IRQ0 and IRQ1. +* Also we are using cpumask_local_spread instead of +* cpumask_first for the node, because the node can be +* mem only. +*/ + start_irqs = irqs; +if (nvec >num_online_cpus ()) { + cpu =cpumask_local_spread (0 , gc->numa_node); +irq_set_affinity_and_hint (irqs[0 ],cpumask_of (cpu)); + irqs++; + nvec -=1 ; + } + + err =irq_setup (irqs, nvec, gc->numa_node, false);if (err) goto free_irq; - gc->max_num_msix = nvec; - gc->num_msix_usable = nvec;cpus_read_unlock (); -kfree (irqs); +kfree (start_irqs);return 0 ; +free_current_gic: +kfree (gic); free_irq: -for (j = i -1 ; j >=0 ; j--) { - irq =pci_irq_vector (pdev, j); - gic = &gc->irq_contexts[j]; +for (i -=1 ; i >=0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ;irq_update_affinity_hint (irq,NULL );free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); } -kfree (gc->irq_contexts); - gc->irq_contexts =NULL ; -free_irq_array: -kfree (irqs); +kfree (start_irqs); free_irq_vector:cpus_read_unlock (); -pci_free_irq_vectors (pdev);return err; }...
Re: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
In reply to Shradha GuptaQuoted 101 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>---
Changes in v4:
* addedBUG_ONat appropriate places
* movedxa_destroytomana_gd_remove()
* rearragned the cleanup logic inmana_gd_setup_dyn_irqs()
* simplified processing aroundstart_irq_indexinmana_gd_setup_irqs()
* return 0 instead of return err as appropriate
---
Changes in v3:
* implementedirq_contextsas xarrays rather than list
* split the patch to create a perparation patch aroundirq_setup()
* add log when IRQ allocation/setup for remaining IRQs fails
---
Changes in v2:
* Use string 'MSI-X vectors' instead of 'pci vectors'
* make skip-cpu a bool instead of int
* rearrange the comment aroutskip_cpuvariable appropriately
* update the capability bit for driver indicating dynamic IRQ allocation
* enforced max line length to 80
* enforced RCT convention
* initialized gic to NULL, for when there is a possibility of gic not being populated correctly--- .../net/ethernet/microsoft/mana/gdma_main.c | 306 +++++++++++++----- include/net/mana/gdma.h | 8 +- 2 files changed, 235 insertions(+), 79 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +19
-5@@ -6,6 +6,8 @@#include <linux/pci.h> #include <linux/utsname.h> #include <linux/version.h> +#include <linux/msi.h> +#include <linux/irqdomain.h> #include <net/mana/mana.h> @@ -80,8 +82,15 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)return err ? err : -EPROTO ; } -if (gc->num_msix_usable > resp.max_msix) - gc->num_msix_usable = resp.max_msix; +if (!pci_msix_can_alloc_dyn (pdev)) { +if (gc->num_msix_usable > resp.max_msix) + gc->num_msix_usable = resp.max_msix; + }else { +/* If dynamic allocation is enabled we have already allocated +* hwc msi +*/ + gc->num_msix_usable =min (resp.max_msix,num_online_cpus () +1 ); + }if (gc->num_msix_usable <=1 )return -ENOSPC ;@@ -482,7 +491,9 @@ static int mana_gd_register_irq(struct gdma_queue *queue, } queue->eq.msix_index = msi_index; - gic = &gc->irq_contexts[msi_index]; + gic =xa_load (&gc->irq_contexts, msi_index); +if (WARN_ON (!gic)) +return -EINVAL ;spin_lock_irqsave (&gic->lock, flags);list_add_rcu (&queue->entry, &gic->eq_list);@@ -507,7 +518,10 @@ static void mana_gd_deregiser_irq(struct gdma_queue *queue)if (WARN_ON (msix_index >= gc->num_msix_usable))return ; - gic = &gc->irq_contexts[msix_index]; + gic =xa_load (&gc->irq_contexts, msix_index); +if (WARN_ON (!gic)) +return ; +spin_lock_irqsave (&gic->lock, flags);list_for_each_entry_rcu (eq, &gic->eq_list, entry) {if (queue == eq) {@@ -1366,47 +1380,113 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node,return 0 ; } -static int mana_gd_setup_irqs (struct pci_dev *pdev) +static int mana_gd_setup_dyn_irqs (struct pci_dev *pdev,int nvec) {struct gdma_context *gc =pci_get_drvdata (pdev);
drivers/net/ethernet/microsoft/mana/gdma_main.c static int irq_setup(unsigned int *irqs, unsigned int len, int node, · line 1390
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-5-unsigned int max_queues_per_port;struct gdma_irq_context *gic; -unsigned int max_irqs, cpu; -int start_irq_index =1 ; -int nvec, *irqs, irq; -int err, i =0 , j; +bool skip_first_cpu = false; +int *irqs, irq, err, i;cpus_read_lock ();
Now that num_online_cpus is moved further down in this new logic, do we want to reduce the critical section ?
I don't think we want kmalloc_array to be protected.
Quoted 93 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +69
-18- max_queues_per_port =num_online_cpus (); -if (max_queues_per_port >MANA_MAX_NUM_QUEUES ) - max_queues_per_port =MANA_MAX_NUM_QUEUES ; -/* Need 1 interrupt for the Hardware communication Channel (HWC) */ - max_irqs = max_queues_per_port +1 ; - - nvec =pci_alloc_irq_vectors (pdev,2 , max_irqs,PCI_IRQ_MSIX ); -if (nvec <0 ) { -cpus_read_unlock (); -return nvec; - } -if (nvec <=num_online_cpus ()) - start_irq_index =0 ; - - irqs =kmalloc_array ((nvec - start_irq_index),sizeof (int ),GFP_KERNEL ); + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL );if (!irqs) { err = -ENOMEM ; goto free_irq_vector; } - gc->irq_contexts =kcalloc (nvec,sizeof (struct gdma_irq_context ), -GFP_KERNEL ); -if (!gc->irq_contexts) { +/* +* While processing the next pci irq vector, we start with index 1, +* as IRQ vector at index 0 is already processed for HWC. +* However, the population of irqs array starts with index 0, to be +* further used in irq_setup() +*/ +for (i =1 ; i <= nvec; i++) { + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events; +INIT_LIST_HEAD (&gic->eq_list); +spin_lock_init (&gic->lock); + +snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , + i -1 ,pci_name (pdev)); + +/* one pci vector is already allocated for HWC */ + irqs[i -1 ] =pci_irq_vector (pdev, i); +if (irqs[i -1 ] <0 ) { + err = irqs[i -1 ]; + goto free_current_gic; + } + + err =request_irq (irqs[i -1 ], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); + } + +/* +* When calling irq_setup() for dynamically added IRQs, if number of +* CPUs is more than or equal to allocated MSI-X, we need to skip the +* first CPU sibling group since they are already affinitized to HWC IRQ +*/ +if (gc->num_msix_usable <=num_online_cpus ()) + skip_first_cpu = true; + + err =irq_setup (irqs, nvec, gc->numa_node, skip_first_cpu); +if (err) + goto free_irq; + +cpus_read_unlock (); +kfree (irqs); +return 0 ; + +free_current_gic: +kfree (gic); +free_irq: +for (i -=1 ; i >0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ; + +irq_update_affinity_hint (irq,NULL ); +free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); + } +kfree (irqs); +free_irq_vector: +cpus_read_unlock (); +return err; +}
drivers/net/ethernet/microsoft/mana/gdma_main.c static int irq_setup(unsigned int *irqs, unsigned int len, int node, · line 1475
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +10
-0+ +static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock ();
Same here
Quoted 8 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-1+ irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) {
drivers/net/ethernet/microsoft/mana/gdma_main.c static int irq_setup(unsigned int *irqs, unsigned int len, int node, · line 1492
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +6
-1- gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);
<snip>
- Saurabh
Re: [PATCH v4 5/5] net: mana: Allocate MSI-X vectors dynamically
Quoted 246 lines · Saurabh Singh Sengar
Quoted 101 lines · Shradha Gupta
Currently, the MANA driver allocates MSI-X vectors statically based on
MANA_MAX_NUM_QUEUESandnum_online_cpus()values and in some cases ends up allocating more vectors than it needs. This is because, by this time we do not have a HW channel and do not know how many IRQs should be allocated.To avoid this, we allocate 1 MSI-X vector during the creation of HWC and after getting the value supported by hardware, dynamically add the remaining MSI-X vectors.
Signed-off-by Shradha Gupta <[email protected]>Reviewed-by Haiyang Zhang <[email protected]>---
Changes in v4:
* addedBUG_ONat appropriate places
* movedxa_destroytomana_gd_remove()
* rearragned the cleanup logic inmana_gd_setup_dyn_irqs()
* simplified processing aroundstart_irq_indexinmana_gd_setup_irqs()
* return 0 instead of return err as appropriate
---
Changes in v3:
* implementedirq_contextsas xarrays rather than list
* split the patch to create a perparation patch aroundirq_setup()
* add log when IRQ allocation/setup for remaining IRQs fails
---
Changes in v2:
* Use string 'MSI-X vectors' instead of 'pci vectors'
* make skip-cpu a bool instead of int
* rearrange the comment aroutskip_cpuvariable appropriately
* update the capability bit for driver indicating dynamic IRQ allocation
* enforced max line length to 80
* enforced RCT convention
* initialized gic to NULL, for when there is a possibility of gic not being populated correctly--- .../net/ethernet/microsoft/mana/gdma_main.c | 306 +++++++++++++----- include/net/mana/gdma.h | 8 +- 2 files changed, 235 insertions(+), 79 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +19
-5@@ -6,6 +6,8 @@#include <linux/pci.h> #include <linux/utsname.h> #include <linux/version.h> +#include <linux/msi.h> +#include <linux/irqdomain.h> #include <net/mana/mana.h> @@ -80,8 +82,15 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)return err ? err : -EPROTO ; } -if (gc->num_msix_usable > resp.max_msix) - gc->num_msix_usable = resp.max_msix; +if (!pci_msix_can_alloc_dyn (pdev)) { +if (gc->num_msix_usable > resp.max_msix) + gc->num_msix_usable = resp.max_msix; + }else { +/* If dynamic allocation is enabled we have already allocated +* hwc msi +*/ + gc->num_msix_usable =min (resp.max_msix,num_online_cpus () +1 ); + }if (gc->num_msix_usable <=1 )return -ENOSPC ;@@ -482,7 +491,9 @@ static int mana_gd_register_irq(struct gdma_queue *queue, } queue->eq.msix_index = msi_index; - gic = &gc->irq_contexts[msi_index]; + gic =xa_load (&gc->irq_contexts, msi_index); +if (WARN_ON (!gic)) +return -EINVAL ;spin_lock_irqsave (&gic->lock, flags);list_add_rcu (&queue->entry, &gic->eq_list);@@ -507,7 +518,10 @@ static void mana_gd_deregiser_irq(struct gdma_queue *queue)if (WARN_ON (msix_index >= gc->num_msix_usable))return ; - gic = &gc->irq_contexts[msix_index]; + gic =xa_load (&gc->irq_contexts, msix_index); +if (WARN_ON (!gic)) +return ; +spin_lock_irqsave (&gic->lock, flags);list_for_each_entry_rcu (eq, &gic->eq_list, entry) {if (queue == eq) {@@ -1366,47 +1380,113 @@ static int irq_setup(unsigned int *irqs, unsigned int len, int node,return 0 ; } -static int mana_gd_setup_irqs (struct pci_dev *pdev) +static int mana_gd_setup_dyn_irqs (struct pci_dev *pdev,int nvec) {struct gdma_context *gc =pci_get_drvdata (pdev);drivers/net/ethernet/microsoft/mana/gdma_main.c static int irq_setup(unsigned int *irqs, unsigned int len, int node, · line 1390
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-5-unsigned int max_queues_per_port;struct gdma_irq_context *gic; -unsigned int max_irqs, cpu; -int start_irq_index =1 ; -int nvec, *irqs, irq; -int err, i =0 , j; +bool skip_first_cpu = false; +int *irqs, irq, err, i;cpus_read_lock ();Now that
num_online_cpusis moved further down in this new logic, do we want to reduce the critical section ?I don't think we want
kmalloc_arrayto be protected.Quoted 93 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +69
-18- max_queues_per_port =num_online_cpus (); -if (max_queues_per_port >MANA_MAX_NUM_QUEUES ) - max_queues_per_port =MANA_MAX_NUM_QUEUES ; -/* Need 1 interrupt for the Hardware communication Channel (HWC) */ - max_irqs = max_queues_per_port +1 ; - - nvec =pci_alloc_irq_vectors (pdev,2 , max_irqs,PCI_IRQ_MSIX ); -if (nvec <0 ) { -cpus_read_unlock (); -return nvec; - } -if (nvec <=num_online_cpus ()) - start_irq_index =0 ; - - irqs =kmalloc_array ((nvec - start_irq_index),sizeof (int ),GFP_KERNEL ); + irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL );if (!irqs) { err = -ENOMEM ; goto free_irq_vector; } - gc->irq_contexts =kcalloc (nvec,sizeof (struct gdma_irq_context ), -GFP_KERNEL ); -if (!gc->irq_contexts) { +/* +* While processing the next pci irq vector, we start with index 1, +* as IRQ vector at index 0 is already processed for HWC. +* However, the population of irqs array starts with index 0, to be +* further used in irq_setup() +*/ +for (i =1 ; i <= nvec; i++) { + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events; +INIT_LIST_HEAD (&gic->eq_list); +spin_lock_init (&gic->lock); + +snprintf (gic->name,MANA_IRQ_NAME_SZ ,"mana_q%d@pci:%s" , + i -1 ,pci_name (pdev)); + +/* one pci vector is already allocated for HWC */ + irqs[i -1 ] =pci_irq_vector (pdev, i); +if (irqs[i -1 ] <0 ) { + err = irqs[i -1 ]; + goto free_current_gic; + } + + err =request_irq (irqs[i -1 ], mana_gd_intr,0 , gic->name, gic); +if (err) + goto free_current_gic; + +xa_store (&gc->irq_contexts, i, gic,GFP_KERNEL ); + } + +/* +* When calling irq_setup() for dynamically added IRQs, if number of +* CPUs is more than or equal to allocated MSI-X, we need to skip the +* first CPU sibling group since they are already affinitized to HWC IRQ +*/ +if (gc->num_msix_usable <=num_online_cpus ()) + skip_first_cpu = true; + + err =irq_setup (irqs, nvec, gc->numa_node, skip_first_cpu); +if (err) + goto free_irq; + +cpus_read_unlock (); +kfree (irqs); +return 0 ; + +free_current_gic: +kfree (gic); +free_irq: +for (i -=1 ; i >0 ; i--) { + irq =pci_irq_vector (pdev, i); + gic =xa_load (&gc->irq_contexts, i); +if (WARN_ON (!gic)) +continue ; + +irq_update_affinity_hint (irq,NULL ); +free_irq (irq, gic); +xa_erase (&gc->irq_contexts, i); +kfree (gic); + } +kfree (irqs); +free_irq_vector: +cpus_read_unlock (); +return err; +}drivers/net/ethernet/microsoft/mana/gdma_main.c static int irq_setup(unsigned int *irqs, unsigned int len, int node, · line 1475
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +10
-0+ +static int mana_gd_setup_irqs (struct pci_dev *pdev,int nvec) +{ +struct gdma_context *gc =pci_get_drvdata (pdev); +struct gdma_irq_context *gic; +int *irqs, *start_irqs, irq; +unsigned int cpu; +int err, i; + +cpus_read_lock ();Same here
Quoted 18 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +9
-2+ irqs =kmalloc_array (nvec,sizeof (int ),GFP_KERNEL ); +if (!irqs) { err = -ENOMEM ; - goto free_irq_array; + goto free_irq_vector; }for (i =0 ; i < nvec; i++) { - gic = &gc->irq_contexts[i]; + gic =kzalloc (sizeof (*gic),GFP_KERNEL ); +if (!gic) { + err = -ENOMEM ; + goto free_irq; + } + gic->handler = mana_gd_process_eq_events;INIT_LIST_HEAD (&gic->eq_list);spin_lock_init (&gic->lock);<snip>
Quoted 1 line
- Saurabh
Thanks Saurabh, will get these too
Simon Horman
reviewerQuoted 13 lines · Shradha Gupta
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.
Quoted 8 lines
The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
Hi Shradha,
It's unclear what the target tree for this patch-set is. But if it is net-next, which seems likely given the code under drivers/net/, then:
Please include that target in the subject of each patch in the patch-set.
Subject: [PATCH v5 net-next 0/5] ...
And, moreover, ...
## Form letter - net-next-closed
The merge window for v6.16 has begun and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.
Please repost when net-next reopens after June 8th.
RFC patches sent for review only are obviously welcome at any time.
Quoted 37 lines · Simon Horman
Quoted 13 lines · Shradha Gupta
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.Quoted 8 lines
The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
Hi Shradha,
It's unclear what the target tree for this patch-set is. But if it is net-next, which seems likely given the code under
drivers/net/, then:Please include that target in the subject of each patch in the patch-set.
Subject: [PATCH v5 net-next 0/5] ...And, moreover, ...
## Form letter - net-next-closed
Quoted 7 lines
The merge window for v6.16 has begun and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.
Please repost when net-next reopens after June 8th.
RFC patches sent for review only are obviously welcome at any time.
Thank you Simon.
While posting this patchset I was a bit confused about what should be the target tree. That's why in the cover letter of the V1 for this series, I had requested more clarity on the same (since there are patches from PCI and net-next both).
In such cases how do we decide which tree to target?
Also, noted about the next merge window for net-next :-)
Regards,
Shradha.
Simon Horman
reviewerQuoted 46 lines · Shradha Gupta
Quoted 37 lines · Simon Horman
Quoted 13 lines · Shradha Gupta
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.Quoted 8 lines
The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
Hi Shradha,
It's unclear what the target tree for this patch-set is. But if it is net-next, which seems likely given the code under
drivers/net/, then:Please include that target in the subject of each patch in the patch-set.
Subject: [PATCH v5 net-next 0/5] ...And, moreover, ...
## Form letter - net-next-closed
Quoted 7 lines
The merge window for v6.16 has begun and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.
Please repost when net-next reopens after June 8th.
RFC patches sent for review only are obviously welcome at any time.
Quoted 8 lines
Thank you Simon.
While posting this patchset I was a bit confused about what should be the target tree. That's why in the cover letter of the V1 for this series, I had requested more clarity on the same (since there are patches from PCI and net-next both).
In such cases how do we decide which tree to target?
Yes, that isn't entirely clear to me either.
Hopefully the maintainers can negotiate this.
Quoted 6 lines
Also, noted about the next merge window for net-next :-)
Regards,
Shradha.
Quoted 54 lines · Simon Horman
Quoted 53 lines · Shradha Gupta
Quoted 37 lines · Simon Horman
Quoted 13 lines · Shradha Gupta
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.Quoted 8 lines
The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
Hi Shradha,
It's unclear what the target tree for this patch-set is. But if it is net-next, which seems likely given the code under
drivers/net/, then:Please include that target in the subject of each patch in the patch-set.
Subject: [PATCH v5 net-next 0/5] ...And, moreover, ...
## Form letter - net-next-closed
Quoted 7 lines
The merge window for v6.16 has begun and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.
Please repost when net-next reopens after June 8th.
RFC patches sent for review only are obviously welcome at any time.
Thank you Simon.
While posting this patchset I was a bit confused about what should be the target tree. That's why in the cover letter of the V1 for this series, I had requested more clarity on the same (since there are patches from PCI and net-next both).
Quoted 5 lines
Quoted 1 line
In such cases how do we decide which tree to target?
Yes, that isn't entirely clear to me either.
Hopefully the maintainers can negotiate this.
Thanks Simon. also since teh target tree is not entirely clear, can I still send out an updated version with suggested changes?
Quoted 6 lines
Quoted 6 lines
Also, noted about the next merge window for net-next :-)
Regards,
Shradha.
Quoted 46 lines · Shradha Gupta
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
---
Change in v4
* add a patch describing the functionality ofirq_setup()through a comment
* Inirq_setup(), avoid using a labelnext_cpumask:
* modify the changes in MANA patch about restructuring the error handling path inmana_gd_setup_dyn_irqs()
* modify themana_gd_setup_irqs()to simplify handling aroundstart_irq_index
* add warning if an invalid gic is returned
* place thexa_destroy()cleanup inmana_gd_remove
---
Changes in v3
* split the 3rd patch into preparation patch aroundirq_setup()and changes in mana driver to allow dynamic IRQ allocation
* Add arm64 support for dynamic MSI-X allocation inpci_hypervcontroller
---
Changes in v2
* split the first patch intotwo(exporting the preapre_desc func and using the function and flag in pci-hyperv)
* replace 'pci vectors' by 'MSI-X vectors'
* Change the cover letter description to align with changes made
---
Quoted 6 lines
Shradha Gupta (5):
PCI/MSI: Exportpci_msix_prepare_desc()for dynamic MSI-X allocations
PCI: hv: Allow dynamic MSI-X vector allocation
net: mana: explainirq_setup()algorithm
net: mana: Allowirq_setup()to skip cpus for affinity
net: mana: Allocate MSI-X vectors dynamically
In this patchset, base-commit seems missing.
Please see this link:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.15#n868
"
When you open ``outgoing/0000-cover-letter.patch`` for editing, you will notice that it will have the ``base-commit:`` trailer at the very bottom, which provides the reviewer and the CI tools enough information to properly perform ``git am`` without worrying about conflicts::
"
When creating patches:
"
git format-patch --base=main origin/main
"
This will include a base-commit: line in each patch file:
"
base-commit: abcdef1234567890...
"
This is useful when submitting patches to mailing lists or other tooling.
Please follow the submitting-patches.rst to add base-commit.
Best Regards,
Zhu Yanjun
Quoted 8 lines
.../net/ethernet/microsoft/mana/gdma_main.c | 356 ++++++++++++++---- drivers/pci/controller/pci-hyperv.c | 5 +- drivers/pci/msi/irqdomain.c | 5 +- include/linux/msi.h | 2 + include/net/mana/gdma.h | 8 +- 5 files changed, 293 insertions(+), 83 deletions(-)
Quoted 77 lines · Zhu Yanjun
??? 2025/5/27 17:57, Shradha Gupta ??????:
Quoted 46 lines
In this patchset we want to enable the MANA driver to be able to allocate MSI-X vectors in PCI dynamically.
The first patch exports
pci_msix_prepare_desc()in PCI to be able to correctly prepare descriptors for dynamically added MSI-X vectors.The second patch adds the support of dynamic vector allocation in pci-hyperv PCI controller by enabling the
MSI_FLAG_PCI_MSIX_ALLOC_DYNflag and using thepci_msix_prepare_desc()exported in first patch.The third patch adds a detailed description of the
irq_setup(), to help understand the function design better.The fourth patch is a preparation patch for mana changes to support dynamic IRQ allocation. It contains changes in
irq_setup()to allow skipping first sibling CPU sets, in case certain IRQs are already affinitized to them.The fifth patch has the changes in MANA driver to be able to allocate MSI-X vectors dynamically. If the support does not exist it defaults to older behavior.
---
Change in v4
* add a patch describing the functionality ofirq_setup()through a comment
* Inirq_setup(), avoid using a labelnext_cpumask:
* modify the changes in MANA patch about restructuring the error handling path inmana_gd_setup_dyn_irqs()
* modify themana_gd_setup_irqs()to simplify handling aroundstart_irq_index
* add warning if an invalid gic is returned
* place thexa_destroy()cleanup inmana_gd_remove
---
Changes in v3
* split the 3rd patch into preparation patch aroundirq_setup()and changes in mana driver to allow dynamic IRQ allocation
* Add arm64 support for dynamic MSI-X allocation inpci_hypervcontroller
---
Changes in v2
* split the first patch intotwo(exporting the preapre_desc func and using the function and flag in pci-hyperv)
* replace 'pci vectors' by 'MSI-X vectors'
* Change the cover letter description to align with changes made
---Quoted 6 lines
Shradha Gupta (5):
PCI/MSI: Exportpci_msix_prepare_desc()for dynamic MSI-X allocations
PCI: hv: Allow dynamic MSI-X vector allocation
net: mana: explainirq_setup()algorithm
net: mana: Allowirq_setup()to skip cpus for affinity
net: mana: Allocate MSI-X vectors dynamicallyIn this patchset, base-commit seems missing.
Please see this link:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.15#n868"
When you open ``outgoing/0000-cover-letter.patch`` for editing, you will notice that it will have the ``base-commit:`` trailer at the very bottom, which provides the reviewer and the CI tools enough information to properly perform ``git am`` without worrying about conflicts::
"When creating patches:
"
git format-patch --base=main origin/main
"This will include a base-commit: line in each patch file:
"
base-commit:abcdef1234567890...
"
Quoted 7 lines
This is useful when submitting patches to mailing lists or other tooling.
Please follow the
submitting-patches.rstto add base-commit.Best Regards,
Zhu Yanjun
Thank you, I will make the necessary changes in the next version.
Regards,
Shradha.
Quoted 8 lines
Quoted 8 lines
.../net/ethernet/microsoft/mana/gdma_main.c | 356 ++++++++++++++---- drivers/pci/controller/pci-hyperv.c | 5 +- drivers/pci/msi/irqdomain.c | 5 +- include/linux/msi.h | 2 + include/net/mana/gdma.h | 8 +- 5 files changed, 293 insertions(+), 83 deletions(-)