irqchip/gic-v3-its: Balance initial LPI affinity across CPUs
Subsystems (2)
Marc Zyngier
maintainerPATCH v2 1/1 irqchip/gic-v3-its: Balance initial LPI affinity across CPUs
When mapping a LPI, the ITS driver picks the first possible affinity, which is in most cases CPU0, assuming that if that's not suitable, someone will come and set the affinity to something more interesting.
It apparently isn't the case, and people complain of poor performance when many interrupts are glued to the same CPU. So let's place the interrupts by finding the "least loaded" CPU (that is, the one that has the fewer LPIs mapped to it). So called 'managed' interrupts are an interesting case where the affinity is actually dictated by the kernel itself, and we should honor this.
---
Reviving this at John's request. The major change is that the affinity follows the x86 model, as described by Thomas. I expect this to have an impact on platforms like D05, where the SAS driver cannot use managed affinity just yet.
drivers/irqchip/irq-gic-v3-its.c | 125 ++++++++++++++++++++++++------- 1 file changed, 99 insertions(+), 26 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +99 -26
@@ -189,6 +189,8 @@ static DEFINE_IDA(its_vpeid_ida);#define gic_data_rdist_rd_base () (gic_data_rdist()->rd_base)#define gic_data_rdist_vlpi_base () (gic_data_rdist_rd_base() + SZ_128K) +static DEFINE_PER_CPU(atomic_t, cpu_lpi_count); +static u16 get_its_list (struct its_vm *vm) {struct its_node *its;
@@ -1500,35 +1502,113 @@ static void its_unmask_irq(struct irq_data *d)lpi_update_config (d,0 ,LPI_PROP_ENABLED ); } +static unsigned int cpumask_pick_least_loaded (const struct cpumask *cpu_mask) +{ +unsigned int cpu = nr_cpu_ids, tmp; +int count =S32_MAX ; + +for_each_cpu (tmp, cpu_mask) { +int this_count =atomic_read (per_cpu_ptr (&cpu_lpi_count, tmp)); +if (this_count < count) { + cpu = tmp; + count = this_count; + } + } + +return cpu; +} + +/* +* As suggested by Thomas Gleixner in: +* https://lore.kernel.org/r/[email protected] +*/ +static int its_select_cpu (struct irq_data *d, +const struct cpumask *aff_mask) +{ +struct its_device *its_dev =irq_data_get_irq_chip_data (d); +cpumask_var_t tmpmask; +int cpu, node; + +if (!alloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ; + + node = its_dev->its->numa_node; + +if (!irqd_affinity_is_managed (d)) { +/* First try the NUMA node */ +if (node !=NUMA_NO_NODE ) { +/* +* Try the intersection of the affinity mask and the +* node mask (and the online mask, just to be safe). +*/ +cpumask_and (tmpmask,cpumask_of_node (node), aff_mask); +cpumask_and (tmpmask, tmpmask, cpu_online_mask); + +/* If that doesn't work, try the nodemask itself */ +if (cpumask_empty (tmpmask)) +cpumask_and (tmpmask,cpumask_of_node (node), cpu_online_mask); + + cpu =cpumask_pick_least_loaded (tmpmask); +if (cpu < nr_cpu_ids) + goto out; + +/* If we can't cross sockets, give up */ +if ((its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 )) + goto out; + +/* If the above failed, expand the search */ + } + +/* Try the intersection of the affinity and online masks */ +cpumask_and (tmpmask, aff_mask, cpu_online_mask); + +/* If that doesn't fly, the online mask is the last resort */ +if (cpumask_empty (tmpmask)) +cpumask_copy (tmpmask, cpu_online_mask); + + cpu =cpumask_pick_least_loaded (tmpmask); + }else { +cpumask_and (tmpmask,irq_data_get_affinity_mask (d), cpu_online_mask); + +/* If we cannot cross sockets, limit the search to that node */ +if ((its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) && + node !=NUMA_NO_NODE ) +cpumask_and (tmpmask, tmpmask,cpumask_of_node (node)); + + cpu =cpumask_pick_least_loaded (tmpmask); + } +out: +free_cpumask_var (tmpmask); + +pr_debug ("IRQ%d -> %*pbl CPU%d\n" , d->irq,cpumask_pr_args (aff_mask), cpu); +return cpu; +} +static int its_set_affinity (struct irq_data *d,const struct cpumask *mask_val,bool force) { -unsigned int cpu; -const struct cpumask *cpu_mask = cpu_online_mask;struct its_device *its_dev =irq_data_get_irq_chip_data (d);struct its_collection *target_col;u32 id =its_get_event_id (d); +int cpu;/* A forwarded interrupt should use irq_set_vcpu_affinity */ if (irqd_is_forwarded_to_vcpu (d))return -EINVAL ; -/* lpi cannot be routed to a redistributor that is on a foreign node */ -if (its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) { -if (its_dev->its->numa_node >=0 ) { - cpu_mask =cpumask_of_node (its_dev->its->numa_node); -if (!cpumask_intersects (mask_val, cpu_mask)) -return -EINVAL ; - } - } - - cpu =cpumask_any_and (mask_val, cpu_mask); +if (!force) + cpu =its_select_cpu (d, mask_val); +else + cpu =cpumask_pick_least_loaded (mask_val); -if (cpu >= nr_cpu_ids) +if (cpu <0 || cpu >= nr_cpu_ids)return -EINVAL ;/* don't set the affinity when the target cpu is same as current one */ if (cpu != its_dev->event_map.col_map[id]) { +atomic_inc (per_cpu_ptr (&cpu_lpi_count, cpu)); +atomic_dec (per_cpu_ptr (&cpu_lpi_count, + its_dev->event_map.col_map[id])); target_col = &its_dev->its->collections[cpu];its_send_movi (its_dev, target_col, id); its_dev->event_map.col_map[id] = cpu;
@@ -3390,22 +3470,13 @@ static int its_irq_domain_activate(struct irq_domain *domain, {struct its_device *its_dev =irq_data_get_irq_chip_data (d);u32 event =its_get_event_id (d); -const struct cpumask *cpu_mask = cpu_online_mask;int cpu; -/* get the cpu_mask of local node */ -if (its_dev->its->numa_node >=0 ) - cpu_mask =cpumask_of_node (its_dev->its->numa_node); - -/* Bind the LPI to the first possible CPU */ - cpu =cpumask_first_and (cpu_mask, cpu_online_mask); -if (cpu >= nr_cpu_ids) { -if (its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) -return -EINVAL ; - - cpu =cpumask_first (cpu_online_mask); - } + cpu =its_select_cpu (d, cpu_online_mask); +if (cpu <0 || cpu >= nr_cpu_ids) +return -EINVAL ; +atomic_inc (per_cpu_ptr (&cpu_lpi_count, cpu)); its_dev->event_map.col_map[event] = cpu;irq_data_update_effective_affinity (d,cpumask_of (cpu));
@@ -3420,6 +3491,8 @@ static void its_irq_domain_deactivate(struct irq_domain *domain,struct its_device *its_dev =irq_data_get_irq_chip_data (d);u32 event =its_get_event_id (d); +atomic_dec (per_cpu_ptr (&cpu_lpi_count, + its_dev->event_map.col_map[event]));/* Stop the delivery of interrupts */ its_send_discard (its_dev, event); }
Signature · 1 line
2.20.1
On 12/03/2020 11:55, Marc Zyngier wrote:
Hi Marc,
Quoted 13 lines
When mapping a LPI, the ITS driver picks the first possible affinity, which is in most cases CPU0, assuming that if that's not suitable, someone will come and set the affinity to something more interesting.
It apparently isn't the case, and people complain of poor performance when many interrupts are glued to the same CPU. So let's place the interrupts by finding the "least loaded" CPU (that is, the one that has the fewer LPIs mapped to it). So called 'managed' interrupts are an interesting case where the affinity is actually dictated by the kernel itself, and we should honor this.
Quoted 7 lines
Reported-by John Garry <[email protected]>Signed-off-by Marc Zyngier <[email protected]>Cc John Garry <[email protected]>Cc Ming Lei <[email protected]>---
Reviving this at John's request.
Thanks very much. I may request a colleague test this due to possible precautionary office closure.
The major change is that the
Quoted 1 line
affinity follows the x86 model, as described by Thomas.
There seems to be a subtle difference between this implementation and what Thomas described for managed interrupts handling on x86. That being, managed interrupt loading is counted separately to total interrupts per CPU for x86. That seems quite important so that we spread managed interrupts evenly.
Quoted 2 lines
I expect this to have an impact on platforms like D05, where the SAS driver cannot use managed affinity just yet.
I need some blk-mq and SCSI changes to go in first to improve the interrupt handling there, hopefully we can make progress on that soon.
Thanks,
John
Quoted 182 lines
drivers/irqchip/irq-gic-v3-its.c | 125 ++++++++++++++++++++++++------- 1 file changed, 99 insertions(+), 26 deletions(-)drivers/irqchip/irq-gic-v3-its.c +93
-13@@ -189,6 +189,8 @@ static DEFINE_IDA(its_vpeid_ida);#define gic_data_rdist_rd_base () (gic_data_rdist()->rd_base)#define gic_data_rdist_vlpi_base () (gic_data_rdist_rd_base() + SZ_128K) +static DEFINE_PER_CPU(atomic_t, cpu_lpi_count); +static u16 get_its_list (struct its_vm *vm) {struct its_node *its;@@ -1500,35 +1502,113 @@ static void its_unmask_irq(struct irq_data *d)lpi_update_config (d,0 ,LPI_PROP_ENABLED ); } +static unsigned int cpumask_pick_least_loaded (const struct cpumask *cpu_mask) +{ +unsigned int cpu = nr_cpu_ids, tmp; +int count =S32_MAX ; + +for_each_cpu (tmp, cpu_mask) { +int this_count =atomic_read (per_cpu_ptr (&cpu_lpi_count, tmp)); +if (this_count < count) { + cpu = tmp; + count = this_count; + } + } + +return cpu; +} + +/* +* As suggested by Thomas Gleixner in: +* https://lore.kernel.org/r/[email protected] +*/ +static int its_select_cpu (struct irq_data *d, +const struct cpumask *aff_mask) +{ +struct its_device *its_dev =irq_data_get_irq_chip_data (d); +cpumask_var_t tmpmask; +int cpu, node; + +if (!alloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ; + + node = its_dev->its->numa_node; + +if (!irqd_affinity_is_managed (d)) { +/* First try the NUMA node */ +if (node !=NUMA_NO_NODE ) { +/* +* Try the intersection of the affinity mask and the +* node mask (and the online mask, just to be safe). +*/ +cpumask_and (tmpmask,cpumask_of_node (node), aff_mask); +cpumask_and (tmpmask, tmpmask, cpu_online_mask); + +/* If that doesn't work, try the nodemask itself */ +if (cpumask_empty (tmpmask)) +cpumask_and (tmpmask,cpumask_of_node (node), cpu_online_mask); + + cpu =cpumask_pick_least_loaded (tmpmask); +if (cpu < nr_cpu_ids) + goto out; + +/* If we can't cross sockets, give up */ +if ((its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 )) + goto out; + +/* If the above failed, expand the search */ + } + +/* Try the intersection of the affinity and online masks */ +cpumask_and (tmpmask, aff_mask, cpu_online_mask); + +/* If that doesn't fly, the online mask is the last resort */ +if (cpumask_empty (tmpmask)) +cpumask_copy (tmpmask, cpu_online_mask); + + cpu =cpumask_pick_least_loaded (tmpmask); + }else { +cpumask_and (tmpmask,irq_data_get_affinity_mask (d), cpu_online_mask); + +/* If we cannot cross sockets, limit the search to that node */ +if ((its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) && + node !=NUMA_NO_NODE ) +cpumask_and (tmpmask, tmpmask,cpumask_of_node (node)); + + cpu =cpumask_pick_least_loaded (tmpmask); + } +out: +free_cpumask_var (tmpmask); + +pr_debug ("IRQ%d -> %*pbl CPU%d\n" , d->irq,cpumask_pr_args (aff_mask), cpu); +return cpu; +} +static int its_set_affinity (struct irq_data *d,const struct cpumask *mask_val,bool force) { -unsigned int cpu; -const struct cpumask *cpu_mask = cpu_online_mask;struct its_device *its_dev =irq_data_get_irq_chip_data (d);struct its_collection *target_col;u32 id =its_get_event_id (d); +int cpu;/* A forwarded interrupt should use irq_set_vcpu_affinity */ if (irqd_is_forwarded_to_vcpu (d))return -EINVAL ; -/* lpi cannot be routed to a redistributor that is on a foreign node */ -if (its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) { -if (its_dev->its->numa_node >=0 ) { - cpu_mask =cpumask_of_node (its_dev->its->numa_node); -if (!cpumask_intersects (mask_val, cpu_mask)) -return -EINVAL ; - } - } - - cpu =cpumask_any_and (mask_val, cpu_mask); +if (!force) + cpu =its_select_cpu (d, mask_val); +else + cpu =cpumask_pick_least_loaded (mask_val); -if (cpu >= nr_cpu_ids) +if (cpu <0 || cpu >= nr_cpu_ids)return -EINVAL ;/* don't set the affinity when the target cpu is same as current one */ if (cpu != its_dev->event_map.col_map[id]) { +atomic_inc (per_cpu_ptr (&cpu_lpi_count, cpu)); +atomic_dec (per_cpu_ptr (&cpu_lpi_count, + its_dev->event_map.col_map[id])); target_col = &its_dev->its->collections[cpu];its_send_movi (its_dev, target_col, id);its_dev->event_map.col_map[id] = cpu;drivers/irqchip/irq-gic-v3-its.c +6
-13@@ -3390,22 +3470,13 @@ static int its_irq_domain_activate(struct irq_domain *domain, {struct its_device *its_dev =irq_data_get_irq_chip_data (d);u32 event =its_get_event_id (d); -const struct cpumask *cpu_mask = cpu_online_mask;int cpu; -/* get the cpu_mask of local node */ -if (its_dev->its->numa_node >=0 ) - cpu_mask =cpumask_of_node (its_dev->its->numa_node); - -/* Bind the LPI to the first possible CPU */ - cpu =cpumask_first_and (cpu_mask, cpu_online_mask); -if (cpu >= nr_cpu_ids) { -if (its_dev->its->flags &ITS_FLAGS_WORKAROUND_CAVIUM_23144 ) -return -EINVAL ; - - cpu =cpumask_first (cpu_online_mask); - } + cpu =its_select_cpu (d, cpu_online_mask); +if (cpu <0 || cpu >= nr_cpu_ids) +return -EINVAL ; +atomic_inc (per_cpu_ptr (&cpu_lpi_count, cpu)); its_dev->event_map.col_map[event] = cpu;irq_data_update_effective_affinity (d,cpumask_of (cpu));@@ -3420,6 +3491,8 @@ static void its_irq_domain_deactivate(struct irq_domain *domain,struct its_device *its_dev =irq_data_get_irq_chip_data (d);u32 event =its_get_event_id (d); +atomic_dec (per_cpu_ptr (&cpu_lpi_count, + its_dev->event_map.col_map[event]));/* Stop the delivery of interrupts */ its_send_discard (its_dev, event); }
Marc Zyngier
maintainerHi John,
Quoted 25 lines · John Garry
On 12/03/2020 11:55, Marc Zyngier wrote:
Hi Marc,
Quoted 13 lines
When mapping a LPI, the ITS driver picks the first possible affinity, which is in most cases CPU0, assuming that if that's not suitable, someone will come and set the affinity to something more interesting.
It apparently isn't the case, and people complain of poor performance when many interrupts are glued to the same CPU. So let's place the interrupts by finding the "least loaded" CPU (that is, the one that has the fewer LPIs mapped to it). So called 'managed' interrupts are an interesting case where the affinity is actually dictated by the kernel itself, and we should honor this.
Quoted 7 lines
Reported-by John Garry <[email protected]>Signed-off-by Marc Zyngier <[email protected]>Cc John Garry <[email protected]>Cc Ming Lei <[email protected]>---
Reviving this at John's request.
Quoted 2 lines
Thanks very much. I may request a colleague test this due to possible precautionary office closure.
Huh. Not great... :-(
Quoted 9 lines
The major change is that the
Quoted 1 line
affinity follows the x86 model, as described by Thomas.
There seems to be a subtle difference between this implementation and what Thomas described for managed interrupts handling on x86. That being, managed interrupt loading is counted separately to total interrupts per CPU for x86. That seems quite important so that we spread managed interrupts evenly.
Hmmm. Yes. That'd require a separate per-CPU counter. Nothing too invasive though. I'll roll that in soon. I still wonder about interaction of collocated managed and non-managed interrupts, but we can cross that bridge later.
Quoted 5 lines
Quoted 2 lines
I expect this to have an impact on platforms like D05, where the SAS driver cannot use managed affinity just yet.
I need some blk-mq and SCSI changes to go in first to improve the interrupt handling there, hopefully we can make progress on that soon.
That'd be good.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Hi Marc,
Quoted 37 lines
Quoted 16 lines
Quoted 16 lines
When mapping a LPI, the ITS driver picks the first possible affinity, which is in most cases CPU0, assuming that if that's not suitable, someone will come and set the affinity to something more interesting.
It apparently isn't the case, and people complain of poor performance when many interrupts are glued to the same CPU. So let's place the interrupts by finding the "least loaded" CPU (that is, the one that has the fewer LPIs mapped to it). So called 'managed' interrupts are an interesting case where the affinity is actually dictated by the kernel itself, and we should honor this.
Quoted 8 lines
Quoted 5 lines
Signed-off-by Marc Zyngier <[email protected]>Cc John Garry <[email protected]>Cc Ming Lei <[email protected]>---
Reviving this at John's request.Thanks very much. I may request a colleague test this due to possible precautionary office closure.
Huh. Not great... :-(
Quoted 9 lines
The major change is that the
Quoted 1 line
affinity follows the x86 model, as described by Thomas.
There seems to be a subtle difference between this implementation and what Thomas described for managed interrupts handling on x86. That being, managed interrupt loading is counted separately to total interrupts per CPU for x86. That seems quite important so that we spread managed interrupts evenly.
Quoted 3 lines
Hmmm. Yes. That'd require a separate per-CPU counter. Nothing too invasive though. I'll roll that in soon. I still wonder about interaction of collocated managed and non-managed interrupts, but we can cross that bridge later.
Great. And I think I may have mentioned this before (or I did and it was not a good idea), it now seems that we may be able to just leverage the generic matrix irq code here.
Cheers,
John