Skip to contentSkip to conversation

x86/irq: assign vectors from numa_node

PATCH Superseded Arthur Kepner 2 participants 5 messages Dec 3, 2010 – Dec 22, 2010 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/irq: assign vectors from numa_node

Several drivers (e.g., mlx4_core) do something similar to:

	err = pci_enable_msix(pdev, entries, num_possible_cpus());

which takes us down this code path:

pci_enable_msix
native_setup_msi_irqs
create_irq_nr
__assign_irq_vector

__assign_irq_vector() preferentially uses vectors from low-numbered CPUs. On a system with a large number (>256) CPUs this can result in a CPU running out of vectors, and subsequent attempts to assign an interrupt to that CPU will fail.

The following patch prefers vectors from the node associated with the device (if the device is associated with a node). This should make it far less likely that a single CPU's vectors will be exhausted.

---

 io_apic.c |   81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 77 insertions(+), 4 deletions(-)
arch/x86/kernel/apic/io_apic.c +77 -4
index 7cc0a72..af5f9d8 100644
@@ -1117,6 +1117,49 @@ next:
 	return err;
 }
 
+static int
+__assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+			 const struct cpumask *mask, int node)
+{
+	int err = -EAGAIN;
+	int cpu, best_cpu = -1, min_vector_count = NR_VECTORS;
+
+	for_each_cpu_and(cpu, cpumask_of_node(node), mask) {
+		/* find the 'best' CPU to take this vector -
+		 * the one with the fewest assigned vectors is
+		 * considered 'best' */
+		int i, vector_count = 0;
+
+		if (!cpu_online(cpu))
+			continue;
+
+		for (i = FIRST_EXTERNAL_VECTOR + VECTOR_OFFSET_START;
+		     i < NR_VECTORS ; i++)
+			if (per_cpu(vector_irq, cpu)[i] != -1)
+				vector_count++;
+
+		if (vector_count < min_vector_count) {
+			min_vector_count = vector_count;
+			best_cpu = cpu;
+		}
+	}
+
+	if (best_cpu >= 0) {
+		cpumask_var_t tmp_mask;
+
+		if (!alloc_cpumask_var(&tmp_mask, GFP_ATOMIC))
+			return -ENOMEM;
+
+		cpumask_clear(tmp_mask);
+		cpumask_set_cpu(best_cpu, tmp_mask);
+		err = __assign_irq_vector(irq, cfg, tmp_mask);
+
+		free_cpumask_var(tmp_mask);
+	}
+
+	return err;
+}
+
 int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 {
 	int err;
@@ -1128,6 +1171,39 @@ int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 	return err;
 }
 
+static int
+assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+		       const struct cpumask *mask, int node)
+{
+	int err;
+	unsigned long flags;
+
+	if (node == NUMA_NO_NODE)
+		return assign_irq_vector(irq, cfg, mask);
+
+	raw_spin_lock_irqsave(&vector_lock, flags);
+	err = __assign_irq_vector_node(irq, cfg, mask, node);
+	raw_spin_unlock_irqrestore(&vector_lock, flags);
+
+	if (err != 0)
+		/* uh oh - try again w/o specifying a node */
+		return assign_irq_vector(irq, cfg, mask);
+	else {
+		/* and set the affinity mask so that only
+		 * CPUs on 'node' will be used */
+		struct irq_desc *desc = irq_to_desc(irq);
+		unsigned long flags;
+
+		raw_spin_lock_irqsave(&desc->lock, flags);
+		cpumask_and(desc->irq_data.affinity, cpu_online_mask,
+			    cpumask_of_node(node));
+		desc->status |= IRQ_AFFINITY_SET;
+		raw_spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	return err;
+}
+
 static void __clear_irq_vector(int irq, struct irq_cfg *cfg)
 {
 	int cpu, vector;
@@ -3057,7 +3133,6 @@ device_initcall(ioapic_init_sysfs);
 unsigned int create_irq_nr(unsigned int from, int node)
 {
 	struct irq_cfg *cfg;
-	unsigned long flags;
 	unsigned int ret = 0;
 	int irq;
 
@@ -3073,10 +3148,8 @@ unsigned int create_irq_nr(unsigned int from, int node)
 		return 0;
 	}
 
-	raw_spin_lock_irqsave(&vector_lock, flags);
-	if (!__assign_irq_vector(irq, cfg, apic->target_cpus()))
+	if (!assign_irq_vector_node(irq, cfg, apic->target_cpus(), node))
 		ret = irq;
-	raw_spin_unlock_irqrestore(&vector_lock, flags);
 
 	if (ret) {
 		set_irq_chip_data(irq, cfg);
#1 lore raw
Quoted 4 lines · Arthur Kepner

Several drivers (e.g., mlx4_core) do something similar to:

	err = pci_enable_msix(pdev, entries, num_possible_cpus());

Which is the main problem and this patch just addresses the symptoms. As Eric pointed out earlier, this needs to be solved differently.

There is absolutely no point assigning 4096 interrupts to a single node in the first place. Especially not, if we end up using only a few of them in the end. And those you use are not necessarily on that very node.

I understand, that you want to work around your problem at hand, but I'm pushing back on it, as it's a crappy solution which just ignores the underlying problem. You don't even mention it in your changelog that this is a work around and not a solution.

No, you just ignore that fact and the requests to look at the underlying problem.

Quoted 11 lines

which takes us down this code path:

pci_enable_msix
native_setup_msi_irqs
create_irq_nr
__assign_irq_vector

__assign_irq_vector() preferentially uses vectors from low-numbered CPUs. On a system with a large number (>256) CPUs this can result in a CPU running out of vectors, and subsequent attempts to assign an interrupt to that CPU will fail.

I agree, that __assign_irq_vector() should honour the request for a node, but I don't agree, that we should magically spread stuff on whatever node we find, when the node ran out of vectors.

There is probably a reason, why you want an interrupt on a specific node and just silently pushing it somewhere else feels very wrong.

This needs to be solved from ground up with proper rules about failure modes and fallback decisions.

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+static int
+__assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+			 const struct cpumask *mask, int node)
+{
+	int err = -EAGAIN;
+	int cpu, best_cpu = -1, min_vector_count = NR_VECTORS;
+
+	for_each_cpu_and(cpu, cpumask_of_node(node), mask) {
+		/* find the 'best' CPU to take this vector -
+		 * the one with the fewest assigned vectors is

arch/x86/kernel/apic/io_apic.c next: · line 1139

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		 * considered 'best' */
+		int i, vector_count = 0;
+
+		if (!cpu_online(cpu))
+			continue;
+
+		for (i = FIRST_EXTERNAL_VECTOR + VECTOR_OFFSET_START;
+		     i < NR_VECTORS ; i++)
+			if (per_cpu(vector_irq, cpu)[i] != -1)
+				vector_count++;

Instead of having proper book keeping of vectors, we loop through nvec
* ncpus to figure that out ? And of course we run that loop with interrupts disabled and vector lock held.

arch/x86/kernel/apic/io_apic.c next: · line 1150

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		if (vector_count < min_vector_count) {
+			min_vector_count = vector_count;
+			best_cpu = cpu;
+		}
+	}
+
+	if (best_cpu >= 0) {
+		cpumask_var_t tmp_mask;
+
+		if (!alloc_cpumask_var(&tmp_mask, GFP_ATOMIC))

Bah. I made create_irq_nr() atomic region small with lots of effort and got rid of all GFP_ATOMIC allocations.

Quoted 35 lines
arch/x86/kernel/apic/io_apic.c +28 -0
+			return -ENOMEM;
+
+		cpumask_clear(tmp_mask);
+		cpumask_set_cpu(best_cpu, tmp_mask);
+		err = __assign_irq_vector(irq, cfg, tmp_mask);
+
+		free_cpumask_var(tmp_mask);
+	}
+
+	return err;
+}
+
 int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 {
 	int err;
@@ -1128,6 +1171,39 @@ int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 	return err;
 }
 
+static int
+assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+		       const struct cpumask *mask, int node)
+{
+	int err;
+	unsigned long flags;
+
+	if (node == NUMA_NO_NODE)
+		return assign_irq_vector(irq, cfg, mask);
+
+	raw_spin_lock_irqsave(&vector_lock, flags);
+	err = __assign_irq_vector_node(irq, cfg, mask, node);
+	raw_spin_unlock_irqrestore(&vector_lock, flags);
+
+	if (err != 0)
+		/* uh oh - try again w/o specifying a node */

arch/x86/kernel/apic/io_apic.c int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask) · line 1199

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		return assign_irq_vector(irq, cfg, mask);
+	else {
+		/* and set the affinity mask so that only
+		 * CPUs on 'node' will be used */
+		struct irq_desc *desc = irq_to_desc(irq);
+		unsigned long flags;
+
+		raw_spin_lock_irqsave(&desc->lock, flags);
+		cpumask_and(desc->irq_data.affinity, cpu_online_mask,
+			    cpumask_of_node(node));

Which leaves us with an empty affinity mask, when the last cpu of that node just went offline before locking desc->lock. Brilliant.

arch/x86/kernel/apic/io_apic.c int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask) · line 1201

Quoted 2 lines
arch/x86/kernel/apic/io_apic.c +2 -0
+		desc->status |= IRQ_AFFINITY_SET;
+		raw_spin_unlock_irqrestore(&desc->lock, flags);

Aside of that low level arch code is not supposed to fiddle in irq_desc at will excatly for such reasons.

As you might have noticed, i'm working on removing access to irq_desc from random places and I spent quite some effort to clean up the whole irq mess. No way to put crap like this in again, so I can twist my brain around it next week how to clean it up.

Thanks,

tglx

#2 lore raw
Quoted 4 lines · Thomas Gleixner
Quoted 3 lines · Arthur Kepner

Several drivers (e.g., mlx4_core) do something similar to:

Quoted 8 lines
Quoted 1 line
	err = pci_enable_msix(pdev, entries, num_possible_cpus());

Which is the main problem and this patch just addresses the symptoms. As Eric pointed out earlier, this needs to be solved differently.

There is absolutely no point assigning 4096 interrupts to a single node in the first place.

I'm not arguing that it's a good idea for a driver to request so many resources. But some do. And what we do now is even worse than assigning all the interrupts to a single node.

Quoted 3 lines

Especially not, if we end up using only a few
of them in the end. And those you use are not necessarily on that very node.

OK. Eric mentioned in a related thread that vector allocation might be deferred until request_irq(). That sounds like a good idea. But unless we change the way initial vector assignment is done, it just defers the problem (assuming that request_irq() is done for every irq allocated in create_irq_nr()).

Using the numa_node of the device to do the initial vector assignment seems like a reasonable default choice, no?

Quoted 133 lines

I understand, that you want to work around your problem at hand, but I'm pushing back on it, as it's a crappy solution which just ignores the underlying problem. You don't even mention it in your changelog that this is a work around and not a solution.

No, you just ignore that fact and the requests to look at the underlying problem.

Quoted 11 lines

which takes us down this code path:

pci_enable_msix
native_setup_msi_irqs
create_irq_nr
__assign_irq_vector

__assign_irq_vector() preferentially uses vectors from low-numbered CPUs. On a system with a large number (>256) CPUs this can result in a CPU running out of vectors, and subsequent attempts to assign an interrupt to that CPU will fail.

I agree, that __assign_irq_vector() should honour the request for a node, but I don't agree, that we should magically spread stuff on whatever node we find, when the node ran out of vectors.

There is probably a reason, why you want an interrupt on a specific node and just silently pushing it somewhere else feels very wrong.

This needs to be solved from ground up with proper rules about failure modes and fallback decisions.

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+static int
+__assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+			 const struct cpumask *mask, int node)
+{
+	int err = -EAGAIN;
+	int cpu, best_cpu = -1, min_vector_count = NR_VECTORS;
+
+	for_each_cpu_and(cpu, cpumask_of_node(node), mask) {
+		/* find the 'best' CPU to take this vector -
+		 * the one with the fewest assigned vectors is

arch/x86/kernel/apic/io_apic.c next: · line 1139

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		 * considered 'best' */
+		int i, vector_count = 0;
+
+		if (!cpu_online(cpu))
+			continue;
+
+		for (i = FIRST_EXTERNAL_VECTOR + VECTOR_OFFSET_START;
+		     i < NR_VECTORS ; i++)
+			if (per_cpu(vector_irq, cpu)[i] != -1)
+				vector_count++;

Instead of having proper book keeping of vectors, we loop through nvec
* ncpus to figure that out ? And of course we run that loop with interrupts disabled and vector lock held.

arch/x86/kernel/apic/io_apic.c next: · line 1150

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		if (vector_count < min_vector_count) {
+			min_vector_count = vector_count;
+			best_cpu = cpu;
+		}
+	}
+
+	if (best_cpu >= 0) {
+		cpumask_var_t tmp_mask;
+
+		if (!alloc_cpumask_var(&tmp_mask, GFP_ATOMIC))

Bah. I made create_irq_nr() atomic region small with lots of effort and got rid of all GFP_ATOMIC allocations.

Quoted 35 lines
arch/x86/kernel/apic/io_apic.c +28 -0
+			return -ENOMEM;
+
+		cpumask_clear(tmp_mask);
+		cpumask_set_cpu(best_cpu, tmp_mask);
+		err = __assign_irq_vector(irq, cfg, tmp_mask);
+
+		free_cpumask_var(tmp_mask);
+	}
+
+	return err;
+}
+
 int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 {
 	int err;
@@ -1128,6 +1171,39 @@ int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask)
 	return err;
 }
 
+static int
+assign_irq_vector_node(int irq, struct irq_cfg *cfg,
+		       const struct cpumask *mask, int node)
+{
+	int err;
+	unsigned long flags;
+
+	if (node == NUMA_NO_NODE)
+		return assign_irq_vector(irq, cfg, mask);
+
+	raw_spin_lock_irqsave(&vector_lock, flags);
+	err = __assign_irq_vector_node(irq, cfg, mask, node);
+	raw_spin_unlock_irqrestore(&vector_lock, flags);
+
+	if (err != 0)
+		/* uh oh - try again w/o specifying a node */

arch/x86/kernel/apic/io_apic.c int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask) · line 1199

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +10 -0
+		return assign_irq_vector(irq, cfg, mask);
+	else {
+		/* and set the affinity mask so that only
+		 * CPUs on 'node' will be used */
+		struct irq_desc *desc = irq_to_desc(irq);
+		unsigned long flags;
+
+		raw_spin_lock_irqsave(&desc->lock, flags);
+		cpumask_and(desc->irq_data.affinity, cpu_online_mask,
+			    cpumask_of_node(node));

Which leaves us with an empty affinity mask, when the last cpu of that node just went offline before locking desc->lock. Brilliant.

arch/x86/kernel/apic/io_apic.c int assign_irq_vector(int irq, struct irq_cfg *cfg, const struct cpumask *mask) · line 1201

Quoted 2 lines
arch/x86/kernel/apic/io_apic.c +2 -0
+		desc->status |= IRQ_AFFINITY_SET;
+		raw_spin_unlock_irqrestore(&desc->lock, flags);

Aside of that low level arch code is not supposed to fiddle in irq_desc at will excatly for such reasons.

As you might have noticed, i'm working on removing access to irq_desc from random places and I spent quite some effort to clean up the whole irq mess. No way to put crap like this in again, so I can twist my brain around it next week how to clean it up.

Thanks,

tglx

Signature · 1 line
Arthur
#3 lore raw
Quoted 11 lines · Arthur Kepner
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines · Arthur Kepner

Several drivers (e.g., mlx4_core) do something similar to:

	err = pci_enable_msix(pdev, entries, num_possible_cpus());

Which is the main problem and this patch just addresses the symptoms. As Eric pointed out earlier, this needs to be solved differently.

Quoted 5 lines
Quoted 2 lines

There is absolutely no point assigning 4096 interrupts to a single node in the first place.

I'm not arguing that it's a good idea for a driver to request so many resources. But some do. ...

Right, some do and we now make efforts to let them do that nonsense no matter what ?

Quoted 2 lines

.... And what we do now is even worse than
assigning all the interrupts to a single node.

We assign it to a single core, which is wrong. But what the heck is the difference between assinging 4k irqs to a single core or to a single node ?

Nothing. And that's the whole problem.

I agree that the per node assignement needs to be resolved, but not in the way that we just ignore the underlying problems and solve them at the wrong place. You did not answer my comment further down in the patch:

Quoted 9 lines
Quoted 9 lines

I agree, that __assign_irq_vector() should honour the request for a node, but I don't agree, that we should magically spread stuff on whatever node we find, when the node ran out of vectors.

There is probably a reason, why you want an interrupt on a specific node and just silently pushing it somewhere else feels very wrong.

This needs to be solved from ground up with proper rules about failure modes and fallback decisions.

Where is the answer to this ?

Quoted 7 lines
Quoted 3 lines

Especially not, if we end up using only a few
of them in the end. And those you use are not necessarily on that very node.

OK. Eric mentioned in a related thread that vector allocation might be deferred until request_irq(). That sounds like a good idea. But unless we change the way initial vector assignment is

It does NOT sound like a good idea. Again, I agree that we need to fix the per node assignment, but we need to discuss the problem of anything which goes beyond the node.

Quoted 2 lines

done, it just defers the problem (assuming that request_irq() is done for every irq allocated in create_irq_nr()).

There is neither a reason for a driver to create so many interrupts in the first place nor a re reason to request them right away. This is just horrible crap, nothing else. Why the hell would a driver need to startup 4k interrupts just to load itself w/o a single user?

We do _NOT_ work around such insanity somewhere else even if we can.

Quoted 2 lines

Using the numa_node of the device to do the initial vector assignment seems like a reasonable default choice, no?

I still agree that we should honour the node request, but that has a totally different scope than what your patch is trying to do.

Thanks,

tglx

#4 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 5 lines · Arthur Kepner
Quoted 3 lines · Thomas Gleixner

.....
There is absolutely no point assigning 4096 interrupts to a single node in the first place.

Quoted 5 lines
Quoted 2 lines

I'm not arguing that it's a good idea for a driver to request so many resources. But some do. ...

Right, some do and we now make efforts to let them do that nonsense no matter what ?

To me, it seems better (and easier) to let drivers do non-sensical things, than to determine what is sensible for a particluar driver.

How would you choose a limit of interrupt allocations?

Quoted 19 lines
Quoted 3 lines

.... And what we do now is even worse than
assigning all the interrupts to a single node.

We assign it to a single core, which is wrong. But what the heck is the difference between assinging 4k irqs to a single core or to a single node ?

Nothing. And that's the whole problem.

I agree that the per node assignement needs to be resolved, but not in the way that we just ignore the underlying problems and solve them at the wrong place. You did not answer my comment further down in the patch:

Quoted 4 lines
Quoted 4 lines

I agree, that __assign_irq_vector() should honour the request for a node, but I don't agree, that we should magically spread stuff on whatever node we find, when the node ran out of vectors.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

There is probably a reason, why you want an interrupt on a specific node and just silently pushing it somewhere else feels very wrong.

This needs to be solved from ground up with proper rules about failure modes and fallback decisions.

Where is the answer to this ?

The patch I sent made a 'best effort' to assign an interrupt to the node where the device is installed. If no vectors were available on that node, it tried elsewhere.

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

Especially not, if we end up using only a few
of them in the end. And those you use are not necessarily on that very node.

OK. Eric mentioned in a related thread that vector allocation might be deferred until request_irq(). That sounds like a good idea. But unless we change the way initial vector assignment is

It does NOT sound like a good idea.

This seems at odds with what you said here:

http://marc.info/?l=linux-kernel&m=128566136604152&w=2

(specifically this bit:

8<---------------------------- [snip] ----------------------------

..... So the solution would be:

create_irq allocates an irq number + irq descriptor, nothing else

chip->startup() will setup the vector and chip->shutdown releases it. ......

8<---------------------------- [snip] ----------------------------

)

Can you clarify?

Quoted 3 lines

... Again, I agree that we need to fix
the per node assignment, but we need to discuss the problem of anything which goes beyond the node.

Suppose that a interrupt is initially assigned to a different node than the device. Is that such a big deal? They can be redistributed later by a user-space irq balancer.

Quoted 17 lines
Quoted 3 lines

done, it just defers the problem (assuming that request_irq() is done for every irq allocated in create_irq_nr()).

There is neither a reason for a driver to create so many interrupts in the first place nor a re reason to request them right away. This is just horrible crap, nothing else. Why the hell would a driver need to startup 4k interrupts just to load itself w/o a single user?

We do _NOT_ work around such insanity somewhere else even if we can.

Quoted 2 lines

Using the numa_node of the device to do the initial vector assignment seems like a reasonable default choice, no?

I still agree that we should honour the node request, but that has a totally different scope than what your patch is trying to do.

Signature · 1 line
Arthur