Skip to contentSkip to conversation

x86: Add irq spillover warning

PATCH Superseded Neil Horman 2 participants 6 messages Jul 16, 2019 x86

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: Add irq spillover warning

On Intel hardware, cpus are limited in the number of irqs they can have affined to them (currently 240), based on section 10.5.2 of:
https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf

If a cpu has more than this number of interrupts affined to it, they will spill over to other cpus, which potentially may be outside of their affinity mask. Given that this might cause unexpected behavior on performance sensitive systems, warn the user should this condition occur so that corrective action can be taken

Reported-by [email protected]
CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
---
 arch/x86/kernel/irq.c | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/kernel/irq.c +8 -0
index 9b68b5b00ac9..ac7ed32de3d5 100644
@@ -244,6 +244,14 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
 
 	desc = __this_cpu_read(vector_irq[vector]);
 
+	/*
+	 * Intel processors are limited in the number of irqs they can address. If we affine
+	 * too many irqs to a given cpu, they can silently spill to another cpu outside of
+	 * their affinity mask. Warn the user when this occurs
+	 */
+	if (unlikely(!cpumask_test_cpu(smp_processor_id(), &desc->irq_common_data.affinity)))
+		pr_emerg_ratelimited("%s: %d.%d handled outside of affinity mask\n");
+
 	if (!handle_irq(desc, regs)) {
 		ack_APIC_irq();
 
Signature · 1 line
2.21.0
#1 lore raw

Neil,

Quoted 3 lines · Neil Horman

On Intel hardware, cpus are limited in the number of irqs they can have affined to them (currently 240), based on section 10.5.2 of:
https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf

That reference is really not useful to explain the problem and the number of vectors is neither. Please explain the conceptual issue.

Quoted 3 lines

If a cpu has more than this number of interrupts affined to it, they will spill over to other cpus, which potentially may be outside of their affinity mask.

Spill over?

The kernel decides to pick a vector on a CPU outside of the affinity when it runs out of vectors on the CPUs in the affinity mask.

Please explain issues technically correct.

Review __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Quoted 5 lines

Given that this might cause unexpected behavior on performance sensitive systems, warn the user should this condition occur so that corrective action can be taken

(quoted diff) +0 -0
@@ -244,6 +244,14 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Why on earth warn in the interrupt delivery hotpath? Just because it's the place which really needs extra instructions and extra cache lines on performance sensitive systems, right?

The fact that the kernel ran out of vectors for the CPUs in the affinity mask is already known when the vector is allocated in activate_reserved().

So there is an obvious place to put such a warning and it's certainly not do_IRQ().

Thanks,

tglx

#2 lore raw
Quoted 11 lines · Thomas Gleixner

Neil,

Quoted 3 lines · Neil Horman

On Intel hardware, cpus are limited in the number of irqs they can have affined to them (currently 240), based on section 10.5.2 of:
https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf

That reference is really not useful to explain the problem and the number of vectors is neither. Please explain the conceptual issue.

You seem to have already done that below. Not really sure what more you are asking for here.

Quoted 9 lines
Quoted 3 lines

If a cpu has more than this number of interrupts affined to it, they will spill over to other cpus, which potentially may be outside of their affinity mask.

Spill over?

The kernel decides to pick a vector on a CPU outside of the affinity when it runs out of vectors on the CPUs in the affinity mask.

Yes.

Quoted 2 lines

Please explain issues technically correct.

I don't know what you mean by this. I explained it above, and you clearly understood it.

Quoted 10 lines

Review __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Quoted 5 lines

Given that this might cause unexpected behavior on performance sensitive systems, warn the user should this condition occur so that corrective action can be taken

(quoted diff) +0 -0
@@ -244,6 +244,14 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Why on earth warn in the interrupt delivery hotpath? Just because it's the place which really needs extra instructions and extra cache lines on performance sensitive systems, right?

Because theres already a check of the same variety in do_IRQ, but if the information is available outside the hotpath, I was unaware, and am happy to update this patch to refelct that.

Quoted 6 lines

The fact that the kernel ran out of vectors for the CPUs in the affinity mask is already known when the vector is allocated in activate_reserved().

So there is an obvious place to put such a warning and it's certainly not do_IRQ().

Sure

Thanks
Neil

Quoted 4 lines

Thanks,

tglx

#3 lore raw

Neil,

Quoted 11 lines · Neil Horman
Quoted 10 lines · Thomas Gleixner
Quoted 3 lines · Neil Horman

If a cpu has more than this number of interrupts affined to it, they will spill over to other cpus, which potentially may be outside of their affinity mask.

Spill over?

The kernel decides to pick a vector on a CPU outside of the affinity when it runs out of vectors on the CPUs in the affinity mask.

Quoted 6 lines

Yes.

Quoted 2 lines

Please explain issues technically correct.

I don't know what you mean by this. I explained it above, and you clearly understood it.

It took me a while to grok it. Simply because I first thought it's some hardware issue. And of course after confusion settled I knew what it is, but just because I know that code like the back of my hand.

Quoted 10 lines
Quoted 10 lines

Review __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Quoted 5 lines

Given that this might cause unexpected behavior on performance sensitive systems, warn the user should this condition occur so that corrective action can be taken

(quoted diff) +0 -0
@@ -244,6 +244,14 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Why on earth warn in the interrupt delivery hotpath? Just because it's the place which really needs extra instructions and extra cache lines on performance sensitive systems, right?

Quoted 3 lines

Because theres already a check of the same variety in do_IRQ, but if the information is available outside the hotpath, I was unaware, and am happy to update this patch to refelct that.

Which check are you referring to?

Thanks,

tglx

#4 lore raw
Quoted 35 lines · Thomas Gleixner

Neil,

Quoted 11 lines · Neil Horman
Quoted 10 lines · Thomas Gleixner
Quoted 3 lines · Neil Horman

If a cpu has more than this number of interrupts affined to it, they will spill over to other cpus, which potentially may be outside of their affinity mask.

Spill over?

The kernel decides to pick a vector on a CPU outside of the affinity when it runs out of vectors on the CPUs in the affinity mask.

Quoted 6 lines

Yes.

Quoted 2 lines

Please explain issues technically correct.

I don't know what you mean by this. I explained it above, and you clearly understood it.

It took me a while to grok it. Simply because I first thought it's some hardware issue. And of course after confusion settled I knew what it is, but just because I know that code like the back of my hand.

Quoted 10 lines
Quoted 10 lines

Review __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Quoted 5 lines

Given that this might cause unexpected behavior on performance sensitive systems, warn the user should this condition occur so that corrective action can be taken

(quoted diff) +0 -0
@@ -244,6 +244,14 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)

Why on earth warn in the interrupt delivery hotpath? Just because it's the place which really needs extra instructions and extra cache lines on performance sensitive systems, right?

Quoted 6 lines
Quoted 3 lines

Because theres already a check of the same variety in do_IRQ, but if the information is available outside the hotpath, I was unaware, and am happy to update this patch to refelct that.

Which check are you referring to?

This one:

if (desc != VECTOR_RETRIGGERED) {
                        pr_emerg_ratelimited("%s: %d.%d No irq handler for vector\n",
                                             __func__, smp_processor_id(),
                                             vector);

I figured it was already checking one condition, another wouldn't hurt too much, but no worries, I'm redoing this in activate_reserved now.

Best
Neil

Quoted 4 lines

Thanks,

tglx

#5 lore raw

Neil,

Quoted 7 lines · Neil Horman
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

Because theres already a check of the same variety in do_IRQ, but if the information is available outside the hotpath, I was unaware, and am happy to update this patch to refelct that.

Which check are you referring to?

Quoted 7 lines

This one:

if (desc != VECTOR_RETRIGGERED) {
                        pr_emerg_ratelimited("%s: %d.%d No irq handler for vector\n",
                                             __func__, smp_processor_id(),
                                             vector);

I figured it was already checking one condition, another wouldn't hurt too much, but no worries, I'm redoing this in activate_reserved now.

That's in the slow path, when handle_irq() failed which is the unlikely case.

Thanks,

tglx