x86: interrupt routing question
Greetings,
I have an x3550 M3 box which shows different behavior than my Q6600 desktop box. The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600 box, IRQs magically appear on every cpu in the affinity mask. Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.
On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange. Is this some kind of BIOS thingie, or is every kernel busted on this hardware? Busted kernel seems highly doubtful.
OT question. I have HT turned off on the x3550 because I do rt testing on it, and occasionally boot idle=poll. Is it still the case that you should not run idle=poll with HT enabled, or can you do that with HT these days (E5620 CPU) without munching too many cycles?
CPU0 CPU1 CPU2 CPU3
0: 132 0 0 0 IO-APIC-edge timer 1: 0 0 0 0 IO-APIC-edge i8042 3: 1 0 0 0 IO-APIC-edge 4: 1 0 0 0 IO-APIC-edge 8: 26 0 0 0 IO-APIC-edge rtc0 9: 0 0 0 0 IO-APIC-fasteoi acpi 14: 0 0 0 0 IO-APIC-edge ata_piix 15: 0 0 0 0 IO-APIC-edge ata_piix 16: 5450 0 0 0 IO-APIC-fasteoi megasas 17: 52 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb5 18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, uhci_hcd:usb6 19: 61939 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7 21: 0 0 0 0 IO-APIC-fasteoi ata_piix 54: 3 0 0 0 PCI-MSI-edge ioat-msix 55: 3 0 0 0 PCI-MSI-edge ioat-msix 56: 3 0 0 0 PCI-MSI-edge ioat-msix 57: 3 0 0 0 PCI-MSI-edge ioat-msix 58: 3 0 0 0 PCI-MSI-edge ioat-msix 59: 3 0 0 0 PCI-MSI-edge ioat-msix 60: 3 0 0 0 PCI-MSI-edge ioat-msix 61: 3 0 0 0 PCI-MSI-edge ioat-msix 62: 178 0 0 0 PCI-MSI-edge eth1-0 63: 6327232 0 0 0 PCI-MSI-edge eth1-1 64: 73 0 0 0 PCI-MSI-edge eth1-2 65: 6327230 0 0 0 PCI-MSI-edge eth1-3 66: 2 0 0 0 PCI-MSI-edge eth1-4 NMI: 0 0 0 0 Non-maskable interrupts LOC: 2096607 2366453 260343 2225391 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts PND: 0 0 0 0 Performance pending work RES: 1981 2973 2405 2001 Rescheduling interrupts CAL: 49 43 74 72 Function call interrupts TLB: 638 784 1175 674 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 603 603 603 606 Machine check polls
ERR: 0
MIS: 0
For the next sod who gets curious about irq routing behavior difference, and tries to ask google.
Quoted 9 lines · Mike Galbraith
Greetings,
I have an x3550 M3 box which shows different behavior than my Q6600 desktop box. The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600 box, IRQs magically appear on every cpu in the affinity mask. Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.
Quoted 5 lines
On x3550 M3, setting eg IRQ 63's mask to '
0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange. Is this some kind of BIOS thingie, or is every kernel busted on this hardware? Busted kernel seems highly doubtful.
The difference is that Q6600 desktop box uses flat routing, and x3550 M3 uses physical flat. What the heck does that mean you (didn't) ask? Dunno, that's a "RTFM for all the gory technical details" thing, but if you rummage around in x86 source, you'll end up here..
arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()
if (assign_irq_vector(irq, cfg, apic->target_cpus()))
return;
dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());
apic_printk(APIC_VERBOSE,KERN_DEBUG
"IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
"IRQ %d Mode:%i Active:%i Dest:%d)\n",
apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector,
irq, trigger, polarity, dest);..and see that in physical flat mode, assign_irq_vector() scribbles only one bit to cfg->domain.
arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()
for_each_cpu_and(cpu, mask, cpu_online_mask) {
int new_cpu;
int vector, offset;
apic->vector_allocation_domain(cpu, tmp_mask);
...
cpumask_copy(cfg->domain, tmp_mask);
err = 0;
break;
}apic.vector_allocation_domain for physical flat is..
arch/x86/kernel/apic/apic_flat_64.c::
static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
cpumask_clear(retmask);
cpumask_set_cpu(cpu, retmask);
}..whereas for flat it's..
static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
...
cpumask_clear(retmask);
cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}So, while __assign_irq_vector() is called with a mask of all CPUs for both boxen, apic->vector_allocation_domain() ensures the mask finally written to cfg->domain has only one CPU bit set in physical flat mode, and all CPU bits set in flat mode.
-Mike
P.S. if you try that RTFM thing, strap a pillow to your forehead first
Quoted 3 lines · Mike Galbraith
For the next sod who gets curious about irq routing behavior difference, and tries to ask google.
Thanks for following up on this!
So, having read the fine manual, do you think this is a Linux bug, or something that could be improved in Linux, either by changing the behavior or making the dmesg more informative? It certainly cost you a lot of time, and it'd be nice if we could save the next person :)
Quoted 83 lines · Mike Galbraith
Quoted 9 lines
Greetings,
I have an x3550 M3 box which shows different behavior than my Q6600 desktop box. The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600 box, IRQs magically appear on every cpu in the affinity mask. Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.
Quoted 5 lines
On x3550 M3, setting eg IRQ 63's mask to '
0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange. Is this some kind of BIOS thingie, or is every kernel busted on this hardware? Busted kernel seems highly doubtful.The difference is that Q6600 desktop box uses flat routing, and x3550 M3 uses physical flat. What the heck does that mean you (didn't) ask? Dunno, that's a "RTFM for all the gory technical details" thing, but if you rummage around in x86 source, you'll end up here..
arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()
if (
assign_irq_vector(irq, cfg, apic->target_cpus())) return;dest =
apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());apic_printk(APIC_VERBOSE,KERN_DEBUG
"IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> " "IRQ %d Mode:%i Active:%i Dest:%d)\n",apic_id,mpc_ioapic_id(apic_id), pin,cfg->vector, irq, trigger, polarity, dest);..and see that in physical flat mode,
assign_irq_vector()scribbles only one bit tocfg->domain.arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()
for_each_cpu_and(cpu, mask,
cpu_online_mask) { intnew_cpu;
int vector, offset;apic->vector_allocation_domain(cpu,
tmp_mask);
...
cpumask_copy(cfg->domain,tmp_mask); err = 0;
break;
}apic.vector_allocation_domain for physical flat is..
arch/x86/kernel/apic/apic_flat_64.c::static void
physflat_vector_allocation_domain(int cpu, struct cpumask *retmask){
cpumask_clear(retmask);
cpumask_set_cpu(cpu, retmask);
}..whereas for flat it's..
static void
flat_vector_allocation_domain(int cpu, struct cpumask *retmask){
...
cpumask_clear(retmask);
cpumask_bits(retmask)[0] =APIC_ALL_CPUS;
}So, while
__assign_irq_vector()is called with a mask of all CPUs for both boxen,apic->vector_allocation_domain()ensures the mask finally written tocfg->domainhas only one CPU bit set in physical flat mode, and all CPU bits set in flat mode.-Mike
P.S. if you try that RTFM thing, strap a pillow to your forehead first--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines · Bjorn Helgaas
Quoted 3 lines · Mike Galbraith
For the next sod who gets curious about irq routing behavior difference, and tries to ask google.
Thanks for following up on this!
So, having read the fine manual, do you think this is a Linux bug, or something that could be improved in Linux, either by changing the behavior or making the dmesg more informative? It certainly cost you a lot of time, and it'd be nice if we could save the next person :)
It didn't take long at all after I stopped trying to ask google and acpidump, and went to the source. I don't think it's a bug, but I only _started_ to RTFM, so whether it could be improved is a question I can't begin to answer.. and kinda hope to never be able to answer :)
-Mike