Skip to contentSkip to conversation

x86: interrupt routing question

DISCUSSION Mike Galbraith 3 participants 4 messages Jul 12, 2011 – Sep 28, 2011

lore
Share & tools
lore
#0 lore raw

Greetings,

I have an x3550 M3 box which shows different behavior than my Q6600 desktop box. The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600 box, IRQs magically appear on every cpu in the affinity mask. Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.

On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange. Is this some kind of BIOS thingie, or is every kernel busted on this hardware? Busted kernel seems highly doubtful.

OT question. I have HT turned off on the x3550 because I do rt testing on it, and occasionally boot idle=poll. Is it still the case that you should not run idle=poll with HT enabled, or can you do that with HT these days (E5620 CPU) without munching too many cycles?

CPU0 CPU1 CPU2 CPU3

   0:        132          0          0          0   IO-APIC-edge      timer
   1:          0          0          0          0   IO-APIC-edge      i8042
   3:          1          0          0          0   IO-APIC-edge    
   4:          1          0          0          0   IO-APIC-edge    
   8:         26          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  14:          0          0          0          0   IO-APIC-edge      ata_piix
  15:          0          0          0          0   IO-APIC-edge      ata_piix
  16:       5450          0          0          0   IO-APIC-fasteoi   megasas
  17:         52          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb5
  18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, uhci_hcd:usb6
  19:      61939          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
  21:          0          0          0          0   IO-APIC-fasteoi   ata_piix
  54:          3          0          0          0   PCI-MSI-edge      ioat-msix
  55:          3          0          0          0   PCI-MSI-edge      ioat-msix
  56:          3          0          0          0   PCI-MSI-edge      ioat-msix
  57:          3          0          0          0   PCI-MSI-edge      ioat-msix
  58:          3          0          0          0   PCI-MSI-edge      ioat-msix
  59:          3          0          0          0   PCI-MSI-edge      ioat-msix
  60:          3          0          0          0   PCI-MSI-edge      ioat-msix
  61:          3          0          0          0   PCI-MSI-edge      ioat-msix
  62:        178          0          0          0   PCI-MSI-edge      eth1-0
  63:    6327232          0          0          0   PCI-MSI-edge      eth1-1
  64:         73          0          0          0   PCI-MSI-edge      eth1-2
  65:    6327230          0          0          0   PCI-MSI-edge      eth1-3
  66:          2          0          0          0   PCI-MSI-edge      eth1-4
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:    2096607    2366453     260343    2225391   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0   Performance pending work
 RES:       1981       2973       2405       2001   Rescheduling interrupts
 CAL:         49         43         74         72   Function call interrupts
 TLB:        638        784       1175        674   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:        603        603        603        606   Machine check polls

ERR: 0
MIS: 0

#1 lore raw

For the next sod who gets curious about irq routing behavior difference, and tries to ask google.

Quoted 9 lines · Mike Galbraith

Greetings,

I have an x3550 M3 box which shows different behavior than my Q6600 desktop box. The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600 box, IRQs magically appear on every cpu in the affinity mask. Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.

Quoted 5 lines

On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange. Is this some kind of BIOS thingie, or is every kernel busted on this hardware? Busted kernel seems highly doubtful.

The difference is that Q6600 desktop box uses flat routing, and x3550 M3 uses physical flat. What the heck does that mean you (didn't) ask? Dunno, that's a "RTFM for all the gory technical details" thing, but if you rummage around in x86 source, you'll end up here..

arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()

        if (assign_irq_vector(irq, cfg, apic->target_cpus()))
                return;

        dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());

        apic_printk(APIC_VERBOSE,KERN_DEBUG
                    "IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
                    "IRQ %d Mode:%i Active:%i Dest:%d)\n",
                    apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector,
                    irq, trigger, polarity, dest);

..and see that in physical flat mode, assign_irq_vector() scribbles only one bit to cfg->domain.

arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()

            for_each_cpu_and(cpu, mask, cpu_online_mask) {
                int new_cpu;
                int vector, offset;

                apic->vector_allocation_domain(cpu, tmp_mask);
		...
                cpumask_copy(cfg->domain, tmp_mask);
                err = 0;
                break;
        }

apic.vector_allocation_domain for physical flat is..

arch/x86/kernel/apic/apic_flat_64.c::

static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask)

{
        cpumask_clear(retmask);
        cpumask_set_cpu(cpu, retmask);
}

..whereas for flat it's..

static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask)

{
	...
        cpumask_clear(retmask);
        cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}

So, while __assign_irq_vector() is called with a mask of all CPUs for both boxen, apic->vector_allocation_domain() ensures the mask finally written to cfg->domain has only one CPU bit set in physical flat mode, and all CPU bits set in flat mode.

-Mike

P.S. if you try that RTFM thing, strap a pillow to your forehead first

#2 lore raw
Quoted 3 lines · Mike Galbraith

For the next sod who gets curious about irq routing behavior difference, and tries to ask google.

Thanks for following up on this!

So, having read the fine manual, do you think this is a Linux bug, or something that could be improved in Linux, either by changing the behavior or making the dmesg more informative? It certainly cost you a lot of time, and it'd be nice if we could save the next person :)

Quoted 83 lines · Mike Galbraith
Quoted 9 lines

Greetings,

I have an x3550 M3 box which shows different behavior than my Q6600 desktop box.  The below is the rt kernel, but it doesn't matter which kernel I boot, all interrupts are on CPU0 unless I move them.  On Q6600 box, IRQs magically appear on every cpu in the affinity mask.  Change the eth0 mask while flood pinging, numbers start/stop changing on the fly.

Quoted 5 lines

On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2, but nothing that I have found will make the thing behave the same as trusty old Q6600 box, which seems strange.  Is this some kind of BIOS thingie, or is every kernel busted on this hardware?  Busted kernel seems highly doubtful.

The difference is that Q6600 desktop box uses flat routing, and x3550 M3 uses physical flat.  What the heck does that mean you (didn't) ask? Dunno, that's a "RTFM for all the gory technical details" thing, but if you rummage around in x86 source, you'll end up here..

arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()

       if (assign_irq_vector(irq, cfg, apic->target_cpus())) return;

       dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());

       apic_printk(APIC_VERBOSE,KERN_DEBUG
                   "IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> " "IRQ %d Mode:%i Active:%i Dest:%d)\n", apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector, irq, trigger, polarity, dest);

..and see that in physical flat mode, assign_irq_vector() scribbles only one bit to cfg->domain.

arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()

           for_each_cpu_and(cpu, mask, cpu_online_mask) { int new_cpu;
               int vector, offset;

               apic->vector_allocation_domain(cpu, tmp_mask);
               ...
               cpumask_copy(cfg->domain, tmp_mask); err = 0;
               break;
       }

apic.vector_allocation_domain for physical flat is..

arch/x86/kernel/apic/apic_flat_64.c::

static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask) {
       cpumask_clear(retmask);
       cpumask_set_cpu(cpu, retmask);
}

..whereas for flat it's..

static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask) {
       ...
       cpumask_clear(retmask);
       cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}

So, while __assign_irq_vector() is called with a mask of all CPUs for both boxen, apic->vector_allocation_domain() ensures the mask finally written to cfg->domain has only one CPU bit set in physical flat mode, and all CPU bits set in flat mode.

       -Mike

P.S.  if you try that RTFM thing, strap a pillow to your forehead first

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#3 lore raw
Quoted 11 lines · Bjorn Helgaas
Quoted 3 lines · Mike Galbraith

For the next sod who gets curious about irq routing behavior difference, and tries to ask google.

Thanks for following up on this!

So, having read the fine manual, do you think this is a Linux bug, or something that could be improved in Linux, either by changing the behavior or making the dmesg more informative? It certainly cost you a lot of time, and it'd be nice if we could save the next person :)

It didn't take long at all after I stopped trying to ask google and acpidump, and went to the source. I don't think it's a bug, but I only _started_ to RTFM, so whether it could be improved is a question I can't begin to answer.. and kinda hope to never be able to answer :)

-Mike