Questions on interrupt routing / balancing (on x86)

Hi all,

Lately, I have been looking into how interrupts are routed / balanced on x86 systems. I've learned a lot of things along the road and feel much more familiar with the whole thing now, however there are still a few things I don't understand and for which I would appreciate explanations or pointers.

I do understand that the IO-APIC has two routing modes, namely flat and physical flat, the former being limited to 8 logical CPUs for technical reasons. I also found that physical flat routing would be selected even with only 8 CPUs installed if the host chipset is known to support more. I have also read about how MSI-X allows smarter balancing of interrupts across CPUs by allowing multiple interrupt queues per device. Now to the things I do not fully understand:

* In physical flat mode, all interrupts are bound to CPU0 by default. As I understand it, it stays that way until user-space (for example irqbalance) adjusts the smp_affinity masks in /proc/irq. Why don't we pick a different CPU for every interrupt by default? For example irq_nr%cpu_max? This would seem a better default for performance, but maybe not for power savings. Is it the reason why it isn't done?

* In (non-physical) flat mode, my understanding is that a given interrupt can be mapped to several CPUs (and this happens by default) and live round-robin balancing can happen. I have seen systems where it actually happens, with interrupt counters perfectly balanced on all CPUs, but I have also seen systems where CPU0 gets all the interrupts all the time. Why is it so? Where is the kernel code which decides if round-robin balancing should happen? Or is this a hardware decision?

* Would it be possible to have a kernel boot parameter to force (non- physical) flat mode even with more than 8 CPUs, in order to restore round-robin balancing? I understand that this would limit interrupt routing to CPUs 0-7, but other than this, would it work?

* Do I properly understand that MSI and MSI-X interrupts do NOT go through the IO-APIC and are thus not affected by flat vs. physical flat APIC routing mode? If so, what determines whether these interrupts get round-robin balanced or not? Here too, I've seen systems where it happens and others where it doesn't (looking at /proc/interrupts.)

Thanks for any answer or pointer you can provide.

Jean Delvare
Suse L3

It really depends on what apic mode is selected and how HW implements etc.

Also that may not always be better default. For example, two heavily loaded irq's like network/disk interrupt can end up on two hyperthreaded siblings leading to poor performance. So irqbalance is the right place to have all the smartness.

Its chipset decision. Some chipsets implement round-robin and some don't.

you can use logical cluster mode which can support upto 60 logical cpu's. Some old linux kernels used to support this mode.

Even though MSI and MSI-X don't go through io-apic, they still use the apic model that is selected by the kernel.

Anyways, newer platforms are supporting x2apic extensions that will support more cpu's in logical/physical mode etc.

thanks,
suresh

Thanks Suresh for all the answers, very useful! :)

Jean Delvare
Suse L3