IRQ affinity on Linux guest

Hello,

I'm trying to assign some IRQ affinities to core 0 by setting smp_affinity to 1. This is on a dual-core embedded Linux virtual machine ran with KVM. However, ISRs continue to run on both cores.
The same technique works well with QEMU with full software emulation.

Here is the output of the following:

host> uname -a
Linux rtcon2 3.19.0-22-generic #22~14.04.1-Ubuntu SMP Wed Jun 17 10:03:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

guest> uname -a
Linux NI-cRIO-Linux-525400123456 3.14.40-rt37-nilrt #1 SMP PREEMPT RT Thu Aug 6 20:35:41 EEST 2015 x86_64 GNU/Linux

host> kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.15), Copyright
(c) 2003-2008 Fabrice Bellard

Here is how IRQ affinity is configured on guest at startup, in an init.d script:

echo 1 > /proc/irq/default_smp_affinity
for x in /proc/irq/*/smp_affinity;
do
echo 1 > $x
done 2> /dev/null

The command line for starting the hardware accelerated VM:
kvm -kernel bzImage -hda rootfs.ext2 -append "root=/dev/sda console=ttyS0 \
rw" -nographic -cpu qemu64 -snapshot -smp 2 -m 2048

... which actually runs this:
qemu-system-x86-64 -enable-kvm -kernel bzImage -hda rootfs.ext2 -append \
"root=/dev/sda console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2 \
-m 2048

On the hardware accelerated guest, 'cat /proc/interrupts' shows:
CPU0 CPU1

  0:         26          0   IO-APIC-edge      timer
  1:          7          4   IO-APIC-edge      i8042
  4:       1137        523   IO-APIC-edge      serial
  8:          0          1   IO-APIC-edge      rtc0
  9:          0          0   IO-APIC-fasteoi   acpi
 11:       4971          4   IO-APIC-fasteoi   eth0
 12:         66         64   IO-APIC-edge      i8042
 14:       1958        714   IO-APIC-edge      ata_piix
 15:       4512         63   IO-APIC-edge      ata_piix

...
Interrupts are serviced on both cores, even though affinity is set to 1.

The command line for starting full software emulation VM is:
qemu-system-x86_64 -kernel bzImage -hda rootfs.ext2 -append "root=/dev/sda \
console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2 -m 2048

On the full software emulation guest, 'cat /proc/interrupts' shows:
CPU0 CPU1

  0:         36          0   IO-APIC-edge      timer
  1:         10          0   IO-APIC-edge      i8042
  4:       1775          0   IO-APIC-edge      serial
  8:          1          0   IO-APIC-edge      rtc0
  9:          0          0   IO-APIC-fasteoi   acpi
 11:        345          0   IO-APIC-fasteoi   eth0
 12:        125          0   IO-APIC-edge      i8042
 14:       1720          0   IO-APIC-edge      ata_piix
 15:        481          0   IO-APIC-edge      ata_piix

...
Interrupts are serviced only on CPU0, therefore setting IRQ affinity worked.

Do you have any idea why KVM doesn't respect IRQ affinity on the guest? What can I do to get it working with IRQ affinity set to 1?

Thanks,
Mihai

2015-08-20 17:16+0300, Mihai Neagu:

KVM's APIC balances interrupts -- until you set the affinity (probably near the end of boot process), both CPUs are going to receive roughly the same amount but after directing subsequent interrupts to CPU0, CPU1 shouldn't receive more.

Please verify that CPU0 is not receiving all interrupts by doing a difference between two cat /proc/interrupts after the affinity was set. (CPU0 has higher numbers in your excerpt, which makes me suspect that it works as expected.)

Thanks.

Radim,

Thanks for your answer. Indeed setting IRQ affinity to a specific core seems to be respected.

However, on software emulation and on the real machine, while IRQ affinity defaults to 3, all interrupts go on CPU0, while on KVM they go on CPU1.

I wonder why KVM would act differently than both the real machine and the software emulation in this particular aspect.

Is there a machine or processor that I can specify at KVM command line to make it behave like the real x86_64 processor which defaults interrupts to CPU0?

Thanks,
Mihai

Are you sure that KVM doesn't do some kind of round robin?

Also, it's probably not _all_ interrupts but most of them. For example my laptop has:

                                                            smp_affinity
        cpu0      cpu1     cpu2      cpu3                           ||||
1:     32824     1764       151       130   IO-APIC-edge  i8042     0123
12:  1587436   203990     16666     17064   IO-APIC-edge  i8042     0123
24:    63361     5160    212545      1276   PCI-MSI-edge  ahci      ..2.
25:        2        0      1082     10710   PCI-MSI-edge  xhci_hcd  ..23
29:  1867844    72169        61     57514   PCI-MSI-edge  iwlwifi   0...

I added a column at the end with the SMP affinity of the interrupts. You can see that:

* real hardware also shows traces of interrupts delivered before the affinity was set (especially vectors 24 and 29)

* real hardware also distributes interrupts across multiple CPUs when the affinity covers multiple processors (see vectors 1, 12 and 25)

The behavior of interrupt arbitration is not specified in the processor documentation and can change across processors and implementations. Presumably it is either patented or an Intel trade secret.

Older processors were documented to pick a processor that was not running another interrupt service routine. This can explain why software emulation shows that all interrupts go on CPU0.

At least some newer processors are not doing it anymore, and the SDM has some high-level explanation of their behavior and its effects:

[...] the chipset bus controller accepts messages from the I/O APIC agents in the system and directs interrupts to the processors on the system bus. When using the lowest priority delivery mode, the chipset chooses a target processor to receive the interrupt out of the set of possible targets.

In operating systems that use the lowest priority delivery mode but do not update the TPR, the TPR information saved in the chipset will potentially cause the interrupt to be always delivered to the same processor from the logical set. This behavior is functionally backward compatible with the P6 family processor but may result in unexpected performance implications.

Other processors are doing so called "vector hashing" (i.e. pick a CPU number based on the affinity mask and vector number, and use it most of the time), which would also explain why the interrupts all go on CPU0.

Paolo