Skip to contentSkip to conversation

Mellanox IB PCI passthrough fails with qemu-kvm>=1.1.2

DISCUSSION Vlastimil Holer 1 participant 1 message Nov 6, 2012 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

we have been using PCI passthrough with the Mellanox IB interface (MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and qemu-kvm-1.0 (both from backports). It worked fine until latest update in backports to qemu-kvm-1.1.2. With newer versions IB device probe in guest fails leaving firmware to kill whole physical machine.

I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex dx360 M4 Server.

On guest the mlx4_core fails to probe device:

| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16

Which immediately results in reset of the whole physical machine:

| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue

Followed by events in hardware management module:

| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI

Kernel logs for both host/guest machines and different qemu-kvm versions are attached. PCI passthrough for e.g. Intel e1000 works fine with all tested qemu-kvm versions.

Any idea or help is very appreciated!

Many thanks,
Vlastimil Holer

Signature · 3 lines
Vlastimil Holer                             phone:   +420-549 49 5349
CERIT Scientific Cloud                      e-mail:  [email protected]
Institute of Computer Science MU            twitter: @vholer