PCI passthrough (shared IRQ) not working / contradicting information
Hi all,
a while ago, I have set up a KVM virtual machine with PCI passthrough. The device which is passed through is an AVM Fritz!card PCI v2.1.
This works, but there is a drawback: At that time, KVM was not able to do PCI passthrough for PCI devices with shared interrupt. So I had to find out which other devices were using the same interrupt, and had to inactivate them by handling them over to the PCI stub driver.
Then, one day, I have been very excited when I was reading that with kernel 3.4, KVM should be able to do PCI passthrough even if the respective device shares its IRQ with other devices. Thus, two day ago, I took the time, got vanilla kernel 3.9, configured, compiled and installed, and got qemu 1.4.1, configured, compiled and installed.
Now, I am disappointed that the passthrough still doesn't work if I leave the other devices active, probably due to a silly error on my side. It still works if I inactivate the other devices which share their IRQ with the AVM card, but that actually was not the goal.
Furthermore, the last two sentences on the following page suggest that PCI passthrough of devices with shared IRQs is still not possible: http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So there is a contradiction between what has been stated when kernel 3.4 came out and what linux-kvm.org says.
Could anybody please tell me if PCI passthrough with shared IRQs is possible or not? Does anybody know why the AVM Fritz!card won't work without inactivating other devices? What mistake could lead to the current situation (passthrough works if other devices are inactivated, and does not work if not)?
This is my command line for starting the VM:
/usr/local/bin/qemu-system-x86_64 -machine type=pc,accel=kvm -cpu host -smp cores=1,threads=2,sockets=1 -rtc base=localtime,clock=host,driftfix=none -drive file=/dev/sda6,if=virtio,cache=none,index=0 -drive file=/dev/sdb,if=virtio,cache=none,index=1 -cdrom /dev/sr0 -pidfile ./qemu-garak.pid -boot c -k de -m 4096 -device pci-assign,host=01:05.0 -daemonize -usb -usbdevice "tablet" -name garak -net nic,vlan=0,model=virtio,macaddr=02:01:01:01:01:01 -net tap,vlan=0,ifname=virtnet1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifup -vnc :1
This is the error message which I get if I don't inactivate the other devices:
qemu-system-x86_64: -device pci-assign,host=01:05.0: Failed to assign irq for "(null)": Input/output error
qemu-system-x86_64: -device pci-assign,host=01:05.0: Perhaps you are assigning a device that shares an IRQ with another device?
qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'kvm-pci-assign' could not be initialized
I don't understand why the first line talks about the device "(null)" (I would have excepted a device name or a PCI bus device number here), and I don't understand the second line (because that is exactly what kernels from 3.4 on should be able to do, right?).
By the way, my method for making the other devices which share their IRQ with the Fritz!card is the following:
modprobe -r kvm-intel
modprobe -r kvm
echo "1244 0a00" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:01:05.0 > /sys/bus/pci/devices/0000:01:05.0/driver/unbind echo 0000:01:05.0 > /sys/bus/pci/drivers/pci-stub/bind
echo "8086 3a39" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:1a.2 > /sys/bus/pci/devices/0000:00:1a.2/driver/unbind echo 0000:00:1a.2 > /sys/bus/pci/drivers/pci-stub/bind
echo "8086 3a35" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:1d.1 > /sys/bus/pci/devices/0000:00:1d.1/driver/unbind echo 0000:00:1d.1 > /sys/bus/pci/drivers/pci-stub/bind
modprobe kvm-intel
Thank you very much for any help,
Binarus
Quoted 27 lines · Binarus
Hi all,
a while ago, I have set up a KVM virtual machine with PCI passthrough. The device which is passed through is an AVM Fritz!card PCI v2.1.
This works, but there is a drawback: At that time, KVM was not able to do PCI passthrough for PCI devices with shared interrupt. So I had to find out which other devices were using the same interrupt, and had to inactivate them by handling them over to the PCI stub driver.
Then, one day, I have been very excited when I was reading that with kernel 3.4, KVM should be able to do PCI passthrough even if the respective device shares its IRQ with other devices. Thus, two day ago, I took the time, got vanilla kernel 3.9, configured, compiled and installed, and got qemu 1.4.1, configured, compiled and installed.
Now, I am disappointed that the passthrough still doesn't work if I leave the other devices active, probably due to a silly error on my side. It still works if I inactivate the other devices which share their IRQ with the AVM card, but that actually was not the goal.
Furthermore, the last two sentences on the following page suggest that PCI passthrough of devices with shared IRQs is still not possible:
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So there is a contradiction between what has been stated when kernel 3.4 came out and what linux-kvm.org says.
Quoted 5 lines
Could anybody please tell me if PCI passthrough with shared IRQs is possible or not? Does anybody know why the AVM Fritz!card won't work without inactivating other devices? What mistake could lead to the current situation (passthrough works if other devices are inactivated, and does not work if not)?
The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is required to share legacy interrupts.
Alex, does VFIO provide any convenient diagnostics if INTx sharing is possible or not? If not, maybe we should add it at least over there.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 28 lines · Jan Kiszka
Quoted 27 lines · Binarus
Hi all,
a while ago, I have set up a KVM virtual machine with PCI passthrough. The device which is passed through is an AVM Fritz!card PCI v2.1.
This works, but there is a drawback: At that time, KVM was not able to do PCI passthrough for PCI devices with shared interrupt. So I had to find out which other devices were using the same interrupt, and had to inactivate them by handling them over to the PCI stub driver.
Then, one day, I have been very excited when I was reading that with kernel 3.4, KVM should be able to do PCI passthrough even if the respective device shares its IRQ with other devices. Thus, two day ago, I took the time, got vanilla kernel 3.9, configured, compiled and installed, and got qemu 1.4.1, configured, compiled and installed.
Now, I am disappointed that the passthrough still doesn't work if I leave the other devices active, probably due to a silly error on my side. It still works if I inactivate the other devices which share their IRQ with the AVM card, but that actually was not the goal.
Furthermore, the last two sentences on the following page suggest that PCI passthrough of devices with shared IRQs is still not possible:
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So there is a contradiction between what has been stated when kernel 3.4 came out and what linux-kvm.org says.
Quoted 8 lines
Quoted 5 lines
Could anybody please tell me if PCI passthrough with shared IRQs is possible or not? Does anybody know why the AVM Fritz!card won't work without inactivating other devices? What mistake could lead to the current situation (passthrough works if other devices are inactivated, and does not work if not)?
The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is required to share legacy interrupts.
Yep. Binarus, the issue is that shared interrupts require hardware support to mask the interrupt at the device. This was introduced as part of the PCI 2.3 standard. If the device does not support this then exclusive interrupts are still required.
Quoted 2 lines
Alex, does VFIO provide any convenient diagnostics if INTx sharing is possible or not? If not, maybe we should add it at least over there.
It does not. It's actually easier to test outside of qemu as vfio will emulate PCI2.3 support. Here's a simple shell script requiring only sudo and setpci:
#!/bin/sh
DESC=$(lspci -s $1)
if [ "x$DESC" == "x" ]; then
echo "Device $1 not found"
exit 1
fi
ORIGCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
NEWCMD=$(printf 0x%04x $(( $ORIGCMD ^ 0x400 )))
setpci -s $1 COMMAND=$NEWCMD
NEWCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
if [ $ORIGCMD == $NEWCMD ]; then
echo "Device $1 does NOT support PCI 2.3 INTx disable" exit 1
fi
setpci -s $1 COMMAND=$ORIGCMD
echo "Device $1 supports PCI 2.3 INTx disable"
Quoted 64 lines · Alex Williamson
Quoted 28 lines · Jan Kiszka
Quoted 27 lines · Binarus
Hi all,
a while ago, I have set up a KVM virtual machine with PCI passthrough. The device which is passed through is an AVM Fritz!card PCI v2.1.
This works, but there is a drawback: At that time, KVM was not able to do PCI passthrough for PCI devices with shared interrupt. So I had to find out which other devices were using the same interrupt, and had to inactivate them by handling them over to the PCI stub driver.
Then, one day, I have been very excited when I was reading that with kernel 3.4, KVM should be able to do PCI passthrough even if the respective device shares its IRQ with other devices. Thus, two day ago, I took the time, got vanilla kernel 3.9, configured, compiled and installed, and got qemu 1.4.1, configured, compiled and installed.
Now, I am disappointed that the passthrough still doesn't work if I leave the other devices active, probably due to a silly error on my side. It still works if I inactivate the other devices which share their IRQ with the AVM card, but that actually was not the goal.
Furthermore, the last two sentences on the following page suggest that PCI passthrough of devices with shared IRQs is still not possible:
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So there is a contradiction between what has been stated when kernel 3.4 came out and what linux-kvm.org says.Quoted 8 lines
Quoted 5 lines
Could anybody please tell me if PCI passthrough with shared IRQs is possible or not? Does anybody know why the AVM Fritz!card won't work without inactivating other devices? What mistake could lead to the current situation (passthrough works if other devices are inactivated, and does not work if not)?
The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is required to share legacy interrupts.
Yep. Binarus, the issue is that shared interrupts require hardware support to mask the interrupt at the device. This was introduced as part of the PCI 2.3 standard. If the device does not support this then exclusive interrupts are still required.
Quoted 2 lines
Alex, does VFIO provide any convenient diagnostics if INTx sharing is possible or not? If not, maybe we should add it at least over there.
It does not. It's actually easier to test outside of qemu as vfio will emulate PCI2.3 support. Here's a simple shell script requiring only sudo and setpci:
#!/bin/sh
DESC=$(lspci -s $1)
if [ "x$DESC" == "x" ]; then
echo "Device $1 not found"
exit 1
fiORIGCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
NEWCMD=$(printf 0x%04x $(( $ORIGCMD ^0x400)))
setpci -s $1 COMMAND=$NEWCMD
NEWCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
Quoted 8 lines
if [ $ORIGCMD == $NEWCMD ]; then
echo "Device $1 does NOT support PCI 2.3 INTx disable" exit 1
fisetpci -s $1 COMMAND=$ORIGCMD
echo "Device $1 supports PCI 2.3 INTx disable"
Easier for the implementers POV... ;)
Well, please push it at least to our wiki page.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux