New: BUG at drivers/pci/ats.c:62! the second time booting a kvm guest with pci passthrough

https://bugzilla.kernel.org/show_bug.cgi?id=88671

Bug ID: 88671
Summary: BUG at drivers/pci/ats.c:62! the second time booting a
kvm guest with pci passthrough
Product: Virtualization
Version: unspecified
Kernel Version: 3.17.3
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Created attachment 158381
--> https://bugzilla.kernel.org/attachment.cgi?id=158381&action=edit
lspci

I'm running into this bug while trying to use pci passthrough of an AMD BONAIRE XT (Radeon HD 7790)

Steps to reproduce:

1. virsh start vm
2. virsh destroy vm
3. virsh start vm

This bug only appears after starting the vm for the second time. The first time the vm boots normally and passthrough works as expected.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #1 from Tom Stellard <[email protected]> --- Created attachment 158391
--> https://bugzilla.kernel.org/attachment.cgi?id=158391&action=edit
Backtrace from BUG_ON

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #2 from Tom Stellard <[email protected]> --- Created attachment 158401
--> https://bugzilla.kernel.org/attachment.cgi?id=158401&action=edit
Virtual machine definition

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #3 from Tom Stellard <[email protected]> ---
I should also mention that I have this hook executing when the machine starts up:

if [ "$2" = "prepare" ]; then

        virsh nodedev-detach pci_0000_01_00_1

fi

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

Alex Williamson <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #4 from Alex Williamson <[email protected]> --- I'm not sure how you're getting to this BUG_ON, but (a) legacy KVM device assignment is deprecated and (b) the card you've chosen has known reset issues.
You might want to try vfio-pci, which I know can make this card work at least
once per host boot, but you're likely to get a BSOD and IOMMU faults on subsequent guest [re]boots. The reset problem with this card has been reported to AMD, but there is no solution at this time.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #5 from Tom Stellard <[email protected]> ---
Thanks for the tip about vfio-pci. I can now sometimes get two working guest boots per host boot.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

Tom Stellard <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

          Component|kvm                         |Video(DRI - non Intel)
            Version|unspecified                 |2.5
            Product|Virtualization              |Drivers

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

Tom Stellard <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|BUG at                      |Radeon driver fails to
                   |drivers/pci/ats.c:62! the   |reset hardware properly
                   |second time booting a kvm   |after kvm guest reboot
                   |guest with pci passthrough  |

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #6 from Tom Stellard <[email protected]> ---
I've been playing with this a little more and it seems to be working correctly, but radeon dynamic power management (dpm) always fails to initialize on the second guest boot. My questions are:

1. What methods are being used by kvm/qemu/libvirt to reset the GPU on guest shutdown?

2. Is the problem only cuased by the fact that GPU reset is not implemented correctly in the radeon driver of are there improvements that are needed in kvm/qemu/libvirt in order to get this working?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=88671

--- Comment #7 from Alex Williamson <[email protected]> --- (In reply to Tom Stellard from comment #6)

Secondary PCI bus reset from the parent bridge.

Without a device specific reset, we're doing to most thorough standard reset available to us. I've also tried to use some of the reset mechanisms implemented in the radeon FOSS driver but they offered no improvement over the bus reset. There seems to be some state retained in the device that is not cleared via bus reset.

You are receiving this mail because:
You are watching the assignee of the bug.