New: PCIe passthrough broken with AMD iommu after s2disk / resume
https://bugzilla.kernel.org/show_bug.cgi?id=42635
Summary: PCIe passthrough broken with AMD iommu after s2disk /
resume
Product: Virtualization
Version: unspecified
Kernel Version: 3.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
A PCIe ethernet device is passed through to the guest and runs fine. Some time later, the guest is shutdown (e.g. virsh shutdown guest) and the host is suspended (s2disk) and resumed again. The guest is started again (and it starts fine), but the device in the guest isn't working any more (it can be seen, it can be pinged itself but nothing is reachable outside the device. The rx or tx counters of the ifconfig output are always 0.
If the guest is shutdown and the device is bound to the host, it's working as expected. If the device is afterwards bound to the guest again, it doesn't work as before in the guest after resume.
kernel: 3.1 / 64bit / smp
kvm: 1.0
board: GA-990XA-UD3
If you need more information, I can provide them - feel free to ask!
Regards,
Klaus
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume
https://bugzilla.kernel.org/show_bug.cgi?id=42635
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from Alex Williamson <[email protected]> 2012-01-23 15:01:29 --- Between the first and second paragraph, you seem to be saying that the device never works in the guest on the second invocation of the guest. Is that true? What is the device?
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=42635
--- Comment #2 from Klaus Mueller <[email protected]> 2012-01-23 16:18:33 --- The device never works in the guest after s2disk/resume cycle of the host. But it always works if it's used by the host itself - even after a s2disk/resume cycle.
The device is (as seen by the host):
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 45
Region 0: I/O ports at ee00 [size=256]
Region 2: Memory at fdcff000 (64-bit, prefetchable) [size=4K] Region 4: Memory at fdcf8000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mAPME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0f00c Data: 4122
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns,
L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
L0 unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000800
Capabilities: [d0] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [160 v1] Device Serial Number 12-34-56-78-12-34-56-78 Kernel driver in use: pci-stub
One more thing: the device can't be used in the guest anymore, too, after a reboot of the guest by rebooting the guest itself. I have to shutdown the guest with virsh shutdown and virsh start to get it working again. But this is useless, if there was a suspend / resume cycle of the host between.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=42635
--- Comment #3 from Klaus Mueller <[email protected]> 2012-04-03 07:27:43 --- Meanwhile I detected, that using the boot option iommu=pt (or additional iommu=1) shows the same problem (regardless of suspend/resume). If the VM is started after the first virsh start / virsh shutdown cycle with virsh start again, the device is dead in the VM. Host must be rebooted to get it working again.
If the options iommu=pt (or additional iommu=1) are not used at all, the VM can be restarted without any problem coming up.
1. Host booted w/ iommu=pt iommu=1
2. virsh start
3. virsh shutdown
4. virsh start
5. -> Passed through device in VM is dead.
1. Host booted w/o iommu=pt iommu=1
2. virsh start
3. virsh shutdown
4. virsh start
5. -> Passed through device in VM is fine.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.