New: DMAR:[fault reason 06] PTE Read access is not set
https://bugzilla.kernel.org/show_bug.cgi?id=82761
Bug ID: 82761
Summary: DMAR:[fault reason 06] PTE Read access is not set
Product: Virtualization
Version: unspecified
Kernel Version: 3.16.1
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoWhen I boot with "intel_iommu=on" parameter, I get these errors repeated over and over again in dmesg:
---
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
---
lspci -vvs 05:00.0
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI
Gigabit Ethernet NIC
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (8000ns min, 16000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: I/O ports at c200 [size=256]
Region 1: Memory at f7862000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at f7840000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mAPME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: r8169
lspci -vt
-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
+-01.0-[01]--+-00.0 NVIDIA Corporation Device 0fc6
| \-00.1 NVIDIA Corporation Device 0e1b
+-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processorGraphics Controller
+-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB
xHCI Host Controller
+-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI
Controller #1
+-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB
Enhanced Host Controller #2
+-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High
Definition Audio Controller
+-1c.0-[02]--
+-1c.1-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B
PCI Express Gigabit Ethernet controller
+-1c.3-[04-05]----00.0-[05]--+-00.0 Realtek Semiconductor Co., Ltd.RTL8169 PCI Gigabit Ethernet Controller
| +-01.0 Realtek Semiconductor Co., Ltd.
RTL8169 PCI Gigabit Ethernet Controller
| \-02.0 Realtek Semiconductor Co., Ltd.
RTL8169 PCI Gigabit Ethernet Controller
+-1d.0 Intel Corporation 7 Series/C210 Series Chipset Family USB
Enhanced Host Controller #1
+-1f.0 Intel Corporation H77 Express Chipset LPC Controller
+-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-portSATA Controller [AHCI mode]
\-1f.3 Intel Corporation 7 Series/C210 Series Chipset Family SMBus
Controller
dmesg | grep IOMMU
Intel-IOMMU: enabled
dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
IOMMU 0 0xfed90000: using Queued invalidation
IOMMU 1 0xfed91000: using Queued invalidation
IOMMU: software identity mapping for device 0000:00:00.0
IOMMU: software identity mapping for device 0000:00:01.0
IOMMU: software identity mapping for device 0000:00:14.0
IOMMU: software identity mapping for device 0000:00:16.0
IOMMU: software identity mapping for device 0000:00:1a.0
IOMMU: software identity mapping for device 0000:00:1b.0
IOMMU: software identity mapping for device 0000:00:1c.0
IOMMU: software identity mapping for device 0000:00:1c.1
IOMMU: software identity mapping for device 0000:00:1c.3
IOMMU: software identity mapping for device 0000:00:1d.0
IOMMU: software identity mapping for device 0000:00:1f.0
IOMMU: software identity mapping for device 0000:00:1f.2
IOMMU: software identity mapping for device 0000:00:1f.3
IOMMU: software identity mapping for device 0000:01:00.0
IOMMU: software identity mapping for device 0000:01:00.1
IOMMU: software identity mapping for device 0000:03:00.0
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
IOMMU: Setting identity map for device 0000:00:14.0 [0xc8d17000 - 0xc8d24fff]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xc8d17000 - 0xc8d24fff]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xc8d17000 - 0xc8d24fff]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
OS: Debian Wheezy
Kernel: 3.16.1 (compiled by hand)
If you need more information, just ask.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
https://bugzilla.kernel.org/show_bug.cgi?id=82761
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from Alex Williamson <[email protected]> --- Does it work on 3.17-rc1? Are all of the 8169 NICs on bus 05 up and running? Please provide lspci -vv info for 04:00.0.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #2 from Ansa89 <[email protected]> ---
1) I would prefer stay on stable kernel if it's possible (which commits of 3.17-rc1 would be relevant for this bug?).
2) Yes, all of the 8169 NICs are up and running.
3) lspci -vvs 04:00.0
04:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=04, secondary=05, subordinate=05, sec-latency=32
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: f7800000-f78fffff
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-<MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mAPME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
L0 <2us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
Capabilities: [c0] Subsystem: Micro-Star International Co., Ltd. Device
7758
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #3 from Alex Williamson <[email protected]> --- (In reply to Ansa89 from comment #2)
Quoted 2 lines
1) I would prefer stay on stable kernel if it's possible (which commits of 3.17-rc1 would be relevant for this bug?).
579305f iommu/vt-d: Update to use PCI DMA aliases
e17f9ff iommu/vt-d: Use iommu_group_get_for_dev()
104a1c1 iommu/core: Create central IOMMU group lookup/creation interface
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #4 from Ansa89 <[email protected]> --- I will try 3.17-rc1 (hoping it's enough stable for home-server).
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #5 from Ansa89 <[email protected]> ---
Tested with 3.17-rc1: the errors still there, but the spam rate seems lower than 3.16.1 (with 3.16.1 I get the errors repeated a lot of times and the count grows fast; with 3.17-rc1 I get the same errors repeated less times and the count seems to grow slower).
After ~10 minutes:
dmesg | grep -i dmar
ACPI: DMAR 0x00000000C8EA83F0 0000B8 (v01 INTEL SNB 00000001 INTL
00000001)
dmar: Host address width 36
dmar: DRHD base: 0x000000fed90000 flags: 0x0
dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
dmar: DRHD base: 0x000000fed91000 flags: 0x1
dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
dmar: RMRR base: 0x000000c8d17000 end: 0x000000c8d24fff
dmar: RMRR base: 0x000000cb800000 end: 0x000000cf9fffff
DMAR: No ATSR found
[drm] DMAR active, disabling use of stolen memory
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
dmar: DRHD: handling fault status reg 3
dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ff3f4000 DMAR:[fault reason 06] PTE Read access is not set
In the end the bug seems not fixed in 3.17-rc1.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #6 from Alex Williamson <[email protected]> --- Ok, then it's probably not a result of the PCIe-to-PCI bridge since 05:00.0 is the correct requester ID for all the devices behind the bridge. Unfortunately that means that the problem may not be fixable. We're only seeing reads to a single address, which may mean the NIC is using that read to synchronize transaction ordering, ex. using a DMA read to flush a DMA write from the device. If the NIC driver has visibility of this address, then it could attempt to do a coherent mapping for the device(s) to avoid the fault. If it doesn't, then these NICs may simply be incompatible with the IOMMU.
Are these 3 separate NICs plugged into PCI slots on the motherboard or is this a single triple-port card with embedded PCIe-to-PCI bridge?
You might be able to run the IOMMU in passthrough mode with iommu=pt r8169.use_dac=1, but note the warning in modinfo "use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot." Unfortunately if you don't enable use_dac, then intel_iommu will ignore the passthrough option for these devices.
Also note that this problem has nothing to do with Virtualization/KVM. Drivers/Network or perhaps Drivers/PCI would be a more appropriate classification.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #7 from Alex Williamson <[email protected]> --- I'm guessing this might be the motherboard here: MSI ZH77A-G43
Since you're apparently trying to use VT-d on this system for KVM and therefore presumably device assignment, I'll note that you will never be able to successfully assign the conventional PCI devices separately between guests or between host and guests. The IOMMU does not have the granularity to create separate IOMMU domains per PCI slot in this topology. Also, some (all?) Realtek NICs have some strange backdoors to PCI configuration space that make them poor targets for PCI device assignment:
http://git.qemu.org/?p=qemu.git;a=commit;h=4cb47d281a995cb49e4652cb26bafb3ab2d9bd28
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
--- Comment #8 from Ansa89 <[email protected]> --- (In reply to Alex Williamson from comment #6)
Quoted 2 lines
Are these 3 separate NICs plugged into PCI slots on the motherboard or is this a single triple-port card with embedded PCIe-to-PCI bridge?
They are 3 separate NICs plugged into 3 separate PCI slots.
Quoted 4 lines
You might be able to run the IOMMU in passthrough mode with iommu=pt r8169.use_dac=1, but note the warning in modinfo "use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot." Unfortunately if you don't enable
use_dac, thenintel_iommuwill ignore the passthrough option for these devices.
I tried using "intel_iommu=pt", but it didn't work (resulted in vt-d disabled). However with "intel_iommu=on iommu=pt" the errors remain (probably because I didn't add "r8169.use_dac=1").
I'm on a 64 bit system, but I think it has nothing to with "32 bit PCI slot".
Quoted 3 lines
Also note that this problem has nothing to do with Virtualization/KVM. Drivers/Network or perhaps Drivers/PCI would be a more appropriate classification.
I searched for "IOMMU" section but it doesn't exist. I will probably change classification to "Drivers/PCI".
(In reply to Alex Williamson from comment #7)
Quoted 1 line
I'm guessing this might be the motherboard here: MSI ZH77A-G43
Yes, that is my motherboard.
Quoted 7 lines
Since you're apparently trying to use VT-d on this system for KVM and therefore presumably device assignment, I'll note that you will never be able to successfully assign the conventional PCI devices separately between guests or between host and guests. The IOMMU does not have the granularity to create separate IOMMU domains per PCI slot in this topology. Also, some (all?) Realtek NICs have some strange backdoors to PCI configuration space that make them poor targets for PCI device assignment:
Yes, I'm trying to do device assignment, but not with those NICs: I want to pass only the nVidia PCIe VGA card to guest; while all NICs (and the integrated VGA card) will remain available to host.
It would be nice if there would be a way to prevent IOMMU on these NICs (or something like that).
SIDE NOTE: in the qemu commit they talk about RTL8168, but I have real RTL8169 devices (the only RTL8168 device is the integrated NIC and for that device I'm using r8168 driver from realtek compiled by hand).
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 82761] DMAR:[fault reason 06] PTE Read access is not set
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=82761
Alan <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] Component|kvm |Network Version|unspecified |2.5 Assignee|virtualization_kvm@kernel-b |drivers_network@kernel-bugs |ugs.osdl.org |.osdl.org Product|Virtualization |Drivers
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.