Skip to contentSkip to conversation

iommu/amd: Workaround for ERBT1312

PATCH Andreas Hartmann 3 participants 12 messages Jun 28, 2013 – Jun 29, 2013 2 messages not in the archive IOMMU

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hello Joerg, hello Alex,

the subsequent patch and the patch "iommu/amd: Re-enable IOMMU event log interrupt after handling." 925fe08bce38d1ff052fe2209b9e2b8d5fbb7f98 spread /var/log/messages with the following line (> 700 lines/second) right after loading vfio:

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

lspci -vvvs 0:14.0
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Besides the enormous pollution I couldn't see any malfunction at all.
At first, I didn't realised it at all (-> the SSD was fast enough to
cover it silently). I saw it the first time I rebooted because X didn't start any more because the /var partition was completely full.

I removed the two mentioned patches and all is working fine again as before.

Any idea?

Thanks,
kind regards,
Andreas

Quoted 82 lines · Greg Kroah-Hartman

3.9-stable review patch. If anyone has any objections, please let me know.

------------------

From: Joerg Roedel <[email protected]>

commit d3263bc29706e42f74d8800807c2dedf320d77f1 upstream.

Work around an IOMMU hardware bug where clearing the EVT_INT or PPR_INT bit in the status register may race with the hardware trying to set it again. When not handled the bit might not be cleared and we lose all future event or ppr interrupts.

---
 drivers/iommu/amd_iommu.c |   34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)
drivers/iommu/amd_iommu.c +26 -8
@@ -700,14 +700,23 @@ retry:
 
 static void iommu_poll_events(struct amd_iommu *iommu)
 {
-	u32 head, tail;
+	u32 head, tail, status;
 	unsigned long flags;
 
-	/* enable event interrupts again */
-	writel(MMIO_STATUS_EVT_INT_MASK, iommu->mmio_base + MMIO_STATUS_OFFSET);
-
 	spin_lock_irqsave(&iommu->lock, flags);
 
+	/* enable event interrupts again */
+	do {
+		/*
+		 * Workaround for Erratum ERBT1312
+		 * Clearing the EVT_INT bit may race in the hardware, so read
+		 * it again and make sure it was really cleared
+		 */
+		status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
+		writel(MMIO_STATUS_EVT_INT_MASK,
+		       iommu->mmio_base + MMIO_STATUS_OFFSET);
+	} while (status & MMIO_STATUS_EVT_INT_MASK);
+
 	head = readl(iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
 	tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);
 
@@ -744,16 +753,25 @@ static void iommu_handle_ppr_entry(struc
 static void iommu_poll_ppr_log(struct amd_iommu *iommu)
 {
 	unsigned long flags;
-	u32 head, tail;
+	u32 head, tail, status;
 
 	if (iommu->ppr_log == NULL)
 		return;
 
-	/* enable ppr interrupts again */
-	writel(MMIO_STATUS_PPR_INT_MASK, iommu->mmio_base + MMIO_STATUS_OFFSET);
-
 	spin_lock_irqsave(&iommu->lock, flags);
 
+	/* enable ppr interrupts again */
+	do {
+		/*
+		 * Workaround for Erratum ERBT1312
+		 * Clearing the PPR_INT bit may race in the hardware, so read
+		 * it again and make sure it was really cleared
+		 */
+		status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
+		writel(MMIO_STATUS_PPR_INT_MASK,
+		       iommu->mmio_base + MMIO_STATUS_OFFSET);
+	} while (status & MMIO_STATUS_PPR_INT_MASK);
+
 	head = readl(iommu->mmio_base + MMIO_PPR_HEAD_OFFSET);
 	tail = readl(iommu->mmio_base + MMIO_PPR_TAIL_OFFSET);
 
#3 lore raw
Quoted 8 lines · Andreas Hartmann

Hello Joerg, hello Alex,

the subsequent patch and the patch "iommu/amd: Re-enable IOMMU event log interrupt after handling." 925fe08bce38d1ff052fe2209b9e2b8d5fbb7f98 spread /var/log/messages with the following line (> 700 lines/second) right after loading vfio:

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

That's interesting, I PXE boot my system from one NIC then use a different NIC for the iSCSI root. The PXE boot NIC now screams like this, _until_ I attach it to vfio, then it quiets down.

Quoted 11 lines

lspci -vvvs 0:14.0
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Besides the enormous pollution I couldn't see any malfunction at all.
At first, I didn't realised it at all (-> the SSD was fast enough to
cover it silently). I saw it the first time I rebooted because X didn't start any more because the /var partition was completely full.

Quoted 4 lines

I removed the two mentioned patches and all is working fine again as before.

Any idea?

Not really without some digging. I wonder if it's a new event each time or if something is just not clearing a previous event. ISTR that a boot used to often, but not always, generate a couple faults between the IOMMU being enabled and the NIC driver being loaded. All the faults I see are to the same address, so my guess is that it's getting replayed. Thanks,

Alex

Quoted 83 lines
Quoted 82 lines · Greg Kroah-Hartman

3.9-stable review patch. If anyone has any objections, please let me know.

------------------

From: Joerg Roedel <[email protected]>

commit d3263bc29706e42f74d8800807c2dedf320d77f1 upstream.

Work around an IOMMU hardware bug where clearing the EVT_INT or PPR_INT bit in the status register may race with the hardware trying to set it again. When not handled the bit might not be cleared and we lose all future event or ppr interrupts.

---
 drivers/iommu/amd_iommu.c |   34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)
drivers/iommu/amd_iommu.c +26 -8
@@ -700,14 +700,23 @@ retry:
 
 static void iommu_poll_events(struct amd_iommu *iommu)
 {
-	u32 head, tail;
+	u32 head, tail, status;
 	unsigned long flags;
 
-	/* enable event interrupts again */
-	writel(MMIO_STATUS_EVT_INT_MASK, iommu->mmio_base + MMIO_STATUS_OFFSET);
-
 	spin_lock_irqsave(&iommu->lock, flags);
 
+	/* enable event interrupts again */
+	do {
+		/*
+		 * Workaround for Erratum ERBT1312
+		 * Clearing the EVT_INT bit may race in the hardware, so read
+		 * it again and make sure it was really cleared
+		 */
+		status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
+		writel(MMIO_STATUS_EVT_INT_MASK,
+		       iommu->mmio_base + MMIO_STATUS_OFFSET);
+	} while (status & MMIO_STATUS_EVT_INT_MASK);
+
 	head = readl(iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
 	tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);
 
@@ -744,16 +753,25 @@ static void iommu_handle_ppr_entry(struc
 static void iommu_poll_ppr_log(struct amd_iommu *iommu)
 {
 	unsigned long flags;
-	u32 head, tail;
+	u32 head, tail, status;
 
 	if (iommu->ppr_log == NULL)
 		return;
 
-	/* enable ppr interrupts again */
-	writel(MMIO_STATUS_PPR_INT_MASK, iommu->mmio_base + MMIO_STATUS_OFFSET);
-
 	spin_lock_irqsave(&iommu->lock, flags);
 
+	/* enable ppr interrupts again */
+	do {
+		/*
+		 * Workaround for Erratum ERBT1312
+		 * Clearing the PPR_INT bit may race in the hardware, so read
+		 * it again and make sure it was really cleared
+		 */
+		status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
+		writel(MMIO_STATUS_PPR_INT_MASK,
+		       iommu->mmio_base + MMIO_STATUS_OFFSET);
+	} while (status & MMIO_STATUS_PPR_INT_MASK);
+
 	head = readl(iommu->mmio_base + MMIO_PPR_HEAD_OFFSET);
 	tail = readl(iommu->mmio_base + MMIO_PPR_TAIL_OFFSET);
 
maintainer
#4 lore raw

Hi Alex,

Quoted 3 lines · Alex Williamson

That's interesting, I PXE boot my system from one NIC then use a different NIC for the iSCSI root. The PXE boot NIC now screams like this, _until_ I attach it to vfio, then it quiets down.

Can you please send an example line of the reported fault? The addresses it faults on would be interesting.

Quoted 7 lines
Quoted 1 line

Any idea?

Not really without some digging. I wonder if it's a new event each time or if something is just not clearing a previous event. ISTR that a boot used to often, but not always, generate a couple faults between the IOMMU being enabled and the NIC driver being loaded. All the faults I see are to the same address, so my guess is that it's getting replayed.

Well, I think it is a problem uncovered by the patch that re-enables the event-log interrupt after it happened once. We need to find a strategy to cope with those problems.

To my mind as a quick-fix comes rate-limiting for the printks. Or we use the suppress-pf bit in the DTE to suppress all page-faults after the first one.

Joerg

#5 lore raw
Quoted 9 lines · Joerg Roedel

Hi Alex,

Quoted 3 lines · Alex Williamson

That's interesting, I PXE boot my system from one NIC then use a different NIC for the iSCSI root. The PXE boot NIC now screams like this, _until_ I attach it to vfio, then it quiets down.

Can you please send an example line of the reported fault? The addresses it faults on would be interesting.

[   99.613489] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0

domain=0x0000 address=0x000000000008f880 flags=0x0050]

Quoted 20 lines
Quoted 7 lines
Quoted 1 line

Any idea?

Not really without some digging. I wonder if it's a new event each time or if something is just not clearing a previous event. ISTR that a boot used to often, but not always, generate a couple faults between the IOMMU being enabled and the NIC driver being loaded. All the faults I see are to the same address, so my guess is that it's getting replayed.

Well, I think it is a problem uncovered by the patch that re-enables the event-log interrupt after it happened once. We need to find a strategy to cope with those problems.

To my mind as a quick-fix comes rate-limiting for the printks. Or we use the suppress-pf bit in the DTE to suppress all page-faults after the first one.

Joerg

#6 lore raw
In reply to Alex Williamson
Quoted 8 lines · Alex Williamson
Quoted 7 lines · Andreas Hartmann

Hello Joerg, hello Alex,

the subsequent patch and the patch "iommu/amd: Re-enable IOMMU event log interrupt after handling." 925fe08bce38d1ff052fe2209b9e2b8d5fbb7f98 spread /var/log/messages with the following line (> 700 lines/second) right after loading vfio:

Quoted 5 lines
Quoted 1 line

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

That's interesting, I PXE boot my system from one NIC then use a different NIC for the iSCSI root. The PXE boot NIC now screams like this, _until_ I attach it to vfio, then it quiets down.

Hmm, I just remembered an active workaround I implemented to "resolve" an error like this when starting my VM to passthrough my intel pci ethernet device since I applied a new kvm version:

qemu-kvm: -device vfio-pci,host=06:06.0: vfio: failed to set iommu for
container: Device or resource busy

qemu-kvm: -device vfio-pci,host=06:06.0: vfio: failed to setup container for group 12

qemu-kvm: -device vfio-pci,host=06:06.0: vfio: failed to get group 12

qemu-kvm: -device vfio-pci,host=06:06.0: Device 'vfio-pci' could not be initialized

The workaround was to bind the individual multifunction devices during boot one time to vfio and release them after 2 seconds again and rebind them to the original drivers as they where bound before (if it was bound to any).

I did this with a script beginning like this:

#!/bin/sh
modprobe vfio-pci

echo "1002 4385" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:00:14.0 > /sys/bus/pci/devices/0000:00:14.0/driver/unbind echo 0000:00:14.0 > /sys/bus/pci/drivers/vfio-pci/bind
...

sleep 2

echo 0000:00:14.0 > /sys/bus/pci/drivers/vfio-pci/unbind echo "1002 4385" > /sys/bus/pci/drivers/vfio-pci/remove_id
...

The logs in messages:

Jun 28 15:54:12 . kernel: [   48.860147] VFIO - User Level meta-driver version: 0.3
Jun 28 15:54:12 . kernel: [   48.875243] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

...

Therefore, the logoutput most probably started after device 14.0 was bound to vfio. If it would have started after removing vfio, I would have expected 2 seconds between the start messages of vfio and the first occurrence of the IO_PAGE_FAULT.

Today, I'm using kvm 1.3.1 and it isn't necessary to use the complete workaround anymore. It is enough to bind / unbind the pci bridge as described above before starting the VM with the passed through pci ethernet device.
Because I now don't touch the 14.0 device any more, the IO_PAGE_FAULT messages disappeared completely.

@Joerg:
Anyway, I'm going to test your provided patch tomorrow!

BTW: what does it mean: IO_PAGE_FAULT - what do I have to expect if I see this message?

Thanks,
regards,
Andreas

maintainer
#7 lore raw
In reply to Andreas Hartmann

Hi Andreas,

Quoted 7 lines · Andreas Hartmann

Hello Joerg, hello Alex,

the subsequent patch and the patch "iommu/amd: Re-enable IOMMU event log interrupt after handling." 925fe08bce38d1ff052fe2209b9e2b8d5fbb7f98 spread /var/log/messages with the following line (> 700 lines/second) right after loading vfio:

Quoted 6 lines

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

lspci -vvvs 0:14.0
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Most likely a BIOS issue that is uncovered by re-enabling the event-log interrupt patch. The device itself is only used by the BIOS and not by the Linux kernel

Quoted 7 lines

Besides the enormous pollution I couldn't see any malfunction at all.
At first, I didn't realised it at all (-> the SSD was fast enough to
cover it silently). I saw it the first time I rebooted because X didn't start any more because the /var partition was completely full.

I removed the two mentioned patches and all is working fine again as before.

Without these two patches, can you check dmesg after boot if there are other lines which report IO_PAGE_FAULTs?

Joerg

#8 lore raw

Hello Joerg,

Quoted 12 lines · Joerg Roedel

Hi Andreas,

Quoted 9 lines · Andreas Hartmann

Hello Joerg, hello Alex,

the subsequent patch and the patch "iommu/amd: Re-enable IOMMU event log interrupt after handling." 925fe08bce38d1ff052fe2209b9e2b8d5fbb7f98 spread /var/log/messages with the following line (> 700 lines/second) right after loading vfio:

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]

Quoted 8 lines
Quoted 4 lines

lspci -vvvs 0:14.0
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Most likely a BIOS issue that is uncovered by re-enabling the event-log interrupt patch. The device itself is only used by the BIOS and not by the Linux kernel

Thanks for this info! Good to know.

[...]

Quoted 5 lines
Quoted 2 lines

I removed the two mentioned patches and all is working fine again as before.

Without these two patches, can you check dmesg after boot if there are other lines which report IO_PAGE_FAULTs?

You're right, there is exactly one entry directly after loading of vfio. I can see this message, too, with linux 3.4.43.

Regards,
Andreas

maintainer
#9 lore raw

Alex, Andreas,

Quoted 2 lines · Andreas Hartmann

You're right, there is exactly one entry directly after loading of vfio. I can see this message, too, with linux 3.4.43.

Can you please test this patch? It should reduce the noise significantly, but a few of those error messages are still expected.

drivers/iommu/amd_iommu_init.c +1 -0
index e3c2d74..74e1d1c 100644
@@ -1431,6 +1431,7 @@ static void init_device_table_dma(void)
 	for (devid = 0; devid <= amd_iommu_last_bdf; ++devid) {
 		set_dev_entry_bit(devid, DEV_ENTRY_VALID);
 		set_dev_entry_bit(devid, DEV_ENTRY_TRANSLATION);
+		set_dev_entry_bit(devid, DEV_ENTRY_ONE_FAULT);
 	}
 }
 
drivers/iommu/amd_iommu_types.h +1 -0
index 083f98c..4872b68 100644
@@ -173,6 +173,7 @@
 #define DEV_ENTRY_TRANSLATION   0x01
 #define DEV_ENTRY_IR            0x3d
 #define DEV_ENTRY_IW            0x3e
+#define DEV_ENTRY_ONE_FAULT	0x61
 #define DEV_ENTRY_NO_PAGE_FAULT	0x62
 #define DEV_ENTRY_EX            0x67
 #define DEV_ENTRY_SYSMGT1       0x68
#10 lore raw
Quoted 8 lines · Joerg Roedel

Alex, Andreas,

Quoted 2 lines · Andreas Hartmann

You're right, there is exactly one entry directly after loading of vfio. I can see this message, too, with linux 3.4.43.

Can you please test this patch? It should reduce the noise significantly, but a few of those error messages are still expected.

Seems to work for me. I generated a few errors, but each one seems unique and is more like what I would expect if the PXE ROM left the NIC running:

[    2.574155] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x001b address=0x00000000000946b0 flags=0x0020]                                        
[    2.574157] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x001b address=0x000000000008f8d0 flags=0x0020]                                        
[    4.992074] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x001b address=0x000000000008f880 flags=0x0000] 

I booted the system four times and twice got just three log messages, like above, and twice got no faults. Thanks,

Alex

Quoted 25 lines
drivers/iommu/amd_iommu_init.c +1 -0
index e3c2d74..74e1d1c 100644
@@ -1431,6 +1431,7 @@ static void init_device_table_dma(void)
 	for (devid = 0; devid <= amd_iommu_last_bdf; ++devid) {
 		set_dev_entry_bit(devid, DEV_ENTRY_VALID);
 		set_dev_entry_bit(devid, DEV_ENTRY_TRANSLATION);
+		set_dev_entry_bit(devid, DEV_ENTRY_ONE_FAULT);
 	}
 }
 
drivers/iommu/amd_iommu_types.h +1 -0
index 083f98c..4872b68 100644
@@ -173,6 +173,7 @@
 #define DEV_ENTRY_TRANSLATION   0x01
 #define DEV_ENTRY_IR            0x3d
 #define DEV_ENTRY_IW            0x3e
+#define DEV_ENTRY_ONE_FAULT	0x61
 #define DEV_ENTRY_NO_PAGE_FAULT	0x62
 #define DEV_ENTRY_EX            0x67
 #define DEV_ENTRY_SYSMGT1       0x68
#11 lore raw
In reply to Joerg Roedel
Quoted 8 lines · Joerg Roedel

Alex, Andreas,

Quoted 2 lines · Andreas Hartmann

You're right, there is exactly one entry directly after loading of vfio. I can see this message, too, with linux 3.4.43.

Can you please test this patch? It should reduce the noise significantly, but a few of those error messages are still expected.

Sorry, but it doesn't work for me at all :-(. Behaviour is unchanged. It is exactly as described in the other mail: at the moment of binding vfio to 14.0, the fire begins.

echo "1002 4385" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:00:14.0 > /sys/bus/pci/devices/0000:00:14.0/driver/unbind echo 0000:00:14.0 > /sys/bus/pci/drivers/vfio-pci/bind

Regards,
Andreas

maintainer
#12 lore raw
Quoted 3 lines · Andreas Hartmann

Sorry, but it doesn't work for me at all :-(. Behaviour is unchanged. It is exactly as described in the other mail: at the moment of binding vfio to 14.0, the fire begins.

Hmm, VFIO attaches the device to a new domain. That clears the bit, how about this patch:

drivers/iommu/amd_iommu.c +2 -0
index 1a5285b..cdf346f 100644
@@ -2111,6 +2111,8 @@ static void set_dte_entry(u16 devid, struct protection_domain *domain, bool ats)
 
 		tmp = DTE_GCR3_VAL_C(gcr3) << DTE_GCR3_SHIFT_C;
 		flags    |= tmp;
+
+		flags	 |= DTE_FLAG_SE;
 	}
 
 	flags &= ~(0xffffUL);
drivers/iommu/amd_iommu_init.c +1 -0
index e3c2d74..74e1d1c 100644
@@ -1431,6 +1431,7 @@ static void init_device_table_dma(void)
 	for (devid = 0; devid <= amd_iommu_last_bdf; ++devid) {
 		set_dev_entry_bit(devid, DEV_ENTRY_VALID);
 		set_dev_entry_bit(devid, DEV_ENTRY_TRANSLATION);
+		set_dev_entry_bit(devid, DEV_ENTRY_ONE_FAULT);
 	}
 }
 
drivers/iommu/amd_iommu_types.h +2 -0
index 083f98c..2104ca4 100644
@@ -173,6 +173,7 @@
 #define DEV_ENTRY_TRANSLATION   0x01
 #define DEV_ENTRY_IR            0x3d
 #define DEV_ENTRY_IW            0x3e
+#define DEV_ENTRY_ONE_FAULT	0x61
 #define DEV_ENTRY_NO_PAGE_FAULT	0x62
 #define DEV_ENTRY_EX            0x67
 #define DEV_ENTRY_SYSMGT1       0x68
@@ -282,6 +283,7 @@
 #define IOMMU_PTE_IW (1ULL << 62)
 
 #define DTE_FLAG_IOTLB	(0x01UL << 32)
+#define DTE_FLAG_SE	(0x01UL << 33)
 #define DTE_FLAG_GV	(0x01ULL << 55)
 #define DTE_GLX_SHIFT	(56)
 #define DTE_GLX_MASK	(3)
#13 lore raw
Quoted 7 lines · Joerg Roedel
Quoted 3 lines · Andreas Hartmann

Sorry, but it doesn't work for me at all :-(. Behaviour is unchanged. It is exactly as described in the other mail: at the moment of binding vfio to 14.0, the fire begins.

Hmm, VFIO attaches the device to a new domain. That clears the bit, how about this patch:

Didn't help, too :-(

Regards,
Andreas