Skip to contentSkip to conversation

VFIO_MAP_DMA error EINVAL

vfio-users Alex Williamson 2 participants 2 messages Dec 11, 2019 1 message not in the archive IOMMU

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 8H43uI497jxPZ2Q0aUNLBWF-djbGHynLoblrAzt2KVqgp1cBsnTmVyVGg_wZMRoS0zItTSs76Uja0hUokWtSpv2U_0vMPgHsrWcUPoF86kM=@protonmail.com

#1 lore raw
Quoted 29 lines · On Wed, 11 Dec 2019 13:17:18 +0000 cprt <[email protected]> wrote:

Hello,
I am using VFIO with QEMU trying to passthrough my audio device.

I successfully did this operation with my previous system, with a 7th generation intel and an older kernel. Now I am using a 10th generation intel and a newer kernel (5.4), and I can no longer make this work.

Looking at the QEMU code and errors, I can see that this call is failing:
ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map)
with error 22 (EINVAL).

This is the qemu log:
qemu-system-x86_64: -device vfio-pci,host=00:1f.3: vfio_dma_map(0x7f86fdc7c480, 0xc0000, 0x7ff40000, 0x7f83f20c0000) = -22 (Invalid argument)

I have setup my system as follows:

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=5b780644-8382-11e9-a363-1b71c3bf36e9 rw loglevel=3 i915.enable_psr=0 intel_iommu=on iommu=pt i915.enable_gvt=1 kvm.ignore_msrs=1 vfio-pci.ids=8086:0284,8086:02c8,8086:02a3,8086:02a4

And this is the information of my PCI layout:

$ lspci -s 00:1f
00:1f.0 ISA bridge: Intel Corporation Device 0284
00:1f.3 Audio device: Intel Corporation Device 02c8
00:1f.4 SMBus: Intel Corporation Device 02a3
00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake SPI (flash) Controller
$ lspci -s 00:1f -n
00:1f.0 0601: 8086:0284
00:1f.3 0403: 8086:02c8
00:1f.4 0c05: 8086:02a3
00:1f.5 0c80: 8086:02a4
Quoted 3 lines

I tried (just as an experiment) to bypass the error check in the QEMU code, and the virtualized audio device works as expected for a while, then it stops.

Do you know what could generate the problem?

I think this is a result of the reserved region support added in v5.4 which intends to prevent userspace from mapping ranges it shouldn't. On my system I have:

$ lspci -nns 00:1f.
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point LPC Controller/eSPI Controller [8086:9d4e] (rev 21)
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-LM [8086:15d7] (rev 21)
$ readlink -f /sys/bus/pci/devices/0000:00:1f.3/iommu_group
/sys/kernel/iommu_groups/9
$ find /sys/kernel/iommu_groups/9/devices -type l
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4
$ cat /sys/kernel/iommu_groups/9/reserved_regions 
0x0000000000000000 0x0000000000ffffff direct
0x00000000fee00000 0x00000000feefffff msi

This direct range seems to be the trouble, your error indicates the problem occurs when QEMU tries to map the GPA range 0xc0000-0x7ff40000, which clearly overlaps 0x0-0xffffff. It seems this reserved range comes from this code:

drivers/iommu/intel-iommu.c:

#ifdef CONFIG_INTEL_IOMMU_FLOPPY_WA
        if (dev_is_pci(device)) {
                struct pci_dev *pdev = to_pci_dev(device);

                if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
                        reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
                                                      IOMMU_RESV_DIRECT);
                        if (reg)
                                list_add_tail(&reg->list, head);
                }
        }

#endif /* CONFIG_INTEL_IOMMU_FLOPPY_WA */

We can see above that we do have an ISA bridge in the same IOMMU group as the audio device, so this is effectively fallout from the poor grouping of these onboard devices.

This code was introduced in v5.3 via:

commit d850c2ee5fe2259968e3889624ad22ea15cb4a38

Author: Lu Baolu <[email protected]>
Date:   Sat May 25 13:41:24 2019 +0800

iommu/vt-d: Expose ISA direct mapping region via iommu_get_resv_regions

To support mapping ISA region via iommu_group_create_direct_mappings, make sure its exposed by iommu_get_resv_regions.

The original Kconfig option this relates to was added long ago in:

commit 49a0429e53f29109cbf1eadd89497286ba81f1ae

Author: Keshavamurthy, Anil S <[email protected]>
Date:   Sun Oct 21 16:41:57 2007 -0700

Intel IOMMU: Iommu floppy workaround

This config option (DMAR_FLPY_WA) sets up 1:1 mapping for the floppy device so that the floppy device which does not use DMA api's will continue to work.

Once the floppy driver starts using DMA api's this config option can be turn off or this patch can be yanked out of kernel at that time.

So AIUI the original floppy workaround created an identity map for the range 0-16MB. The reserved region attempts to reflect that reservation to userspace, however I believe this is a software imposed reserved region for the benefit of legacy kernel drivers and should therefore use the relaxable version of the reserved region as introduced in adfd37382090 ("iommu: Introduce IOMMU_RESV_DIRECT_RELAXABLE reserved memory regions"). vfio is able to allow userspace to ignore relaxable direct mapping reserved ranges. I think the fix is then simply:

drivers/iommu/intel-iommu.c +1 -1
index 0c8d81f56a30..6eb0dd7489a1 100644
@@ -5737,7 +5737,7 @@ static void intel_iommu_get_resv_regions(struct device *device,
 
 		if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
 			reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
-						      IOMMU_RESV_DIRECT);
+						   IOMMU_RESV_DIRECT_RELAXABLE);
 			if (reg)
 				list_add_tail(&reg->list, head);
 		}

If you're able to build a kernel with this change, verification would be appreciated. I'll post a patch with stable fixes tags unless there are objections to this solution. Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#2 lore raw

Hello Alex,
I tried the suggested changes on kernel 5.4.2 and now it is working perfectly.

Thank you for your detailed answer!

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Quoted 145 lines · Alex Williamson
Quoted 10 lines · On Wed, 11 Dec 2019 13:17:18 +0000 cprt [email protected] wrote:

Hello,
I am using VFIO with QEMU trying to passthrough my audio device.
I successfully did this operation with my previous system, with a 7th generation intel and an older kernel.
Now I am using a 10th generation intel and a newer kernel (5.4), and I can no longer make this work.
Looking at the QEMU code and errors, I can see that this call is failing:
ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map)
with error 22 (EINVAL).
This is the qemu log:
qemu-system-x86_64: -device vfio-pci,host=00:1f.3: vfio_dma_map(0x7f86fdc7c480, 0xc0000, 0x7ff40000, 0x7f83f20c0000) = -22 (Invalid argument) I have setup my system as follows:

Quoted 15 lines
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=5b780644-8382-11e9-a363-1b71c3bf36e9 rw loglevel=3 i915.enable_psr=0 intel_iommu=on iommu=pt i915.enable_gvt=1 kvm.ignore_msrs=1 vfio-pci.ids=8086:0284,8086:02c8,8086:02a3,8086:02a4
And this is the information of my PCI layout:
$ lspci -s 00:1f
00:1f.0 ISA bridge: Intel Corporation Device 0284
00:1f.3 Audio device: Intel Corporation Device 02c8
00:1f.4 SMBus: Intel Corporation Device 02a3
00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake SPI (flash) Controller
$ lspci -s 00:1f -n
00:1f.0 0601: 8086:0284
00:1f.3 0403: 8086:02c8
00:1f.4 0c05: 8086:02a3
00:1f.5 0c80: 8086:02a4
I tried (just as an experiment) to bypass the error check in the QEMU code, and the virtualized audio device works as expected for a while, then it stops.
Do you know what could generate the problem?

I think this is a result of the reserved region support added in v5.4 which intends to prevent userspace from mapping ranges it shouldn't. On my system I have:

$ lspci -nns 00:1f.
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point LPC Controller/eSPI Controller [8086:9d4e] (rev 21)
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-LM [8086:15d7] (rev 21)
$ readlink -f /sys/bus/pci/devices/0000:00:1f.3/iommu_group
/sys/kernel/iommu_groups/9
$ find /sys/kernel/iommu_groups/9/devices -type l
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4
$ cat /sys/kernel/iommu_groups/9/reserved_regions
0x0000000000000000 0x0000000000ffffff direct
0x00000000fee00000 0x00000000feefffff msi

This direct range seems to be the trouble, your error indicates the problem occurs when QEMU tries to map the GPA range 0xc0000-0x7ff40000, which clearly overlaps 0x0-0xffffff. It seems this reserved range comes from this code:

drivers/iommu/intel-iommu.c:
#ifdef CONFIG_INTEL_IOMMU_FLOPPY_WA
if (dev_is_pci(device)) {
struct pci_dev *pdev = to_pci_dev(device);

if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {

                        reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
                                                      IOMMU_RESV_DIRECT);
                        if (reg)
                                list_add_tail(&reg->list, head);

                }
        }

#endif /* CONFIG_INTEL_IOMMU_FLOPPY_WA */

We can see above that we do have an ISA bridge in the same IOMMU group as the audio device, so this is effectively fallout from the poor grouping of these onboard devices.

This code was introduced in v5.3 via:

commit d850c2ee5fe2259968e3889624ad22ea15cb4a38
Author: Lu Baolu [email protected]
Date: Sat May 25 13:41:24 2019 +0800

iommu/vt-d: Expose ISA direct mapping region via iommu_get_resv_regions

To support mapping ISA region via iommu_group_create_direct_mappings, make sure its exposed by iommu_get_resv_regions.

The original Kconfig option this relates to was added long ago in:

commit 49a0429e53f29109cbf1eadd89497286ba81f1ae
Author: Keshavamurthy, Anil S [email protected]
Date: Sun Oct 21 16:41:57 2007 -0700

Intel IOMMU: Iommu floppy workaround

This config option (DMAR_FLPY_WA) sets up 1:1 mapping for the floppy device so that the floppy device which does not use DMA api's will continue to work.

Once the floppy driver starts using DMA api's this config option can be turn off or this patch can be yanked out of kernel at that time.

So AIUI the original floppy workaround created an identity map for the range 0-16MB. The reserved region attempts to reflect that reservation to userspace, however I believe this is a software imposed reserved region for the benefit of legacy kernel drivers and should therefore use the relaxable version of the reserved region as introduced in adfd37382090 ("iommu: Introduce IOMMU_RESV_DIRECT_RELAXABLE reserved memory regions"). vfio is able to allow userspace to ignore relaxable direct mapping reserved ranges. I think the fix is then simply:

drivers/iommu/intel-iommu.c +0 -2
index 0c8d81f56a30..6eb0dd7489a1 100644
@@ -5737,7 +5737,7 @@ static void intel_iommu_get_resv_regions(struct device *device,
 
 if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
 
    		reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
 
 
-         				      IOMMU_RESV_DIRECT);
 
 
 
-         				   IOMMU_RESV_DIRECT_RELAXABLE);
          	if (reg)
          		list_add_tail(&reg->list, head);
 
          }
Quoted 5 lines

If you're able to build a kernel with this change, verification would be appreciated. I'll post a patch with stable fixes tags unless there are objections to this solution. Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu