Skip to contentSkip to conversation

vfio iommu: Add dma available capability

PATCH Superseded Matthew Rosato 2 participants 5 messages Sep 14, 2020 – Sep 15, 2020

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 vfio iommu: Add dma available capability

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added a limit to the number of concurrent DMA requests for a vfio container. However, lazy unmapping in s390 can in fact cause quite a large number of outstanding DMA requests to build up prior to being purged, potentially the entire guest DMA space. This results in unexpected 'VFIO_MAP_DMA
failed: No space left on device' conditions seen in QEMU.

This patch proposes to provide the remaining number of allowable DMA requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability. A subsequent patchset to QEMU would collect this information and use it in s390 PCI support to tap the guest on the shoulder before overrunning the vfio limit.

Changes from v1:
- Report dma_avail instead of the limit, which might not have been accurate anyway
- Text/naming changes throughout due to the above

Matthew Rosato (1):
vfio iommu: Add dma available capability

 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 16 ++++++++++++++++
 2 files changed, 33 insertions(+)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v2 1/1 vfio iommu: Add dma available capability

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added the ability to limit the number of memory backed DMA mappings. However on s390x, when lazy mapping is in use, we use a very large number of concurrent mappings. Let's provide the current allowable number of DMA mappings to userspace via the IOMMU info chain so that userspace can take appropriate mitigation.

---
 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 16 ++++++++++++++++
 2 files changed, 33 insertions(+)
drivers/vfio/vfio_iommu_type1.c +17 -0
index 5fbf0c1..15e21db 100644
@@ -2609,6 +2609,20 @@ static int vfio_iommu_migration_build_caps(struct vfio_iommu *iommu,
 	return vfio_info_add_capability(caps, &cap_mig.header, sizeof(cap_mig));
 }
 
+static int vfio_iommu_dma_avail_build_caps(struct vfio_iommu *iommu,
+					   struct vfio_info_cap *caps)
+{
+	struct vfio_iommu_type1_info_dma_avail cap_dma_avail;
+
+	cap_dma_avail.header.id = VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL;
+	cap_dma_avail.header.version = 1;
+
+	cap_dma_avail.avail = iommu->dma_avail;
+
+	return vfio_info_add_capability(caps, &cap_dma_avail.header,
+					sizeof(cap_dma_avail));
+}
+
 static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 				     unsigned long arg)
 {
@@ -2642,6 +2656,9 @@ static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 	ret = vfio_iommu_migration_build_caps(iommu, &caps);
 
 	if (!ret)
+		ret = vfio_iommu_dma_avail_build_caps(iommu, &caps);
+
+	if (!ret)
 		ret = vfio_iommu_iova_build_caps(iommu, &caps);
 
 	mutex_unlock(&iommu->lock);
include/uapi/linux/vfio.h +16 -0
index 9204705..a8cc4a5 100644
@@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
 	__u64	max_dirty_bitmap_size;		/* in bytes */
 };
 
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structures below define version 1 of this capability.
+ *
+ * max: specifies the maximum number of outstanding DMA mappings allowed.
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+	struct	vfio_info_cap_header header;
+	__u32	avail;
+};
+
+
 #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
 
 /**
Signature · 1 line
1.8.3.1
#2 lore raw
Quoted 12 lines · On Mon, 14 Sep 2020 18:25:31 -0400 Matthew Rosato <[email protected]>…

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added the ability to limit the number of memory backed DMA mappings. However on s390x, when lazy mapping is in use, we use a very large number of concurrent mappings. Let's provide the current allowable number of DMA mappings to userspace via the IOMMU info chain so that userspace can take appropriate mitigation.

---
 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 16 ++++++++++++++++
 2 files changed, 33 insertions(+)

(...)

include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_migration { · line 1046

Quoted 13 lines
include/uapi/linux/vfio.h +5 -0
index 9204705..a8cc4a5 100644
@@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
 	__u64	max_dirty_bitmap_size;		/* in bytes */
 };
 
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structures below define version 1 of this capability.

"The structure below defines..." ?

include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_migration { · line 1048

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ *
+ * max: specifies the maximum number of outstanding DMA mappings allowed.

I think you forgot to tweak that one:

"avail: specifies the current number of outstanding DMA mappings allowed."

?

Quoted 12 lines
include/uapi/linux/vfio.h +9 -0
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+	struct	vfio_info_cap_header header;
+	__u32	avail;
+};
+
+
 #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
 
 /**
#3 lore raw
Quoted 26 lines · Cornelia Huck
Quoted 12 lines · On Mon, 14 Sep 2020 18:25:31 -0400 Matthew Rosato <[email protected]>…

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added the ability to limit the number of memory backed DMA mappings. However on s390x, when lazy mapping is in use, we use a very large number of concurrent mappings. Let's provide the current allowable number of DMA mappings to userspace via the IOMMU info chain so that userspace can take appropriate mitigation.

---
  drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
  include/uapi/linux/vfio.h       | 16 ++++++++++++++++
  2 files changed, 33 insertions(+)

(...)

Quoted 8 lines
include/uapi/linux/vfio.h +0 -0
index 9204705..a8cc4a5 100644
@@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
  	__u64	max_dirty_bitmap_size;		/* in bytes */
  };
 
Quoted 8 lines

include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_migration { · line 1046

Quoted 5 lines
include/uapi/linux/vfio.h +5 -0
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structures below define version 1 of this capability.

"The structure below defines..." ?

OK

Quoted 8 lines

include/uapi/linux/vfio.h struct vfio_iommu_type1_info_cap_migration { · line 1048

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ *
+ * max: specifies the maximum number of outstanding DMA mappings allowed.

I think you forgot to tweak that one:

"avail: specifies the current number of outstanding DMA mappings allowed."

?

Yep, that's a leftover from v1 :(

Quoted 14 lines
Quoted 13 lines
include/uapi/linux/vfio.h +9 -0
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+	struct	vfio_info_cap_header header;
+	__u32	avail;
+};
+
+
  #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
 
  /**
#4 lore raw
In reply to Matthew Rosato
Quoted 7 lines · Matthew Rosato

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added a limit to the number of concurrent DMA requests for a vfio container. However, lazy unmapping in s390 can in fact cause quite a large number of outstanding DMA requests to build up prior to being purged, potentially the entire guest DMA space. This results in unexpected 'VFIO_MAP_DMA
failed: No space left on device' conditions seen in QEMU.

Quoted 5 lines

This patch proposes to provide the remaining number of allowable DMA requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability. A subsequent patchset to QEMU would collect this information and use it in s390 PCI support to tap the guest on the shoulder before overrunning the vfio limit.

Link to associated v2 QEMU patchset:
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg05067.html

Quoted 13 lines

Changes from v1:
- Report dma_avail instead of the limit, which might not have been accurate anyway
- Text/naming changes throughout due to the above

Matthew Rosato (1):
vfio iommu: Add dma available capability

  drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
  include/uapi/linux/vfio.h       | 16 ++++++++++++++++
  2 files changed, 33 insertions(+)