Skip to contentSkip to conversation

Fix virtio-blk issue with SWIOTLB

PATCH In review Joerg Roedel 7 participants 52 messages Jan 10, 2019 – Jan 16, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Fix virtio-blk issue with SWIOTLB

Hi,

there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.

Please review.

Thanks,

Joerg

Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introduce virtio_max_dma_size()
virtio-blk: Consider virtio_max_dma_size() for maximum segment size

 drivers/block/virtio_blk.c   | 10 ++++++----
 drivers/virtio/virtio_ring.c | 11 +++++++++++
 include/linux/swiotlb.h      | 12 ++++++++++++
 include/linux/virtio.h       |  2 ++
 4 files changed, 31 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/3 swiotlb: Export maximum allocation size

From: Joerg Roedel <[email protected]>

The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.

This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.

---
 include/linux/swiotlb.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
index 7c007ed7505f..0bcc80a97036 100644
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)
 	return paddr >= io_tlb_start && paddr < io_tlb_end;
 }
 
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	return ((1UL << IO_TLB_SHIFT) * IO_TLB_SEGSIZE);
+}
+
 bool swiotlb_map(struct device *dev, phys_addr_t *phys, dma_addr_t *dma_addr,
 		size_t size, enum dma_data_direction dir, unsigned long attrs);
 void __init swiotlb_exit(void);
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void)
 {
 	return 0;
 }
+
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	/* There is no limit when SWIOTLB isn't used */
+	return ~0UL;
+}
+
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

Quoted 8 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.

This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

Quoted 39 lines
---
 include/linux/swiotlb.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
index 7c007ed7505f..0bcc80a97036 100644
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)
 	return paddr >= io_tlb_start && paddr < io_tlb_end;
 }
 
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	return ((1UL << IO_TLB_SHIFT) * IO_TLB_SEGSIZE);
+}
+
 bool swiotlb_map(struct device *dev, phys_addr_t *phys, dma_addr_t *dma_addr,
 		size_t size, enum dma_data_direction dir, unsigned long attrs);
 void __init swiotlb_exit(void);
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void)
 {
 	return 0;
 }
+
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	/* There is no limit when SWIOTLB isn't used */
+	return ~0UL;
+}
+
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Regards,

Joerg

#4 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Or alternatively we could make SWIOTLB handle bigger sizes..

Quoted 4 lines

Regards,

Joerg

#5 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Quoted 7 lines

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Or alternatively we could make SWIOTLB handle bigger sizes..

Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.

Quoted 4 lines
Quoted 4 lines

Regards,

Joerg

#6 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Quoted 6 lines
Quoted 1 line

Or alternatively we could make SWIOTLB handle bigger sizes..

Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.

Yes, we should. Both the existing DMA size communicated through dma_params which is set by the driver, and this new DMA-ops exposed one which needs to be added. I'm working on some preliminary patches for the first part, as I think I introduced a bug related to that in the SCSI layer in 5.0..

#7 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

In reply to Michael S. Tsirkin
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Quoted 6 lines
Quoted 1 line

Or alternatively we could make SWIOTLB handle bigger sizes..

Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.

Yes, we should. Both the existing DMA size communicated through dma_params which is set by the driver, and this new DMA-ops exposed one which needs to be added. I'm working on some preliminary patches for the first part, as I think I introduced a bug related to that in the SCSI layer in 5.0..

#8 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

In reply to Konrad Rzeszutek Wilk
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Quoted 7 lines

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Or alternatively we could make SWIOTLB handle bigger sizes..

Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.

Quoted 4 lines
Quoted 4 lines

Regards,

Joerg

#9 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

In reply to Joerg Roedel
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).

But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.

Or alternatively we could make SWIOTLB handle bigger sizes..

Quoted 4 lines

Regards,

Joerg

#10 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

In reply to Konrad Rzeszutek Wilk
Quoted 2 lines · Konrad Rzeszutek Wilk

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.

Regards,

Joerg

#11 lore raw

Re: [PATCH 1/3] swiotlb: Export maximum allocation size

In reply to Joerg Roedel
Quoted 8 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.

This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.

Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.

Quoted 39 lines
---
 include/linux/swiotlb.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
index 7c007ed7505f..0bcc80a97036 100644
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)
 	return paddr >= io_tlb_start && paddr < io_tlb_end;
 }
 
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	return ((1UL << IO_TLB_SHIFT) * IO_TLB_SEGSIZE);
+}
+
 bool swiotlb_map(struct device *dev, phys_addr_t *phys, dma_addr_t *dma_addr,
 		size_t size, enum dma_data_direction dir, unsigned long attrs);
 void __init swiotlb_exit(void);
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void)
 {
 	return 0;
 }
+
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	/* There is no limit when SWIOTLB isn't used */
+	return ~0UL;
+}
+
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
Signature · 1 line
2.17.1
#12 lore raw

PATCH 1/3 swiotlb: Export maximum allocation size

In reply to Joerg Roedel

From: Joerg Roedel <[email protected]>

The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.

This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.

---
 include/linux/swiotlb.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
index 7c007ed7505f..0bcc80a97036 100644
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)
 	return paddr >= io_tlb_start && paddr < io_tlb_end;
 }
 
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	return ((1UL << IO_TLB_SHIFT) * IO_TLB_SEGSIZE);
+}
+
 bool swiotlb_map(struct device *dev, phys_addr_t *phys, dma_addr_t *dma_addr,
 		size_t size, enum dma_data_direction dir, unsigned long attrs);
 void __init swiotlb_exit(void);
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void)
 {
 	return 0;
 }
+
+static inline size_t swiotlb_max_alloc_size(void)
+{
+	/* There is no limit when SWIOTLB isn't used */
+	return ~0UL;
+}
+
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
Signature · 1 line
2.17.1
#13 lore raw

PATCH 2/3 virtio: Introduce virtio_max_dma_size()

In reply to Joerg Roedel

From: Joerg Roedel <[email protected]>

This function returns the maximum segment size for a single dma transaction of a virtio device. The possible limit comes from the SWIOTLB implementation in the Linux kernel, that has an upper limit of (currently) 256kb of contiguous memory it can map.

The functions return the lower limie when SWIOTLB is used for DMA transactions of the device and -1U otherwise.

---
 drivers/virtio/virtio_ring.c | 11 +++++++++++
 include/linux/virtio.h       |  2 ++
 2 files changed, 13 insertions(+)
drivers/virtio/virtio_ring.c +11 -0
index cd7e755484e3..c8d229da203e 100644
@@ -266,6 +266,17 @@ static bool vring_use_dma_api(struct virtio_device *vdev)
 	return false;
 }
 
+size_t virtio_max_dma_size(struct virtio_device *vdev)
+{
+	const struct dma_map_ops *ops = get_dma_ops(&vdev->dev);
+	size_t max_segment_size = -1U;
+
+	if (vring_use_dma_api(vdev) && dma_is_direct(ops))
+		max_segment_size = swiotlb_max_alloc_size();
+
+	return max_segment_size;
+}
+
 static void *vring_alloc_queue(struct virtio_device *vdev, size_t size,
 			      dma_addr_t *dma_handle, gfp_t flag)
 {
include/linux/virtio.h +2 -0
index fa1b5da2804e..673fe3ef3607 100644
@@ -157,6 +157,8 @@ int virtio_device_freeze(struct virtio_device *dev);
 int virtio_device_restore(struct virtio_device *dev);
 #endif
 
+size_t virtio_max_dma_size(struct virtio_device *vdev);
+
 #define virtio_device_for_each_vq(vdev, vq) \
 	list_for_each_entry(vq, &vdev->vqs, list)
 
Signature · 1 line
2.17.1
#14 lore raw

PATCH 2/3 virtio: Introduce virtio_max_dma_size()

In reply to Joerg Roedel

From: Joerg Roedel <[email protected]>

This function returns the maximum segment size for a single dma transaction of a virtio device. The possible limit comes from the SWIOTLB implementation in the Linux kernel, that has an upper limit of (currently) 256kb of contiguous memory it can map.

The functions return the lower limie when SWIOTLB is used for DMA transactions of the device and -1U otherwise.

---
 drivers/virtio/virtio_ring.c | 11 +++++++++++
 include/linux/virtio.h       |  2 ++
 2 files changed, 13 insertions(+)
drivers/virtio/virtio_ring.c +11 -0
index cd7e755484e3..c8d229da203e 100644
@@ -266,6 +266,17 @@ static bool vring_use_dma_api(struct virtio_device *vdev)
 	return false;
 }
 
+size_t virtio_max_dma_size(struct virtio_device *vdev)
+{
+	const struct dma_map_ops *ops = get_dma_ops(&vdev->dev);
+	size_t max_segment_size = -1U;
+
+	if (vring_use_dma_api(vdev) && dma_is_direct(ops))
+		max_segment_size = swiotlb_max_alloc_size();
+
+	return max_segment_size;
+}
+
 static void *vring_alloc_queue(struct virtio_device *vdev, size_t size,
 			      dma_addr_t *dma_handle, gfp_t flag)
 {
include/linux/virtio.h +2 -0
index fa1b5da2804e..673fe3ef3607 100644
@@ -157,6 +157,8 @@ int virtio_device_freeze(struct virtio_device *dev);
 int virtio_device_restore(struct virtio_device *dev);
 #endif
 
+size_t virtio_max_dma_size(struct virtio_device *vdev);
+
 #define virtio_device_for_each_vq(vdev, vq) \
 	list_for_each_entry(vq, &vdev->vqs, list)
 
Signature · 1 line
2.17.1
#15 lore raw

PATCH 3/3 virtio-blk: Consider virtio_max_dma_size() for maximum segment size

In reply to Joerg Roedel

From: Joerg Roedel <[email protected]>

Segments can't be larger than the maximum DMA allocation size supported by virtio on the platform. Take that into account when setting the maximum segment size for a block device.

---
 drivers/block/virtio_blk.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +6 -4
index b16a887bbd02..6062faa8d213 100644
@@ -723,7 +723,7 @@ static int virtblk_probe(struct virtio_device *vdev)
 	struct request_queue *q;
 	int err, index;
 
-	u32 v, blk_size, sg_elems, opt_io_size;
+	u32 v, blk_size, seg_size, sg_elems, opt_io_size;
 	u16 min_io_size;
 	u8 physical_block_exp, alignment_offset;
 
@@ -826,14 +826,16 @@ static int virtblk_probe(struct virtio_device *vdev)
 	/* No real sector limit. */
 	blk_queue_max_hw_sectors(q, -1U);
 
+	seg_size = virtio_max_dma_size(vdev);
+
 	/* Host can optionally specify maximum segment size and number of
 	 * segments. */
 	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
 				   struct virtio_blk_config, size_max, &v);
 	if (!err)
-		blk_queue_max_segment_size(q, v);
-	else
-		blk_queue_max_segment_size(q, -1U);
+		seg_size = min(v, seg_size);
+
+	blk_queue_max_segment_size(q, seg_size);
 
 	/* Host can optionally specify the block size of the device */
 	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_BLK_SIZE,
Signature · 1 line
2.17.1
#16 lore raw

PATCH 3/3 virtio-blk: Consider virtio_max_dma_size() for maximum segment size

In reply to Joerg Roedel

From: Joerg Roedel <[email protected]>

Segments can't be larger than the maximum DMA allocation size supported by virtio on the platform. Take that into account when setting the maximum segment size for a block device.

---
 drivers/block/virtio_blk.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +6 -4
index b16a887bbd02..6062faa8d213 100644
@@ -723,7 +723,7 @@ static int virtblk_probe(struct virtio_device *vdev)
 	struct request_queue *q;
 	int err, index;
 
-	u32 v, blk_size, sg_elems, opt_io_size;
+	u32 v, blk_size, seg_size, sg_elems, opt_io_size;
 	u16 min_io_size;
 	u8 physical_block_exp, alignment_offset;
 
@@ -826,14 +826,16 @@ static int virtblk_probe(struct virtio_device *vdev)
 	/* No real sector limit. */
 	blk_queue_max_hw_sectors(q, -1U);
 
+	seg_size = virtio_max_dma_size(vdev);
+
 	/* Host can optionally specify maximum segment size and number of
 	 * segments. */
 	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
 				   struct virtio_blk_config, size_max, &v);
 	if (!err)
-		blk_queue_max_segment_size(q, v);
-	else
-		blk_queue_max_segment_size(q, -1U);
+		seg_size = min(v, seg_size);
+
+	blk_queue_max_segment_size(q, seg_size);
 
 	/* Host can optionally specify the block size of the device */
 	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_BLK_SIZE,
Signature · 1 line
2.17.1
#17 lore raw
In reply to Joerg Roedel
Quoted 8 lines · Joerg Roedel

Hi,

there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.

Quoted 5 lines

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.

Quoted 4 lines

This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.

I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).

#18 lore raw

Hi Christoph,

Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Joerg Roedel

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.

'Allocation' in the sense that there are address ranges allocated, not memory, but mappings would also be right.

Quoted 4 lines

I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).

I though about exposing it through DMA-API, but didn't go that route as I didn't want to extend a generic API for some SWIOTLB specifics. But if there are more implementations that have a size limitation it makes certainly sense to put it into the DMA-API. I'll change that in the next version.

Thanks,

Joerg

#19 lore raw
In reply to Christoph Hellwig

Hi Christoph,

Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Joerg Roedel

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.

'Allocation' in the sense that there are address ranges allocated, not memory, but mappings would also be right.

Quoted 4 lines

I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).

I though about exposing it through DMA-API, but didn't go that route as I didn't want to extend a generic API for some SWIOTLB specifics. But if there are more implementations that have a size limitation it makes certainly sense to put it into the DMA-API. I'll change that in the next version.

Thanks,

Joerg

#20 lore raw
In reply to Joerg Roedel
Quoted 8 lines · Joerg Roedel

Hi,

there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.

Quoted 5 lines

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.

Quoted 4 lines

This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.

I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).

#21 lore raw
In reply to Joerg Roedel
Quoted 14 lines · Joerg Roedel

Hi,

there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

Quoted 4 lines

This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Thanks

Quoted 18 lines

Please review.

Thanks,

Joerg

Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introduce virtio_max_dma_size()
virtio-blk: Consider virtio_max_dma_size() for maximum segment size

  drivers/block/virtio_blk.c   | 10 ++++++----
  drivers/virtio/virtio_ring.c | 11 +++++++++++
  include/linux/swiotlb.h      | 12 ++++++++++++
  include/linux/virtio.h       |  2 ++
  4 files changed, 31 insertions(+), 4 deletions(-)
#22 lore raw
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Joerg

#23 lore raw
Quoted 10 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

#24 lore raw
Quoted 9 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 6 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

#25 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

#26 lore raw
Quoted 22 lines · Jason Wang
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

Signature · 1 line
MST
#27 lore raw
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 5 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 9 lines
Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

In case of SEV, emulated DMA is performed through the SWIOTLB (which bounces the encrypted buffers). The issue reported here will happen on any platform which is making use of SWIOTLB. We could easily reproduce the the virtio-blk failure if we configure swiotlb=force in non SEV guest. Unfortunately in case of SEV the SWIOTLB is must. As Jorge highlighted the main issue is limitation of the SWIOTLB, it does not support allocation/map larger than 256Kb.

Quoted 11 lines

In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

reviewer
#28 lore raw
In reply to Michael S. Tsirkin
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 5 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 8 lines
Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

Robin.

#29 lore raw
Quoted 10 lines · Robin Murphy

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

drivers/block/virtio_blk.c:virtblk_probe():

	/* Host can optionally specify maximum segment size and number of
         * segments. */
	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
				   struct virtio_blk_config, size_max, &v);
        if (!err)
		blk_queue_max_segment_size(q, v);
	else
		blk_queue_max_segment_size(q, -1U);

So it really is virtio_blk that is special here. The host could set VIRTIO_BLK_F_SIZE_MAX to paper over the problem, but I really think we need a dma_max_segment_size API that is wired up through the dma_map_ops to sort this out for real.

#30 lore raw
In reply to Robin Murphy
Quoted 10 lines · Robin Murphy

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

drivers/block/virtio_blk.c:virtblk_probe():

	/* Host can optionally specify maximum segment size and number of
         * segments. */
	err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
				   struct virtio_blk_config, size_max, &v);
        if (!err)
		blk_queue_max_segment_size(q, v);
	else
		blk_queue_max_segment_size(q, -1U);

So it really is virtio_blk that is special here. The host could set VIRTIO_BLK_F_SIZE_MAX to paper over the problem, but I really think we need a dma_max_segment_size API that is wired up through the dma_map_ops to sort this out for real.

#31 lore raw
In reply to Robin Murphy
Quoted 33 lines · Robin Murphy
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 5 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 8 lines
Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

Quoted 5 lines

Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.

It seems reasonable to be able to ask it what it's capabilities are though.

Quoted 11 lines

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

Quoted 1 line

Robin.

I guess this is what you are looking for:

        /* Host can optionally specify maximum segment size and number of
         * segments. */
        err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
                                   struct virtio_blk_config, size_max, &v);
        if (!err)
                blk_queue_max_segment_size(q, v);
        else
                blk_queue_max_segment_size(q, -1U);

virtio isn't the only caller with a value >64K:

$ git grep -A1 blk_queue_max_segment_size
Documentation/block/biodoc.txt: blk_queue_max_segment_size(q, max_seg_size)
Documentation/block/biodoc.txt-         Maximum size of a clustered segment, 64kB default.
--
block/blk-settings.c: * blk_queue_max_segment_size - set max segment size for blk_rq_map_sg
block/blk-settings.c- * @q:  the request queue for the device
--
block/blk-settings.c:void blk_queue_max_segment_size(struct request_queue *q, unsigned int max_size)
block/blk-settings.c-{
--
block/blk-settings.c:EXPORT_SYMBOL(blk_queue_max_segment_size);
block/blk-settings.c-
--
drivers/block/mtip32xx/mtip32xx.c:      blk_queue_max_segment_size(dd->queue, 0x400000);
drivers/block/mtip32xx/mtip32xx.c-      blk_queue_io_min(dd->queue, 4096);
--
drivers/block/nbd.c:    blk_queue_max_segment_size(disk->queue, UINT_MAX);
drivers/block/nbd.c-    blk_queue_max_segments(disk->queue, USHRT_MAX);
--
drivers/block/ps3disk.c:        blk_queue_max_segment_size(queue, dev->bounce_size);
drivers/block/ps3disk.c-
--
drivers/block/ps3vram.c:        blk_queue_max_segment_size(queue, BLK_MAX_SEGMENT_SIZE);
drivers/block/ps3vram.c-        blk_queue_max_hw_sectors(queue, BLK_SAFE_MAX_SECTORS);
--
drivers/block/rbd.c:    blk_queue_max_segment_size(q, UINT_MAX);
drivers/block/rbd.c-    blk_queue_io_min(q, objset_bytes);
--
drivers/block/sunvdc.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/block/sunvdc.c-
--
drivers/block/virtio_blk.c:             blk_queue_max_segment_size(q, v);
drivers/block/virtio_blk.c-     else
drivers/block/virtio_blk.c:             blk_queue_max_segment_size(q, -1U);
drivers/block/virtio_blk.c-
--
drivers/block/xen-blkfront.c:   blk_queue_max_segment_size(rq, PAGE_SIZE);
drivers/block/xen-blkfront.c-
--
drivers/cdrom/gdrom.c:  blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);
drivers/cdrom/gdrom.c-  gd.disk->queue = gd.gdrom_rq;
--
drivers/memstick/core/ms_block.c:       blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/ms_block.c-                                  MS_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/memstick/core/mspro_block.c:    blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/mspro_block.c-                               MSPRO_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/mmc/core/queue.c:       blk_queue_max_segment_size(mq->queue, host->max_seg_size);
drivers/mmc/core/queue.c-
--
drivers/s390/block/dasd.c:      blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/s390/block/dasd.c-      blk_queue_segment_boundary(q, PAGE_SIZE - 1);
--
drivers/scsi/be2iscsi/be_main.c:        blk_queue_max_segment_size(sdev->request_queue, 65536);
drivers/scsi/be2iscsi/be_main.c-        return 0;
--
drivers/scsi/scsi_debug.c:      blk_queue_max_segment_size(sdp->request_queue, -1U);
drivers/scsi/scsi_debug.c-      if (sdebug_no_uld)
--
drivers/scsi/scsi_lib.c:        blk_queue_max_segment_size(q, dma_get_max_seg_size(dev));
drivers/scsi/scsi_lib.c-
--
drivers/scsi/ufs/ufshcd.c:      blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
drivers/scsi/ufs/ufshcd.c-
--
include/linux/blkdev.h:extern void blk_queue_max_segment_size(struct request_queue *, unsigned int);
include/linux/blkdev.h-extern void blk_queue_max_discard_sectors(struct request_queue *q,
--
include/linux/mmc/host.h:       unsigned int            max_seg_size;   /* see blk_queue_max_segment_size */
include/linux/mmc/host.h-       unsigned short          max_segs;       /* see blk_queue_max_segments */

Some of these devices are probably not going to work well if passed through to a SEV guest.

Going back to virtio, at some level virtio is like a stacking device so it does not necessarily need a limit.

Signature · 1 line
MST
#32 lore raw
In reply to Robin Murphy
Quoted 33 lines · Robin Murphy
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 5 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 8 lines
Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

Quoted 5 lines

Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.

It seems reasonable to be able to ask it what it's capabilities are though.

Quoted 11 lines

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

Quoted 1 line

Robin.

I guess this is what you are looking for:

        /* Host can optionally specify maximum segment size and number of
         * segments. */
        err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
                                   struct virtio_blk_config, size_max, &v);
        if (!err)
                blk_queue_max_segment_size(q, v);
        else
                blk_queue_max_segment_size(q, -1U);

virtio isn't the only caller with a value >64K:

$ git grep -A1 blk_queue_max_segment_size
Documentation/block/biodoc.txt: blk_queue_max_segment_size(q, max_seg_size)
Documentation/block/biodoc.txt-         Maximum size of a clustered segment, 64kB default.
--
block/blk-settings.c: * blk_queue_max_segment_size - set max segment size for blk_rq_map_sg
block/blk-settings.c- * @q:  the request queue for the device
--
block/blk-settings.c:void blk_queue_max_segment_size(struct request_queue *q, unsigned int max_size)
block/blk-settings.c-{
--
block/blk-settings.c:EXPORT_SYMBOL(blk_queue_max_segment_size);
block/blk-settings.c-
--
drivers/block/mtip32xx/mtip32xx.c:      blk_queue_max_segment_size(dd->queue, 0x400000);
drivers/block/mtip32xx/mtip32xx.c-      blk_queue_io_min(dd->queue, 4096);
--
drivers/block/nbd.c:    blk_queue_max_segment_size(disk->queue, UINT_MAX);
drivers/block/nbd.c-    blk_queue_max_segments(disk->queue, USHRT_MAX);
--
drivers/block/ps3disk.c:        blk_queue_max_segment_size(queue, dev->bounce_size);
drivers/block/ps3disk.c-
--
drivers/block/ps3vram.c:        blk_queue_max_segment_size(queue, BLK_MAX_SEGMENT_SIZE);
drivers/block/ps3vram.c-        blk_queue_max_hw_sectors(queue, BLK_SAFE_MAX_SECTORS);
--
drivers/block/rbd.c:    blk_queue_max_segment_size(q, UINT_MAX);
drivers/block/rbd.c-    blk_queue_io_min(q, objset_bytes);
--
drivers/block/sunvdc.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/block/sunvdc.c-
--
drivers/block/virtio_blk.c:             blk_queue_max_segment_size(q, v);
drivers/block/virtio_blk.c-     else
drivers/block/virtio_blk.c:             blk_queue_max_segment_size(q, -1U);
drivers/block/virtio_blk.c-
--
drivers/block/xen-blkfront.c:   blk_queue_max_segment_size(rq, PAGE_SIZE);
drivers/block/xen-blkfront.c-
--
drivers/cdrom/gdrom.c:  blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);
drivers/cdrom/gdrom.c-  gd.disk->queue = gd.gdrom_rq;
--
drivers/memstick/core/ms_block.c:       blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/ms_block.c-                                  MS_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/memstick/core/mspro_block.c:    blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/mspro_block.c-                               MSPRO_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/mmc/core/queue.c:       blk_queue_max_segment_size(mq->queue, host->max_seg_size);
drivers/mmc/core/queue.c-
--
drivers/s390/block/dasd.c:      blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/s390/block/dasd.c-      blk_queue_segment_boundary(q, PAGE_SIZE - 1);
--
drivers/scsi/be2iscsi/be_main.c:        blk_queue_max_segment_size(sdev->request_queue, 65536);
drivers/scsi/be2iscsi/be_main.c-        return 0;
--
drivers/scsi/scsi_debug.c:      blk_queue_max_segment_size(sdp->request_queue, -1U);
drivers/scsi/scsi_debug.c-      if (sdebug_no_uld)
--
drivers/scsi/scsi_lib.c:        blk_queue_max_segment_size(q, dma_get_max_seg_size(dev));
drivers/scsi/scsi_lib.c-
--
drivers/scsi/ufs/ufshcd.c:      blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
drivers/scsi/ufs/ufshcd.c-
--
include/linux/blkdev.h:extern void blk_queue_max_segment_size(struct request_queue *, unsigned int);
include/linux/blkdev.h-extern void blk_queue_max_discard_sectors(struct request_queue *q,
--
include/linux/mmc/host.h:       unsigned int            max_seg_size;   /* see blk_queue_max_segment_size */
include/linux/mmc/host.h-       unsigned short          max_segs;       /* see blk_queue_max_segments */

Some of these devices are probably not going to work well if passed through to a SEV guest.

Going back to virtio, at some level virtio is like a stacking device so it does not necessarily need a limit.

Signature · 5 lines
MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
reviewer
#33 lore raw
In reply to Michael S. Tsirkin
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 5 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 8 lines
Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.

Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?

Robin.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#34 lore raw
In reply to Michael S. Tsirkin
Quoted 3 lines · Michael S. Tsirkin

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

The DMA API handles the SEV case perfectly. Its just that virtio_blk supports huge segments that virtio does not generally support, but that is not related to SEV in any way.

Quoted 2 lines

In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.

No, the above happens only when we set IOMMU_PLATFORM.

Quoted 7 lines

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

No. The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.

As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.

Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.

Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.

#35 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

No.

Oh ok then.

Quoted 9 lines

The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.

As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.

I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.

So I would make that a SHOULD probably.

Quoted 3 lines

Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.

Yes. If host does not have access to all of memory it SHOULD set VIRTIO_F_ACCESS_PLATFORM.

It seems rather straight-forward to add conformance clauses for this, thanks for reminding me.

Quoted 2 lines

Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.

Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.

If you want to list the most problematic cases, e.g. in a github issue, we might get to clarifying them sooner.

Thanks!

Signature · 1 line
MST
#36 lore raw
Quoted 6 lines · Michael S. Tsirkin

I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.

So I would make that a SHOULD probably.

The problem is that without using DMA ops you can't just plug the device into a random system, which is a total no-go for periphals.

And cache flushing is not that uncommmon, e.g. every sparc systems needs it, many arm/arm64 ones, some power ones, lots of mips ones, etc.

But cache flushing is just one side of it - lots of systems have either mandatory or option IOMMUs, and without the platform access flag that doesn't work. As does that case where you have a DMA offset, which is extremly common on arm and power systems. So you might find a few non-x86 systems you can plug it in and it'll just work, but it won't be all that many. And even on x86 your device will be completely broken if the users dares to use an IOMMU.

Quoted 6 lines
Quoted 1 line

flags really needs some major updates in terms of DMA access.

Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.

At least say bus address - as said above the CPU and bus concepts of physicall addresses are different in many systems. Including x86 when you use IOMMUs.

#37 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin

I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.

So I would make that a SHOULD probably.

The problem is that without using DMA ops you can't just plug the device into a random system, which is a total no-go for periphals.

And cache flushing is not that uncommmon, e.g. every sparc systems needs it, many arm/arm64 ones, some power ones, lots of mips ones, etc.

But cache flushing is just one side of it - lots of systems have either mandatory or option IOMMUs, and without the platform access flag that doesn't work. As does that case where you have a DMA offset, which is extremly common on arm and power systems. So you might find a few non-x86 systems you can plug it in and it'll just work, but it won't be all that many. And even on x86 your device will be completely broken if the users dares to use an IOMMU.

Quoted 6 lines
Quoted 1 line

flags really needs some major updates in terms of DMA access.

Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.

At least say bus address - as said above the CPU and bus concepts of physicall addresses are different in many systems. Including x86 when you use IOMMUs.

#38 lore raw
In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

No.

Oh ok then.

Quoted 9 lines

The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.

As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.

I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.

So I would make that a SHOULD probably.

Quoted 3 lines

Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.

Yes. If host does not have access to all of memory it SHOULD set VIRTIO_F_ACCESS_PLATFORM.

It seems rather straight-forward to add conformance clauses for this, thanks for reminding me.

Quoted 2 lines

Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.

Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.

If you want to list the most problematic cases, e.g. in a github issue, we might get to clarifying them sooner.

Thanks!

Signature · 1 line
MST
#39 lore raw
In reply to Michael S. Tsirkin
Quoted 3 lines · Michael S. Tsirkin

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

The DMA API handles the SEV case perfectly. Its just that virtio_blk supports huge segments that virtio does not generally support, but that is not related to SEV in any way.

Quoted 2 lines

In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.

No, the above happens only when we set IOMMU_PLATFORM.

Quoted 7 lines

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

No. The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.

As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.

Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.

Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.

#40 lore raw
In reply to Michael S. Tsirkin
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

Regards,

Joerg

#41 lore raw
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

True. But the NULL-ops dma direct case still has two issues that might not work for virtio:

(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:

#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
	dev->dma_coherent = true;

#endif

except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field

(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.

So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..

Quoted 4 lines

Regards,

Joerg

---end quoted text---

#42 lore raw
Quoted 37 lines · Christoph Hellwig
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

True. But the NULL-ops dma direct case still has two issues that might not work for virtio:

(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:

#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
	dev->dma_coherent = true;

#endif

except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field

(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.

Quoted 3 lines

So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..

IIUC some emulated hardware has no reasonable way to specify that some devices bypass the IOMMU, and no cheap way to cover all memory with an IOMMU mapping.

So I suspect !ACCESS_PLATFORM will stay a supported configuration forever :(

Quoted 5 lines
Quoted 4 lines

Regards,

Joerg

---end quoted text---

#43 lore raw
In reply to Christoph Hellwig
Quoted 37 lines · Christoph Hellwig
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

True. But the NULL-ops dma direct case still has two issues that might not work for virtio:

(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:

#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
	dev->dma_coherent = true;

#endif

except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field

(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.

Quoted 3 lines

So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..

IIUC some emulated hardware has no reasonable way to specify that some devices bypass the IOMMU, and no cheap way to cover all memory with an IOMMU mapping.

So I suspect !ACCESS_PLATFORM will stay a supported configuration forever :(

Quoted 5 lines
Quoted 4 lines

Regards,

Joerg

---end quoted text---

#44 lore raw
In reply to Joerg Roedel
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

True. But the NULL-ops dma direct case still has two issues that might not work for virtio:

(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:

#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
    defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
	dev->dma_coherent = true;

#endif

except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field

(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.

So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..

Quoted 4 lines

Regards,

Joerg

---end quoted text---

#45 lore raw
In reply to Michael S. Tsirkin
Quoted 2 lines · Michael S. Tsirkin

Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.

Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.

Regards,

Joerg

#46 lore raw
In reply to Jason Wang
Quoted 22 lines · Jason Wang
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Quoted 4 lines

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.

In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.

Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.

Signature · 5 lines
MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#47 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.

Yes, I fully agree with you. This is probably an exact example of such problem.

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#48 lore raw
In reply to Jason Wang
Quoted 9 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Quoted 6 lines
Quoted 3 lines

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#49 lore raw
In reply to Joerg Roedel
Quoted 10 lines · Joerg Roedel
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Joerg

Thanks, have you tested vhost-net in this case. I suspect it may not work

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#50 lore raw
In reply to Jason Wang
Quoted 2 lines · Jason Wang

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.

Regards,

Joerg

#51 lore raw
In reply to Joerg Roedel
Quoted 14 lines · Joerg Roedel

Hi,

there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.

The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.

Quoted 4 lines

This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.

Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?

Thanks

Quoted 12 lines

Please review.

Thanks,

Joerg

Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introduce virtio_max_dma_size()
virtio-blk: Consider virtio_max_dma_size() for maximum segment size

Quoted 6 lines
  drivers/block/virtio_blk.c   | 10 ++++++----
  drivers/virtio/virtio_ring.c | 11 +++++++++++
  include/linux/swiotlb.h      | 12 ++++++++++++
  include/linux/virtio.h       |  2 ++
  4 files changed, 31 insertions(+), 4 deletions(-)

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization