Fix virtio-blk issue with SWIOTLB
cover Fix virtio-blk issue with SWIOTLB
Hi,
there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.
Please review.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introduce virtio_max_dma_size()
virtio-blk: Consider virtio_max_dma_size() for maximum segment size
drivers/block/virtio_blk.c | 10 ++++++---- drivers/virtio/virtio_ring.c | 11 +++++++++++ include/linux/swiotlb.h | 12 ++++++++++++ include/linux/virtio.h | 2 ++ 4 files changed, 31 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/3 swiotlb: Export maximum allocation size
From: Joerg Roedel <[email protected]>
The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.
This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.
--- include/linux/swiotlb.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)return paddr >= io_tlb_start && paddr < io_tlb_end; } +static inline size_t swiotlb_max_alloc_size (void ) +{ +return ((1UL <<IO_TLB_SHIFT ) *IO_TLB_SEGSIZE ); +} +bool swiotlb_map (struct device *dev,phys_addr_t *phys,dma_addr_t *dma_addr,size_t size,enum dma_data_direction dir,unsigned long attrs);void __init swiotlb_exit (void );
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void) {return 0 ; } + +static inline size_t swiotlb_max_alloc_size (void ) +{ +/* There is no limit when SWIOTLB isn't used */ +return ~0UL ; +} +#endif /* CONFIG_SWIOTLB */ extern void swiotlb_print_info (void );
Signature · 1 line
2.17.1
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
Quoted 8 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.
This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.
Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.
Quoted 39 lines
Signed-off-by Joerg Roedel <[email protected]>--- include/linux/swiotlb.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)include/linux/swiotlb.h +12
-0@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)return paddr >= io_tlb_start && paddr < io_tlb_end; } +static inline size_t swiotlb_max_alloc_size (void ) +{ +return ((1UL <<IO_TLB_SHIFT ) *IO_TLB_SEGSIZE ); +} +bool swiotlb_map (struct device *dev,phys_addr_t *phys,dma_addr_t *dma_addr,size_t size,enum dma_data_direction dir,unsigned long attrs);void __init swiotlb_exit (void );@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void) {return 0 ; } + +static inline size_t swiotlb_max_alloc_size (void ) +{ +/* There is no limit when SWIOTLB isn't used */ +return ~0UL ; +} +#endif /* CONFIG_SWIOTLB */ extern void swiotlb_print_info (void );Signature · 1 line
2.17.1
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.
That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Regards,
Joerg
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Or alternatively we could make SWIOTLB handle bigger sizes..
Quoted 4 lines
Regards,
Joerg
Michael S. Tsirkin
maintainerRe: [PATCH 1/3] swiotlb: Export maximum allocation size
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Quoted 7 lines
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Or alternatively we could make SWIOTLB handle bigger sizes..
Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.
Quoted 4 lines
Quoted 4 lines
Regards,
Joerg
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Quoted 6 lines
Quoted 1 line
Or alternatively we could make SWIOTLB handle bigger sizes..
Just a thought: is it a good idea to teach
blk_queue_max_segment_sizeto get the dma size? This will help us find other devices possibly missing this check.
Yes, we should. Both the existing DMA size communicated through dma_params which is set by the driver, and this new DMA-ops exposed one which needs to be added. I'm working on some preliminary patches for the first part, as I think I introduced a bug related to that in the SCSI layer in 5.0..
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
In reply to Michael S. TsirkinQuoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Quoted 6 lines
Quoted 1 line
Or alternatively we could make SWIOTLB handle bigger sizes..
Just a thought: is it a good idea to teach
blk_queue_max_segment_sizeto get the dma size? This will help us find other devices possibly missing this check.
Yes, we should. Both the existing DMA size communicated through dma_params which is set by the driver, and this new DMA-ops exposed one which needs to be added. I'm working on some preliminary patches for the first part, as I think I introduced a bug related to that in the SCSI layer in 5.0..
Michael S. Tsirkin
maintainerRe: [PATCH 1/3] swiotlb: Export maximum allocation size
In reply to Konrad Rzeszutek WilkQuoted 8 lines · Konrad Rzeszutek Wilk
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Quoted 7 lines
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Or alternatively we could make SWIOTLB handle bigger sizes..
Just a thought: is it a good idea to teach blk_queue_max_segment_size to get the dma size? This will help us find other devices possibly missing this check.
Quoted 4 lines
Quoted 4 lines
Regards,
Joerg
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
In reply to Joerg RoedelQuoted 6 lines · Joerg Roedel
Quoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Yes. The other drivers just assumed that if there is SWIOTLB they would use the smaller size by default (as in they knew the limitation).
But I agree it would be better to have something smarter - and also convert the DRM drivers to piggy back on this.
Or alternatively we could make SWIOTLB handle bigger sizes..
Quoted 4 lines
Regards,
Joerg
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
In reply to Konrad Rzeszutek WilkQuoted 2 lines · Konrad Rzeszutek Wilk
Why not use
swiotlb_nr_tbl? That is howdrivers/gpu/drmuse to figure if they need to limit the size of pages.
That function just exports the overall size of the swiotlb aperture, no? What I need here is the maximum size for a single mapping.
Regards,
Joerg
Re: [PATCH 1/3] swiotlb: Export maximum allocation size
In reply to Joerg RoedelQuoted 8 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.
This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.
Why not use swiotlb_nr_tbl ? That is how drivers/gpu/drm use to figure if they need to limit the size of pages.
Quoted 39 lines
Signed-off-by Joerg Roedel <[email protected]>--- include/linux/swiotlb.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)include/linux/swiotlb.h +12
-0@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)return paddr >= io_tlb_start && paddr < io_tlb_end; } +static inline size_t swiotlb_max_alloc_size (void ) +{ +return ((1UL <<IO_TLB_SHIFT ) *IO_TLB_SEGSIZE ); +} +bool swiotlb_map (struct device *dev,phys_addr_t *phys,dma_addr_t *dma_addr,size_t size,enum dma_data_direction dir,unsigned long attrs);void __init swiotlb_exit (void );@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void) {return 0 ; } + +static inline size_t swiotlb_max_alloc_size (void ) +{ +/* There is no limit when SWIOTLB isn't used */ +return ~0UL ; +} +#endif /* CONFIG_SWIOTLB */ extern void swiotlb_print_info (void );Signature · 1 line
2.17.1
PATCH 1/3 swiotlb: Export maximum allocation size
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The SWIOTLB implementation has a maximum size it can allocate dma-handles for. This needs to be exported so that device drivers don't try to allocate larger chunks.
This is especially important for block device drivers like virtio-blk, that might do DMA through SWIOTLB.
--- include/linux/swiotlb.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
include/linux/swiotlb.h +12 -0
@@ -72,6 +72,11 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)return paddr >= io_tlb_start && paddr < io_tlb_end; } +static inline size_t swiotlb_max_alloc_size (void ) +{ +return ((1UL <<IO_TLB_SHIFT ) *IO_TLB_SEGSIZE ); +} +bool swiotlb_map (struct device *dev,phys_addr_t *phys,dma_addr_t *dma_addr,size_t size,enum dma_data_direction dir,unsigned long attrs);void __init swiotlb_exit (void );
@@ -95,6 +100,13 @@ static inline unsigned int swiotlb_max_segment(void) {return 0 ; } + +static inline size_t swiotlb_max_alloc_size (void ) +{ +/* There is no limit when SWIOTLB isn't used */ +return ~0UL ; +} +#endif /* CONFIG_SWIOTLB */ extern void swiotlb_print_info (void );
Signature · 1 line
2.17.1
PATCH 2/3 virtio: Introduce virtio_max_dma_size()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
This function returns the maximum segment size for a single dma transaction of a virtio device. The possible limit comes from the SWIOTLB implementation in the Linux kernel, that has an upper limit of (currently) 256kb of contiguous memory it can map.
The functions return the lower limie when SWIOTLB is used for DMA transactions of the device and -1U otherwise.
--- drivers/virtio/virtio_ring.c | 11 +++++++++++ include/linux/virtio.h | 2 ++ 2 files changed, 13 insertions(+)
drivers/virtio/virtio_ring.c +11 -0
@@ -266,6 +266,17 @@ static bool vring_use_dma_api(struct virtio_device *vdev)return false; } +size_t virtio_max_dma_size (struct virtio_device *vdev) +{ +const struct dma_map_ops *ops =get_dma_ops (&vdev->dev); +size_t max_segment_size =-1U ; + +if (vring_use_dma_api (vdev) &&dma_is_direct (ops)) + max_segment_size =swiotlb_max_alloc_size (); + +return max_segment_size; +} +static void *vring_alloc_queue (struct virtio_device *vdev,size_t size,dma_addr_t *dma_handle,gfp_t flag) {
include/linux/virtio.h +2 -0
@@ -157,6 +157,8 @@ int virtio_device_freeze(struct virtio_device *dev);int virtio_device_restore (struct virtio_device *dev);#endif +size_t virtio_max_dma_size (struct virtio_device *vdev); +#define virtio_device_for_each_vq (vdev, vq) \ list_for_each_entry(vq, &vdev->vqs, list)
Signature · 1 line
2.17.1
PATCH 2/3 virtio: Introduce virtio_max_dma_size()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
This function returns the maximum segment size for a single dma transaction of a virtio device. The possible limit comes from the SWIOTLB implementation in the Linux kernel, that has an upper limit of (currently) 256kb of contiguous memory it can map.
The functions return the lower limie when SWIOTLB is used for DMA transactions of the device and -1U otherwise.
--- drivers/virtio/virtio_ring.c | 11 +++++++++++ include/linux/virtio.h | 2 ++ 2 files changed, 13 insertions(+)
drivers/virtio/virtio_ring.c +11 -0
@@ -266,6 +266,17 @@ static bool vring_use_dma_api(struct virtio_device *vdev)return false; } +size_t virtio_max_dma_size (struct virtio_device *vdev) +{ +const struct dma_map_ops *ops =get_dma_ops (&vdev->dev); +size_t max_segment_size =-1U ; + +if (vring_use_dma_api (vdev) &&dma_is_direct (ops)) + max_segment_size =swiotlb_max_alloc_size (); + +return max_segment_size; +} +static void *vring_alloc_queue (struct virtio_device *vdev,size_t size,dma_addr_t *dma_handle,gfp_t flag) {
include/linux/virtio.h +2 -0
@@ -157,6 +157,8 @@ int virtio_device_freeze(struct virtio_device *dev);int virtio_device_restore (struct virtio_device *dev);#endif +size_t virtio_max_dma_size (struct virtio_device *vdev); +#define virtio_device_for_each_vq (vdev, vq) \ list_for_each_entry(vq, &vdev->vqs, list)
Signature · 1 line
2.17.1
PATCH 3/3 virtio-blk: Consider virtio_max_dma_size() for maximum segment size
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Segments can't be larger than the maximum DMA allocation size supported by virtio on the platform. Take that into account when setting the maximum segment size for a block device.
--- drivers/block/virtio_blk.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +6 -4
@@ -723,7 +723,7 @@ static int virtblk_probe(struct virtio_device *vdev)struct request_queue *q;int err, index; -u32 v, blk_size, sg_elems, opt_io_size; +u32 v, blk_size, seg_size, sg_elems, opt_io_size;u16 min_io_size;u8 physical_block_exp, alignment_offset;
@@ -826,14 +826,16 @@ static int virtblk_probe(struct virtio_device *vdev)/* No real sector limit. */ blk_queue_max_hw_sectors (q,-1U ); + seg_size =virtio_max_dma_size (vdev); +/* Host can optionally specify maximum segment size and number of * segments. */ err =virtio_cread_feature (vdev,VIRTIO_BLK_F_SIZE_MAX , struct virtio_blk_config, size_max, &v);if (!err) -blk_queue_max_segment_size (q, v); -else -blk_queue_max_segment_size (q,-1U ); + seg_size =min (v, seg_size); + +blk_queue_max_segment_size (q, seg_size);/* Host can optionally specify the block size of the device */ err = virtio_cread_feature(vdev, VIRTIO_BLK_F_BLK_SIZE,
Signature · 1 line
2.17.1
PATCH 3/3 virtio-blk: Consider virtio_max_dma_size() for maximum segment size
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Segments can't be larger than the maximum DMA allocation size supported by virtio on the platform. Take that into account when setting the maximum segment size for a block device.
--- drivers/block/virtio_blk.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +6 -4
@@ -723,7 +723,7 @@ static int virtblk_probe(struct virtio_device *vdev)struct request_queue *q;int err, index; -u32 v, blk_size, sg_elems, opt_io_size; +u32 v, blk_size, seg_size, sg_elems, opt_io_size;u16 min_io_size;u8 physical_block_exp, alignment_offset;
@@ -826,14 +826,16 @@ static int virtblk_probe(struct virtio_device *vdev)/* No real sector limit. */ blk_queue_max_hw_sectors (q,-1U ); + seg_size =virtio_max_dma_size (vdev); +/* Host can optionally specify maximum segment size and number of * segments. */ err =virtio_cread_feature (vdev,VIRTIO_BLK_F_SIZE_MAX , struct virtio_blk_config, size_max, &v);if (!err) -blk_queue_max_segment_size (q, v); -else -blk_queue_max_segment_size (q,-1U ); + seg_size =min (v, seg_size); + +blk_queue_max_segment_size (q, seg_size);/* Host can optionally specify the block size of the device */ err = virtio_cread_feature(vdev, VIRTIO_BLK_F_BLK_SIZE,
Signature · 1 line
2.17.1
Quoted 8 lines · Joerg Roedel
Hi,
there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.
Quoted 5 lines
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.
Quoted 4 lines
This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.
I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).
Hi Christoph,
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Joerg Roedel
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.
'Allocation' in the sense that there are address ranges allocated, not memory, but mappings would also be right.
Quoted 4 lines
I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).
I though about exposing it through DMA-API, but didn't go that route as I didn't want to extend a generic API for some SWIOTLB specifics. But if there are more implementations that have a size limitation it makes certainly sense to put it into the DMA-API. I'll change that in the next version.
Thanks,
Joerg
Hi Christoph,
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Joerg Roedel
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.
'Allocation' in the sense that there are address ranges allocated, not memory, but mappings would also be right.
Quoted 4 lines
I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).
I though about exposing it through DMA-API, but didn't go that route as I didn't want to extend a generic API for some SWIOTLB specifics. But if there are more implementations that have a size limitation it makes certainly sense to put it into the DMA-API. I'll change that in the next version.
Thanks,
Joerg
Quoted 8 lines · Joerg Roedel
Hi,
there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.
Quoted 5 lines
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
s/allocations/mappings/, right? We don't use the swiotlb buffer for the coherent allocations anymore, and I don't think virtio-blk uses them anyway.
Quoted 4 lines
This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.
I really don't like the fact that we hardcode swiotlb specific. This needs to be indirect through the dma ops or struct device, as various iommu implementations also have limits (although usually much larger ones).
Quoted 14 lines · Joerg Roedel
Hi,
there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
Quoted 4 lines
This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.
Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?
Thanks
Quoted 18 lines
Please review.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introducevirtio_max_dma_size()
virtio-blk: Considervirtio_max_dma_size()for maximum segment sizedrivers/block/virtio_blk.c | 10 ++++++---- drivers/virtio/virtio_ring.c | 11 +++++++++++ include/linux/swiotlb.h | 12 ++++++++++++ include/linux/virtio.h | 2 ++ 4 files changed, 31 insertions(+), 4 deletions(-)
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?
Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Joerg
Quoted 10 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Quoted 9 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 6 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
Michael S. Tsirkin
maintainerQuoted 22 lines · Jason Wang
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
Signature · 1 line
MST
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 5 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 9 lines
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
In case of SEV, emulated DMA is performed through the SWIOTLB (which bounces the encrypted buffers). The issue reported here will happen on any platform which is making use of SWIOTLB. We could easily reproduce the the virtio-blk failure if we configure swiotlb=force in non SEV guest. Unfortunately in case of SEV the SWIOTLB is must. As Jorge highlighted the main issue is limitation of the SWIOTLB, it does not support allocation/map larger than 256Kb.
Quoted 11 lines
In fact whoever sets
IOMMU_PLATFORMis completely unaffected by Christoph's pet peeve.Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
Robin Murphy
reviewerQuoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 5 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 8 lines
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
Robin.
Quoted 10 lines · Robin Murphy
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in
drivers/virtiois callingdma_set_max_seg_size()or otherwise assigning anydma_parmsto replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
drivers/block/virtio_blk.c:virtblk_probe():
/* Host can optionally specify maximum segment size and number of
* segments. */
err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
struct virtio_blk_config, size_max, &v);
if (!err)
blk_queue_max_segment_size(q, v);
else
blk_queue_max_segment_size(q, -1U);So it really is virtio_blk that is special here. The host could set VIRTIO_BLK_F_SIZE_MAX to paper over the problem, but I really think we need a dma_max_segment_size API that is wired up through the dma_map_ops to sort this out for real.
Quoted 10 lines · Robin Murphy
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in
drivers/virtiois callingdma_set_max_seg_size()or otherwise assigning anydma_parmsto replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
drivers/block/virtio_blk.c:virtblk_probe():
/* Host can optionally specify maximum segment size and number of
* segments. */
err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
struct virtio_blk_config, size_max, &v);
if (!err)
blk_queue_max_segment_size(q, v);
else
blk_queue_max_segment_size(q, -1U);So it really is virtio_blk that is special here. The host could set VIRTIO_BLK_F_SIZE_MAX to paper over the problem, but I really think we need a dma_max_segment_size API that is wired up through the dma_map_ops to sort this out for real.
Michael S. Tsirkin
maintainerQuoted 33 lines · Robin Murphy
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 5 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 8 lines
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
Quoted 5 lines
Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.
It seems reasonable to be able to ask it what it's capabilities are though.
Quoted 11 lines
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in
drivers/virtiois callingdma_set_max_seg_size()or otherwise assigning anydma_parmsto replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
Quoted 1 line
Robin.
I guess this is what you are looking for:
/* Host can optionally specify maximum segment size and number of
* segments. */
err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
struct virtio_blk_config, size_max, &v);
if (!err)
blk_queue_max_segment_size(q, v);
else
blk_queue_max_segment_size(q, -1U);virtio isn't the only caller with a value >64K:
$ git grep -A1 blk_queue_max_segment_size
Documentation/block/biodoc.txt: blk_queue_max_segment_size(q, max_seg_size)
Documentation/block/biodoc.txt- Maximum size of a clustered segment, 64kB default.
--
block/blk-settings.c: * blk_queue_max_segment_size - set max segment size for blk_rq_map_sg
block/blk-settings.c- * @q: the request queue for the device
--
block/blk-settings.c:void blk_queue_max_segment_size(struct request_queue *q, unsigned int max_size)
block/blk-settings.c-{
--
block/blk-settings.c:EXPORT_SYMBOL(blk_queue_max_segment_size);
block/blk-settings.c-
--
drivers/block/mtip32xx/mtip32xx.c: blk_queue_max_segment_size(dd->queue, 0x400000);
drivers/block/mtip32xx/mtip32xx.c- blk_queue_io_min(dd->queue, 4096);
--
drivers/block/nbd.c: blk_queue_max_segment_size(disk->queue, UINT_MAX);
drivers/block/nbd.c- blk_queue_max_segments(disk->queue, USHRT_MAX);
--
drivers/block/ps3disk.c: blk_queue_max_segment_size(queue, dev->bounce_size);
drivers/block/ps3disk.c-
--
drivers/block/ps3vram.c: blk_queue_max_segment_size(queue, BLK_MAX_SEGMENT_SIZE);
drivers/block/ps3vram.c- blk_queue_max_hw_sectors(queue, BLK_SAFE_MAX_SECTORS);
--
drivers/block/rbd.c: blk_queue_max_segment_size(q, UINT_MAX);
drivers/block/rbd.c- blk_queue_io_min(q, objset_bytes);
--
drivers/block/sunvdc.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/block/sunvdc.c-
--
drivers/block/virtio_blk.c: blk_queue_max_segment_size(q, v);
drivers/block/virtio_blk.c- else
drivers/block/virtio_blk.c: blk_queue_max_segment_size(q, -1U);
drivers/block/virtio_blk.c-
--
drivers/block/xen-blkfront.c: blk_queue_max_segment_size(rq, PAGE_SIZE);
drivers/block/xen-blkfront.c-
--
drivers/cdrom/gdrom.c: blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);
drivers/cdrom/gdrom.c- gd.disk->queue = gd.gdrom_rq;
--
drivers/memstick/core/ms_block.c: blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/ms_block.c- MS_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/memstick/core/mspro_block.c: blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/mspro_block.c- MSPRO_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/mmc/core/queue.c: blk_queue_max_segment_size(mq->queue, host->max_seg_size);
drivers/mmc/core/queue.c-
--
drivers/s390/block/dasd.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/s390/block/dasd.c- blk_queue_segment_boundary(q, PAGE_SIZE - 1);
--
drivers/scsi/be2iscsi/be_main.c: blk_queue_max_segment_size(sdev->request_queue, 65536);
drivers/scsi/be2iscsi/be_main.c- return 0;
--
drivers/scsi/scsi_debug.c: blk_queue_max_segment_size(sdp->request_queue, -1U);
drivers/scsi/scsi_debug.c- if (sdebug_no_uld)
--
drivers/scsi/scsi_lib.c: blk_queue_max_segment_size(q, dma_get_max_seg_size(dev));
drivers/scsi/scsi_lib.c-
--
drivers/scsi/ufs/ufshcd.c: blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
drivers/scsi/ufs/ufshcd.c-
--
include/linux/blkdev.h:extern void blk_queue_max_segment_size(struct request_queue *, unsigned int);
include/linux/blkdev.h-extern void blk_queue_max_discard_sectors(struct request_queue *q,
--
include/linux/mmc/host.h: unsigned int max_seg_size; /* see blk_queue_max_segment_size */
include/linux/mmc/host.h- unsigned short max_segs; /* see blk_queue_max_segments */Some of these devices are probably not going to work well if passed through to a SEV guest.
Going back to virtio, at some level virtio is like a stacking device so it does not necessarily need a limit.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerQuoted 33 lines · Robin Murphy
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 5 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 8 lines
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
Quoted 5 lines
Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.
It seems reasonable to be able to ask it what it's capabilities are though.
Quoted 11 lines
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in
drivers/virtiois callingdma_set_max_seg_size()or otherwise assigning anydma_parmsto replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
Quoted 1 line
Robin.
I guess this is what you are looking for:
/* Host can optionally specify maximum segment size and number of
* segments. */
err = virtio_cread_feature(vdev, VIRTIO_BLK_F_SIZE_MAX,
struct virtio_blk_config, size_max, &v);
if (!err)
blk_queue_max_segment_size(q, v);
else
blk_queue_max_segment_size(q, -1U);virtio isn't the only caller with a value >64K:
$ git grep -A1 blk_queue_max_segment_size
Documentation/block/biodoc.txt: blk_queue_max_segment_size(q, max_seg_size)
Documentation/block/biodoc.txt- Maximum size of a clustered segment, 64kB default.
--
block/blk-settings.c: * blk_queue_max_segment_size - set max segment size for blk_rq_map_sg
block/blk-settings.c- * @q: the request queue for the device
--
block/blk-settings.c:void blk_queue_max_segment_size(struct request_queue *q, unsigned int max_size)
block/blk-settings.c-{
--
block/blk-settings.c:EXPORT_SYMBOL(blk_queue_max_segment_size);
block/blk-settings.c-
--
drivers/block/mtip32xx/mtip32xx.c: blk_queue_max_segment_size(dd->queue, 0x400000);
drivers/block/mtip32xx/mtip32xx.c- blk_queue_io_min(dd->queue, 4096);
--
drivers/block/nbd.c: blk_queue_max_segment_size(disk->queue, UINT_MAX);
drivers/block/nbd.c- blk_queue_max_segments(disk->queue, USHRT_MAX);
--
drivers/block/ps3disk.c: blk_queue_max_segment_size(queue, dev->bounce_size);
drivers/block/ps3disk.c-
--
drivers/block/ps3vram.c: blk_queue_max_segment_size(queue, BLK_MAX_SEGMENT_SIZE);
drivers/block/ps3vram.c- blk_queue_max_hw_sectors(queue, BLK_SAFE_MAX_SECTORS);
--
drivers/block/rbd.c: blk_queue_max_segment_size(q, UINT_MAX);
drivers/block/rbd.c- blk_queue_io_min(q, objset_bytes);
--
drivers/block/sunvdc.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/block/sunvdc.c-
--
drivers/block/virtio_blk.c: blk_queue_max_segment_size(q, v);
drivers/block/virtio_blk.c- else
drivers/block/virtio_blk.c: blk_queue_max_segment_size(q, -1U);
drivers/block/virtio_blk.c-
--
drivers/block/xen-blkfront.c: blk_queue_max_segment_size(rq, PAGE_SIZE);
drivers/block/xen-blkfront.c-
--
drivers/cdrom/gdrom.c: blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);
drivers/cdrom/gdrom.c- gd.disk->queue = gd.gdrom_rq;
--
drivers/memstick/core/ms_block.c: blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/ms_block.c- MS_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/memstick/core/mspro_block.c: blk_queue_max_segment_size(msb->queue,
drivers/memstick/core/mspro_block.c- MSPRO_BLOCK_MAX_PAGES * msb->page_size);
--
drivers/mmc/core/queue.c: blk_queue_max_segment_size(mq->queue, host->max_seg_size);
drivers/mmc/core/queue.c-
--
drivers/s390/block/dasd.c: blk_queue_max_segment_size(q, PAGE_SIZE);
drivers/s390/block/dasd.c- blk_queue_segment_boundary(q, PAGE_SIZE - 1);
--
drivers/scsi/be2iscsi/be_main.c: blk_queue_max_segment_size(sdev->request_queue, 65536);
drivers/scsi/be2iscsi/be_main.c- return 0;
--
drivers/scsi/scsi_debug.c: blk_queue_max_segment_size(sdp->request_queue, -1U);
drivers/scsi/scsi_debug.c- if (sdebug_no_uld)
--
drivers/scsi/scsi_lib.c: blk_queue_max_segment_size(q, dma_get_max_seg_size(dev));
drivers/scsi/scsi_lib.c-
--
drivers/scsi/ufs/ufshcd.c: blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
drivers/scsi/ufs/ufshcd.c-
--
include/linux/blkdev.h:extern void blk_queue_max_segment_size(struct request_queue *, unsigned int);
include/linux/blkdev.h-extern void blk_queue_max_discard_sectors(struct request_queue *q,
--
include/linux/mmc/host.h: unsigned int max_seg_size; /* see blk_queue_max_segment_size */
include/linux/mmc/host.h- unsigned short max_segs; /* see blk_queue_max_segments */Some of these devices are probably not going to work well if passed through to a SEV guest.
Going back to virtio, at some level virtio is like a stacking device so it does not necessarily need a limit.
Signature · 5 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Robin Murphy
reviewerQuoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 18 lines · Christoph Hellwig
Quoted 8 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 5 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 8 lines
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
Huh? Regardless of which virtio devices use it or not, the DMA API is handling the SEV case as correctly as it possibly can, by forcing everything through the unencrypted bounce buffer. If the segments being mapped are too big for that bounce buffer in the first place, there's nothing it can possibly do except fail, gracefully or otherwise.
Now, in theory, yes, the real issue at hand is not unique to virtio-blk nor SEV - any driver whose device has a sufficiently large DMA segment size and who manages to get sufficient physically-contiguous memory could technically generate a scatterlist segment longer than SWIOTLB can handle. However, in practice that basically never happens, not least because very few drivers ever override the default 64K DMA segment limit. AFAICS nothing in drivers/virtio is calling dma_set_max_seg_size() or otherwise assigning any dma_parms to replace the defaults either, so the really interesting question here is how are these apparently-out-of-spec 256K segments getting generated at all?
Robin.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 3 lines · Michael S. Tsirkin
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
The DMA API handles the SEV case perfectly. Its just that virtio_blk supports huge segments that virtio does not generally support, but that is not related to SEV in any way.
Quoted 2 lines
In fact whoever sets
IOMMU_PLATFORMis completely unaffected by Christoph's pet peeve.
No, the above happens only when we set IOMMU_PLATFORM.
Quoted 7 lines
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
No. The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.
As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.
Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.
Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.
Michael S. Tsirkin
maintainerQuoted 9 lines · Christoph Hellwig
Quoted 7 lines
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
No.
Oh ok then.
Quoted 9 lines
The problem is that we still haven't fully specified what
IOMMU_PLATFORMand !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.As a start
VIRTIO_F_ORDER_PLATFORMandVIRTIO_F_ACCESS_PLATFORMabsolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.
I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.
So I would make that a SHOULD probably.
Quoted 3 lines
Second software implementations SHOULD set
VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.
Yes. If host does not have access to all of memory it SHOULD set VIRTIO_F_ACCESS_PLATFORM.
It seems rather straight-forward to add conformance clauses for this, thanks for reminding me.
Quoted 2 lines
Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.
Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.
If you want to list the most problematic cases, e.g. in a github issue, we might get to clarifying them sooner.
Thanks!
Signature · 1 line
MST
Quoted 6 lines · Michael S. Tsirkin
I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.
So I would make that a SHOULD probably.
The problem is that without using DMA ops you can't just plug the device into a random system, which is a total no-go for periphals.
And cache flushing is not that uncommmon, e.g. every sparc systems needs it, many arm/arm64 ones, some power ones, lots of mips ones, etc.
But cache flushing is just one side of it - lots of systems have either mandatory or option IOMMUs, and without the platform access flag that doesn't work. As does that case where you have a DMA offset, which is extremly common on arm and power systems. So you might find a few non-x86 systems you can plug it in and it'll just work, but it won't be all that many. And even on x86 your device will be completely broken if the users dares to use an IOMMU.
Quoted 6 lines
Quoted 1 line
flags really needs some major updates in terms of DMA access.
Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.
At least say bus address - as said above the CPU and bus concepts of physicall addresses are different in many systems. Including x86 when you use IOMMUs.
Quoted 6 lines · Michael S. Tsirkin
I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.
So I would make that a SHOULD probably.
The problem is that without using DMA ops you can't just plug the device into a random system, which is a total no-go for periphals.
And cache flushing is not that uncommmon, e.g. every sparc systems needs it, many arm/arm64 ones, some power ones, lots of mips ones, etc.
But cache flushing is just one side of it - lots of systems have either mandatory or option IOMMUs, and without the platform access flag that doesn't work. As does that case where you have a DMA offset, which is extremly common on arm and power systems. So you might find a few non-x86 systems you can plug it in and it'll just work, but it won't be all that many. And even on x86 your device will be completely broken if the users dares to use an IOMMU.
Quoted 6 lines
Quoted 1 line
flags really needs some major updates in terms of DMA access.
Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.
At least say bus address - as said above the CPU and bus concepts of physicall addresses are different in many systems. Including x86 when you use IOMMUs.
Michael S. Tsirkin
maintainerQuoted 9 lines · Christoph Hellwig
Quoted 7 lines
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
No.
Oh ok then.
Quoted 9 lines
The problem is that we still haven't fully specified what
IOMMU_PLATFORMand !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.As a start
VIRTIO_F_ORDER_PLATFORMandVIRTIO_F_ACCESS_PLATFORMabsolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.
I think you are exaggerating a little here. Limited access with e.g. need to flush caches is common on lower end but less common on higher end systems used for virtualization. As you point out that changes with e.g. SEV but then SEV is a pretty niche thing so far.
So I would make that a SHOULD probably.
Quoted 3 lines
Second software implementations SHOULD set
VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.
Yes. If host does not have access to all of memory it SHOULD set VIRTIO_F_ACCESS_PLATFORM.
It seems rather straight-forward to add conformance clauses for this, thanks for reminding me.
Quoted 2 lines
Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.
Thanks for the reminder. Yes generally spec tends to still say e.g. "physical address" where it really means a "DMA address" in Linux terms. Whether changing that will make things much clearer I'm not sure. E.g. hardware designers don't much care I guess.
If you want to list the most problematic cases, e.g. in a github issue, we might get to clarifying them sooner.
Thanks!
Signature · 1 line
MST
Quoted 3 lines · Michael S. Tsirkin
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
The DMA API handles the SEV case perfectly. Its just that virtio_blk supports huge segments that virtio does not generally support, but that is not related to SEV in any way.
Quoted 2 lines
In fact whoever sets
IOMMU_PLATFORMis completely unaffected by Christoph's pet peeve.
No, the above happens only when we set IOMMU_PLATFORM.
Quoted 7 lines
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
No. The problem is that we still haven't fully specified what IOMMU_PLATFORM and !IOMMU_PLATFORM actually mean. Your "ACCESS_PLATFORM/ORDER_PLATFORM" commit in the virtio-spec repo improves it a little bit, but it is still far from enough.
As a start VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM absolutely MUST be set for hardware implementations. Otherwise said hardware has no chance of working on anything but the most x86-like systems.
Second software implementations SHOULD set VIRTIO_F_ACCESS_PLATFORM, because otherwise we can't add proper handling for things like SEV or the IBM "secure hypervisor" thing.
Last but not least a lot of wording outside the area describing these flags really needs some major updates in terms of DMA access.
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.
Regards,
Joerg
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset
dma_opsthe DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.
True. But the NULL-ops dma direct case still has two issues that might not work for virtio:
(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
dev->dma_coherent = true;#endif
except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field
(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.
So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..
Quoted 4 lines
Regards,
Joerg
---end quoted text---
Michael S. Tsirkin
maintainerQuoted 37 lines · Christoph Hellwig
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset
dma_opsthe DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.True. But the NULL-ops dma direct case still has two issues that might not work for virtio:
(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) dev->dma_coherent = true;#endif
except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field
(b) we can still mangle the DMA address, either using the
dma_pfn_offsetfield in struct device, or by a full override of__phys_to_dma/__dma_to_physby the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.
Quoted 3 lines
So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..
IIUC some emulated hardware has no reasonable way to specify that some devices bypass the IOMMU, and no cheap way to cover all memory with an IOMMU mapping.
So I suspect !ACCESS_PLATFORM will stay a supported configuration forever :(
Quoted 5 lines
Quoted 4 lines
Regards,
Joerg
---end quoted text---
Michael S. Tsirkin
maintainerQuoted 37 lines · Christoph Hellwig
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset
dma_opsthe DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.True. But the NULL-ops dma direct case still has two issues that might not work for virtio:
(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) dev->dma_coherent = true;#endif
except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field
(b) we can still mangle the DMA address, either using the
dma_pfn_offsetfield in struct device, or by a full override of__phys_to_dma/__dma_to_physby the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.
Quoted 3 lines
So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..
IIUC some emulated hardware has no reasonable way to specify that some devices bypass the IOMMU, and no cheap way to cover all memory with an IOMMU mapping.
So I suspect !ACCESS_PLATFORM will stay a supported configuration forever :(
Quoted 5 lines
Quoted 4 lines
Regards,
Joerg
---end quoted text---
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset
dma_opsthe DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.
True. But the NULL-ops dma direct case still has two issues that might not work for virtio:
(a) if the architecture supports devices that are not DMA coherent it will dip into a special allocator and do cache maintainance for streaming mappings. Although it would be a bit of a hack we could work around this in virtio doings something like:
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
dev->dma_coherent = true;#endif
except that won't work for mips, which has a mode where it does a system instead of device level coherency flag and thus doesn't look at this struct device field
(b) we can still mangle the DMA address, either using the dma_pfn_offset field in struct device, or by a full override of __phys_to_dma / __dma_to_phys by the architecture code. The first could be solved using a hack like the one above, but the latter would be a little harder. In the long run I'd love to get rid of that hook and have everyone use the generic offset code, but for that we first need to support multiple ranges with different offset and do quite some nasty arch code surgery.
So for the legacy virtio case I fear we need to keep local dma mapping implementation for now. I just wish now recent hypervisor would ever offer devices in this broken legacy mode..
Quoted 4 lines
Regards,
Joerg
---end quoted text---
Quoted 2 lines · Michael S. Tsirkin
Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance.
Which indirect calls? In case of unset dma_ops the DMA-API functions call directly into the dma-direct implementation, no indirect calls at all.
Regards,
Joerg
Michael S. Tsirkin
maintainerQuoted 22 lines · Jason Wang
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Quoted 4 lines
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
I don't think so - the issue is really that DMA API does not yet handle the SEV case 100% correctly. I suspect passthrough devices would have the same issue.
In fact whoever sets IOMMU_PLATFORM is completely unaffected by Christoph's pet peeve.
Christoph is saying that !IOMMU_PLATFORM devices should hide the compatibility code in a special per-device DMA API implementation. Which would be fine especially if we can manage not to introduce a bunch of indirect calls all over the place and hurt performance. It's just that the benefit is unlikely to be big (e.g. we can't also get rid of the virtio specific memory barriers) so no one was motivated enough to work on it.
Signature · 5 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Wang
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
Yes, I fully agree with you. This is probably an exact example of such problem.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 9 lines · Jason Wang
Quoted 7 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Quoted 6 lines
Quoted 3 lines
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
Which brings me back to my pet pevee that we need to take actions that virtio uses the proper dma mapping API by default with quirks for legacy cases. The magic bypass it uses is just causing problems over problems.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 10 lines · Joerg Roedel
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Joerg
Thanks, have you tested vhost-net in this case. I suspect it may not work
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 2 lines · Jason Wang
Just wonder if my understanding is correct
IOMMU_PLATFORMmust be set for all virtio devices under AMD-SEV guests?
Yes, that is correct. Emulated DMA can only happen on the SWIOTLB aperture, because that memory is not encrypted. The guest bounces the data then to its encrypted memory.
Regards,
Joerg
Quoted 14 lines · Joerg Roedel
Hi,
there is a problem with virtio-blk driven devices when virtio-ring uses SWIOTLB through the DMA-API. This happens for example in AMD-SEV enabled guests, where the guest RAM is mostly encrypted and all emulated DMA has to happen to/from the SWIOTLB aperture.
The problem is a limitation of the SWIOTLB implementation, which does not support allocations larger than 256kb. When the virtio-blk driver tries to read/write a block larger than that, the allocation of the dma-handle fails and an IO error is reported.
Quoted 4 lines
This patch-set adds a check to the virtio-code whether it might be using SWIOTLB bounce buffering and limits the maximum segment size in the virtio-blk driver in this case, so that it doesn't try to do larger reads/writes.
Just wonder if my understanding is correct IOMMU_PLATFORM must be set for all virtio devices under AMD-SEV guests?
Thanks
Quoted 12 lines
Please review.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Export maximum allocation size
virtio: Introducevirtio_max_dma_size()
virtio-blk: Considervirtio_max_dma_size()for maximum segment size
Quoted 6 lines
drivers/block/virtio_blk.c | 10 ++++++---- drivers/virtio/virtio_ring.c | 11 +++++++++++ include/linux/swiotlb.h | 12 ++++++++++++ include/linux/virtio.h | 2 ++ 4 files changed, 31 insertions(+), 4 deletions(-)
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization