virtio: support devices that own their virtqueue memory
Subsystems (3)
v2 cover virtio: support devices that own their virtqueue memory
Virtio drivers use guest memory to back virtqueues and their buffers. That means a VMM needs to be able to map guest memory. That is ok in the normal virt case. It gets icky with confidential computing (where we use swiotlb as workaround) and it defeats the purpose of isolated vhost-user backing devices, because they end up with full RAM access to the guest.
So instead, I'm proposing an extension to virtio which allows it to give each virtio device its own dedicated memory region to communicate with the host, called DMB (Device Memory Buffer). A trusted hypervisor can force DMB to be present, which then enables safer, more isolated and resilient communication between guest and host.
With DMB, the device provides a shared memory region that both parties agree is the full memory map both have access to. All memory offsets that previously would have been into guest RAM, are then offsets into this shared memory buffer region. One nice property of this is that it is a generic mechanism in the virtio transport layer, so higher level drivers work unmodified.
I was exploring to use swiotlb instead to create individual pools. But that approach has multiple downsides:
1. Swiotlb is an OS primitive which is not available in all Operating Systems. DMB however lives in the virtio transport layer, which means we can add support for it in any OS independent of generic layers. This helps with Windows support.
2. We munge DMA space together. DMB provides a separate DMA space per virtio device. This means we can for example implement a device in vhost-user and give the implementing process only visibility to the DMB region, not all of guest memory. That reduces the exposure the vhost-user provider has, improving security.
3. Devices can opt-in. A hypervisor can choose to use standard virtio semantics for self-implemented devices (e.g. NSM), while requiring DMB for devices implemented by less trustworthy providers. The non-trustworthy devices do not get any visibility into the trustworthy ones, even with DMB in place for both.
I also looked at virtio-iommu. To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.
== Limitations ==
- Only PCI is wired up.
- Feature bit 44 and the two registers at offsets 0x40 and 0x42 of the PCI common configuration are provisional: the OASIS technical committee has the specification and has allocated none of them.
https://lore.kernel.org/virtio-comment/[email protected]/
== Testing ==
I ran this against a device that offers DMB and not VIRTIO_F_ACCESS_PLATFORM, with a 16 MiB region on each device. A kernel without patch 10 finds the region and then hangs with nothing in the log; with it, 512 MiB of O_DIRECT block reads and 137 MiB of loopback network traffic go through the regions with no allocation failure.
Patches 1 to 4 do not need the rest of the series. They remove an API no driver calls and a stale worked example in a vdpa comment. Patch 3 and patch 4 carry Fixes:; patch 4's are older than this series, a failed mapping arriving from a packed ring as -EIO, which fails an I/O that on a split ring is only back-pressure.
I wrote this series with an AI coding assistant, which drafted the code, the changelogs and this cover letter. I reviewed and reworked all of it, and every commit carries an Assisted-by: trailer.
Alex
v1: https://lore.kernel.org/all/[email protected]/
v1 -> v2:
- Remove the unused map sync API (Michael)
- Drop the sync operations from virtio_map_ops (Michael)
- Drop the stale mask expansion instead of correcting it (Michael)
- Treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
- Read the region's memory type and accept only a coherent one
- Rewrite the region allocator over gen_pool
- Return -ENOMEM instead of -EIO from a failed packed ring mapping
- Drop the patch validating premapped addresses through the map
- Drop the patch reporting a bounded pool's exhaustion as -ENOSPC
- Drop the range withheld for a virtqueue's first descriptor chain
- Drop the .rst and describe the region in the headers instead
Alexander Graf (12):
virtio_ring: remove the unused map sync API
virtio: drop the sync operations from virtio_map_ops
vdpa: drop the VIRTIO_DEVICE_F_MASK example value
virtio_ring: return -ENOMEM when a packed ring mapping fails
virtio: add the VIRTIO_F_DMB feature bit
virtio_pci: read the device memory buffer registers
virtio_pci: create virtqueues with the device's mapping token
virtio: add a device memory buffer region allocator
virtio: locate the device memory buffer after feature negotiation
virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
virtio_pci: support VIRTIO_F_DMB
virtio: expose device memory buffer occupancy over debugfs
drivers/vdpa/vdpa.c | 7 +- drivers/vdpa/vdpa_user/iova_domain.c | 20 - drivers/vdpa/vdpa_user/iova_domain.h | 8 - drivers/vdpa/vdpa_user/vduse_dev.c | 40 -- drivers/virtio/Kconfig | 17 + drivers/virtio/Makefile | 3 +- drivers/virtio/virtio.c | 25 +- drivers/virtio/virtio_dmb.c | 959 +++++++++++++++++++++++++ drivers/virtio/virtio_dmb.h | 28 + drivers/virtio/virtio_pci_modern.c | 75 +- drivers/virtio/virtio_pci_modern_dev.c | 47 +- drivers/virtio/virtio_ring.c | 93 +-- include/linux/virtio.h | 11 +- include/linux/virtio_config.h | 41 +- include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_config.h | 24 +- include/uapi/linux/virtio_pci.h | 19 + tools/virtio/linux/dma-mapping.h | 7 - 18 files changed, 1217 insertions(+), 209 deletions(-) create mode 100644 drivers/virtio/virtio_dmb.c create mode 100644 drivers/virtio/virtio_dmb.h
base-commit: fc02acf6ac0ccde0c805c2daa9148683cdd01ba8
PATCH v2 1/12 virtio_ring: remove the unused map sync API
virtqueue_map_need_sync(), virtqueue_map_sync_single_range_for_cpu() and virtqueue_map_sync_single_range_for_device() are exported, but no driver in the tree calls them. They are the only path into the sync members of struct virtio_map_ops, so whoever implements that struct next has to decide what those members do with no caller to learn it from.
Remove the three functions, their prototypes and their exports, and the DMA sync shims that the virtio userspace harness kept for them. With this, virtio_ring no longer exports a way to sync a mapping.
--- drivers/virtio/virtio_ring.c | 89 -------------------------------- include/linux/virtio.h | 8 --- tools/virtio/linux/dma-mapping.h | 7 --- 3 files changed, 104 deletions(-)
drivers/virtio/virtio_ring.c +0 -89
@@ -3890,94 +3890,5 @@ int virtqueue_map_mapping_error(const struct virtqueue *_vq, dma_addr_t addr) }EXPORT_SYMBOL_GPL (virtqueue_map_mapping_error); -/** -* virtqueue_map_need_sync - check a dma address needs sync -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* -* Check if the dma address mapped by the virtqueue_map_* APIs needs to be -* synchronized -* -* return bool -*/ -bool virtqueue_map_need_sync (const struct virtqueue *_vq,dma_addr_t addr) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return false; - -if (vdev->map) -return vdev->map->need_sync (vq->map, addr); -else -return dma_need_sync (vring_dma_dev (vq), addr); -} -EXPORT_SYMBOL_GPL (virtqueue_map_need_sync); - -/** -* virtqueue_map_sync_single_range_for_cpu - map sync for cpu -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* @offset: DMA address offset -* @size: buf size for sync -* @dir: DMA direction -* -* Before calling this function, use virtqueue_map_need_sync() to confirm that -* the DMA address really needs to be synchronized -* -*/ -void virtqueue_map_sync_single_range_for_cpu (const struct virtqueue *_vq, -dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return ; - -if (vdev->map) - vdev->map->sync_single_for_cpu (vq->map, - addr + offset, size, dir); -else -dma_sync_single_range_for_cpu (vring_dma_dev (vq), - addr, offset, size, dir); -} -EXPORT_SYMBOL_GPL (virtqueue_map_sync_single_range_for_cpu); - -/** -* virtqueue_map_sync_single_range_for_device - map sync for device -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* @offset: DMA address offset -* @size: buf size for sync -* @dir: DMA direction -* -* Before calling this function, use virtqueue_map_need_sync() to confirm that -* the DMA address really needs to be synchronized -*/ -void virtqueue_map_sync_single_range_for_device (const struct virtqueue *_vq, -dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return ; - -if (vdev->map) - vdev->map->sync_single_for_device (vq->map, - addr + offset, - size, dir); -else -dma_sync_single_range_for_device (vring_dma_dev (vq), addr, - offset, size, dir); -} -EXPORT_SYMBOL_GPL (virtqueue_map_sync_single_range_for_device); -MODULE_DESCRIPTION ("Virtio ring implementation" );MODULE_LICENSE ("GPL" );
include/linux/virtio.h +0 -8
@@ -312,14 +312,6 @@ void virtqueue_unmap_single_attrs(const struct virtqueue *_vq, dma_addr_t addr, unsigned long attrs);int virtqueue_map_mapping_error (const struct virtqueue *_vq,dma_addr_t addr); -bool virtqueue_map_need_sync (const struct virtqueue *_vq,dma_addr_t addr); -void virtqueue_map_sync_single_range_for_cpu (const struct virtqueue *_vq,dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir); -void virtqueue_map_sync_single_range_for_device (const struct virtqueue *_vq,dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir); -#ifdef CONFIG_VIRTIO_DEBUG void virtio_debug_device_init (struct virtio_device *dev);void virtio_debug_device_exit (struct virtio_device *dev);
tools/virtio/linux/dma-mapping.h +0 -7
@@ -36,16 +36,9 @@ enum dma_data_direction {#define sg_dma_address (sg) (0)#define sg_dma_len (sg) (0) -#define dma_need_sync (v, a) (0)#define dma_unmap_single_attrs (d, a, s, r, t) do { \ (void)(d); (void)(a); (void)(s); (void)(r); (void)(t); \ } while (0) -#define dma_sync_single_range_for_cpu (d, a, o, s, r) do { \ - (void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \ -} while (0) -#define dma_sync_single_range_for_device (d, a, o, s, r) do { \ - (void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \ -} while (0)#define dma_max_mapping_size (...) SIZE_MAX/*
Re: [PATCH v2 01/12] virtio_ring: remove the unused map sync API
Quoted 7 lines · Alexander Graf
virtqueue_map_need_sync(),virtqueue_map_sync_single_range_for_cpu()andvirtqueue_map_sync_single_range_for_device()are exported, but no driver in the tree calls them. They are the only path into the sync members of structvirtio_map_ops, so whoever implements that struct next has to decide what those members do with no caller to learn it from.
Quoted 5 lines
Remove the three functions, their prototypes and their exports, and the DMA sync shims that the virtio userspace harness kept for them. With this,
virtio_ringno longer exports a way to sync a mapping.Suggested-by Michael S. Tsirkin <[email protected]>
Thanks!
Quoted 148 lines
Assisted-by Kiro:claude-opus-5 checkpatchSigned-off-by Alexander Graf <[email protected]>--- drivers/virtio/virtio_ring.c | 89 -------------------------------- include/linux/virtio.h | 8 --- tools/virtio/linux/dma-mapping.h | 7 --- 3 files changed, 104 deletions(-)drivers/virtio/virtio_ring.c +0
-89@@ -3890,94 +3890,5 @@ int virtqueue_map_mapping_error(const struct virtqueue *_vq, dma_addr_t addr) }EXPORT_SYMBOL_GPL (virtqueue_map_mapping_error); -/** -* virtqueue_map_need_sync - check a dma address needs sync -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* -* Check if the dma address mapped by the virtqueue_map_* APIs needs to be -* synchronized -* -* return bool -*/ -bool virtqueue_map_need_sync (const struct virtqueue *_vq,dma_addr_t addr) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return false; - -if (vdev->map) -return vdev->map->need_sync (vq->map, addr); -else -return dma_need_sync (vring_dma_dev (vq), addr); -} -EXPORT_SYMBOL_GPL (virtqueue_map_need_sync); - -/** -* virtqueue_map_sync_single_range_for_cpu - map sync for cpu -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* @offset: DMA address offset -* @size: buf size for sync -* @dir: DMA direction -* -* Before calling this function, use virtqueue_map_need_sync() to confirm that -* the DMA address really needs to be synchronized -* -*/ -void virtqueue_map_sync_single_range_for_cpu (const struct virtqueue *_vq, -dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return ; - -if (vdev->map) - vdev->map->sync_single_for_cpu (vq->map, - addr + offset, size, dir); -else -dma_sync_single_range_for_cpu (vring_dma_dev (vq), - addr, offset, size, dir); -} -EXPORT_SYMBOL_GPL (virtqueue_map_sync_single_range_for_cpu); - -/** -* virtqueue_map_sync_single_range_for_device - map sync for device -* @_vq: the struct virtqueue we're talking about. -* @addr: DMA address -* @offset: DMA address offset -* @size: buf size for sync -* @dir: DMA direction -* -* Before calling this function, use virtqueue_map_need_sync() to confirm that -* the DMA address really needs to be synchronized -*/ -void virtqueue_map_sync_single_range_for_device (const struct virtqueue *_vq, -dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir) -{ -const struct vring_virtqueue *vq =to_vvq (_vq); -struct virtio_device *vdev = _vq->vdev; - -if (!vq->use_map_api) -return ; - -if (vdev->map) - vdev->map->sync_single_for_device (vq->map, - addr + offset, - size, dir); -else -dma_sync_single_range_for_device (vring_dma_dev (vq), addr, - offset, size, dir); -} -EXPORT_SYMBOL_GPL (virtqueue_map_sync_single_range_for_device); -MODULE_DESCRIPTION ("Virtio ring implementation" );MODULE_LICENSE ("GPL" );include/linux/virtio.h +0
-8@@ -312,14 +312,6 @@ void virtqueue_unmap_single_attrs(const struct virtqueue *_vq, dma_addr_t addr, unsigned long attrs);int virtqueue_map_mapping_error (const struct virtqueue *_vq,dma_addr_t addr); -bool virtqueue_map_need_sync (const struct virtqueue *_vq,dma_addr_t addr); -void virtqueue_map_sync_single_range_for_cpu (const struct virtqueue *_vq,dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir); -void virtqueue_map_sync_single_range_for_device (const struct virtqueue *_vq,dma_addr_t addr, -unsigned long offset,size_t size, -enum dma_data_direction dir); -#ifdef CONFIG_VIRTIO_DEBUG void virtio_debug_device_init (struct virtio_device *dev);void virtio_debug_device_exit (struct virtio_device *dev);tools/virtio/linux/dma-mapping.h +0
-7@@ -36,16 +36,9 @@ enum dma_data_direction {#define sg_dma_address (sg) (0)#define sg_dma_len (sg) (0) -#define dma_need_sync (v, a) (0)#define dma_unmap_single_attrs (d, a, s, r, t) do { \ (void)(d); (void)(a); (void)(s); (void)(r); (void)(t); \ } while (0) -#define dma_sync_single_range_for_cpu (d, a, o, s, r) do { \ - (void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \ -} while (0) -#define dma_sync_single_range_for_device (d, a, o, s, r) do { \ - (void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \ -} while (0)#define dma_max_mapping_size (...) SIZE_MAX/*
PATCH v2 2/12 virtio: drop the sync operations from virtio_map_ops
In reply to Alexander GrafWith the map sync API gone, sync_single_for_cpu, sync_single_for_device and need_sync in struct virtio_map_ops are members the virtqueue never calls. vduse is the only implementation in the tree and provides all three, and the next implementer has to decide what to put in them.
Drop the three members and their kernel-doc, and remove the vduse functions that provided them. vduse_domain_bounce() stays, because vduse_domain_map_page() and vduse_domain_unmap_page() bounce through it as well. That way the struct asks an implementation for six operations instead of nine.
--- drivers/vdpa/vdpa_user/iova_domain.c | 20 -------------- drivers/vdpa/vdpa_user/iova_domain.h | 8 ------ drivers/vdpa/vdpa_user/vduse_dev.c | 40 ---------------------------- include/linux/virtio_config.h | 20 -------------- 4 files changed, 88 deletions(-)
drivers/vdpa/vdpa_user/iova_domain.c +0 -20
@@ -424,26 +424,6 @@ static void vduse_domain_free_iova(struct iova_domain *iovad,free_iova_fast (iovad, iova >> shift, iova_len); } -void vduse_domain_sync_single_for_device (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -read_lock (&domain->bounce_lock); -if (dir ==DMA_TO_DEVICE || dir ==DMA_BIDIRECTIONAL ) -vduse_domain_bounce (domain, dma_addr, size,DMA_TO_DEVICE ); -read_unlock (&domain->bounce_lock); -} - -void vduse_domain_sync_single_for_cpu (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -read_lock (&domain->bounce_lock); -if (dir ==DMA_FROM_DEVICE || dir ==DMA_BIDIRECTIONAL ) -vduse_domain_bounce (domain, dma_addr, size,DMA_FROM_DEVICE ); -read_unlock (&domain->bounce_lock); -} - dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,
drivers/vdpa/vdpa_user/iova_domain.h +0 -8
@@ -48,14 +48,6 @@ int vduse_domain_set_map(struct vduse_iova_domain *domain,void vduse_domain_clear_map (struct vduse_iova_domain *domain,struct vhost_iotlb *iotlb); -void vduse_domain_sync_single_for_device (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir); - -void vduse_domain_sync_single_for_cpu (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir); - dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,
drivers/vdpa/vdpa_user/vduse_dev.c +0 -40
@@ -932,34 +932,6 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .free = vduse_vdpa_free, }; -static void vduse_dev_sync_single_for_device (union virtio_map token, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -struct vduse_iova_domain *domain; - -if (!token.group) -return ; - -guard (vq_group_as_read_lock)(token.group); - domain = token.group->as->domain; -vduse_domain_sync_single_for_device (domain, dma_addr, size, dir); -} - -static void vduse_dev_sync_single_for_cpu (union virtio_map token, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -struct vduse_iova_domain *domain; - -if (!token.group) -return ; - -guard (vq_group_as_read_lock)(token.group); - domain = token.group->as->domain; -vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); -} - static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,
@@ -1037,15 +1009,6 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,free_pages_exact (vaddr, size); } -static bool vduse_dev_need_sync (union virtio_map token,dma_addr_t dma_addr) -{ -if (!token.group) -return false; - -guard (vq_group_as_read_lock)(token.group); -return dma_addr < token.group->as->domain->bounce_size; -} -static int vduse_dev_mapping_error (union virtio_map token,dma_addr_t dma_addr) {if (unlikely (dma_addr ==DMA_MAPPING_ERROR ))
@@ -1063,13 +1026,10 @@ static size_t vduse_dev_max_mapping_size(union virtio_map token) }static const struct virtio_map_ops vduse_map_ops = { - .sync_single_for_device = vduse_dev_sync_single_for_device, - .sync_single_for_cpu = vduse_dev_sync_single_for_cpu, .map_page = vduse_dev_map_page, .unmap_page = vduse_dev_unmap_page, .alloc = vduse_dev_alloc_coherent, .free = vduse_dev_free_coherent, - .need_sync = vduse_dev_need_sync, .mapping_error = vduse_dev_mapping_error, .max_mapping_size = vduse_dev_max_mapping_size, };
include/linux/virtio_config.h +0 -20
@@ -157,16 +157,6 @@ struct virtio_config_ops {* size: the buffer size * dir: mapping direction * attrs: unmapping attributes -* @sync_single_for_cpu: sync a single buffer from device to cpu -* map: metadata for performing mapping -* map_handle: the mapping address to sync -* size: the size of the buffer -* dir: synchronization direction -* @sync_single_for_device: sync a single buffer from cpu to device -* map: metadata for performing mapping -* map_handle: the mapping address to sync -* size: the size of the buffer -* dir: synchronization direction * @alloc: alloc a coherent buffer mapping * map: metadata for performing mapping * size: the size of the buffer
@@ -179,10 +169,6 @@ struct virtio_config_ops {* vaddr: virtual address of the buffer * map_handle: the mapping address that needs to be freed * attrs: unmapping attributes -* @need_sync: if the buffer needs synchronization -* map: metadata for performing mapping -* map_handle: the mapped address -* Returns whether the buffer needs synchronization * @mapping_error: if the mapping address is error * map: metadata for performing mapping * map_handle: the mapped address
@@ -197,16 +183,10 @@ struct virtio_map_ops {void (*unmap_page)(union virtio_map map,dma_addr_t map_handle,size_t size,enum dma_data_direction dir,unsigned long attrs); -void (*sync_single_for_cpu)(union virtio_map map,dma_addr_t map_handle, -size_t size,enum dma_data_direction dir); -void (*sync_single_for_device)(union virtio_map map, -dma_addr_t map_handle,size_t size, -enum dma_data_direction dir);void *(*alloc)(union virtio_map map,size_t size,dma_addr_t *map_handle,gfp_t gfp);void (*free)(union virtio_map map,size_t size,void *vaddr,dma_addr_t map_handle,unsigned long attrs); -bool (*need_sync)(union virtio_map map,dma_addr_t map_handle);int (*mapping_error)(union virtio_map map,dma_addr_t map_handle);size_t (*max_mapping_size)(union virtio_map map); };
Re: [PATCH v2 02/12] virtio: drop the sync operations from virtio_map_ops
Quoted 6 lines · Alexander Graf
With the map sync API gone,
sync_single_for_cpu,sync_single_for_deviceandneed_syncin structvirtio_map_opsare members the virtqueue never calls. vduse is the only implementation in the tree and provides all three, and the next implementer has to decide what to put in them.
Quoted 7 lines
Drop the three members and their kernel-doc, and remove the vduse functions that provided them.
vduse_domain_bounce()stays, becausevduse_domain_map_page()andvduse_domain_unmap_page()bounce through it as well. That way the struct asks an implementation for six operations instead of nine.Suggested-by Michael S. Tsirkin <[email protected]>
Quoted 178 lines
Assisted-by Kiro:claude-opus-5 checkpatchSigned-off-by Alexander Graf <[email protected]>--- drivers/vdpa/vdpa_user/iova_domain.c | 20 -------------- drivers/vdpa/vdpa_user/iova_domain.h | 8 ------ drivers/vdpa/vdpa_user/vduse_dev.c | 40 ---------------------------- include/linux/virtio_config.h | 20 -------------- 4 files changed, 88 deletions(-)drivers/vdpa/vdpa_user/iova_domain.c +0
-20@@ -424,26 +424,6 @@ static void vduse_domain_free_iova(struct iova_domain *iovad,free_iova_fast (iovad, iova >> shift, iova_len); } -void vduse_domain_sync_single_for_device (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -read_lock (&domain->bounce_lock); -if (dir ==DMA_TO_DEVICE || dir ==DMA_BIDIRECTIONAL ) -vduse_domain_bounce (domain, dma_addr, size,DMA_TO_DEVICE ); -read_unlock (&domain->bounce_lock); -} - -void vduse_domain_sync_single_for_cpu (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -read_lock (&domain->bounce_lock); -if (dir ==DMA_FROM_DEVICE || dir ==DMA_BIDIRECTIONAL ) -vduse_domain_bounce (domain, dma_addr, size,DMA_FROM_DEVICE ); -read_unlock (&domain->bounce_lock); -} - dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,drivers/vdpa/vdpa_user/iova_domain.h +0
-8@@ -48,14 +48,6 @@ int vduse_domain_set_map(struct vduse_iova_domain *domain,void vduse_domain_clear_map (struct vduse_iova_domain *domain,struct vhost_iotlb *iotlb); -void vduse_domain_sync_single_for_device (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir); - -void vduse_domain_sync_single_for_cpu (struct vduse_iova_domain *domain, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir); - dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,drivers/vdpa/vdpa_user/vduse_dev.c +0
-40@@ -932,34 +932,6 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = { .free = vduse_vdpa_free, }; -static void vduse_dev_sync_single_for_device (union virtio_map token, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -struct vduse_iova_domain *domain; - -if (!token.group) -return ; - -guard (vq_group_as_read_lock)(token.group); - domain = token.group->as->domain; -vduse_domain_sync_single_for_device (domain, dma_addr, size, dir); -} - -static void vduse_dev_sync_single_for_cpu (union virtio_map token, -dma_addr_t dma_addr,size_t size, -enum dma_data_direction dir) -{ -struct vduse_iova_domain *domain; - -if (!token.group) -return ; - -guard (vq_group_as_read_lock)(token.group); - domain = token.group->as->domain; -vduse_domain_sync_single_for_cpu (domain, dma_addr, size, dir); -} - static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir,@@ -1037,15 +1009,6 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,free_pages_exact (vaddr, size); } -static bool vduse_dev_need_sync (union virtio_map token,dma_addr_t dma_addr) -{ -if (!token.group) -return false; - -guard (vq_group_as_read_lock)(token.group); -return dma_addr < token.group->as->domain->bounce_size; -} -static int vduse_dev_mapping_error (union virtio_map token,dma_addr_t dma_addr) {if (unlikely (dma_addr ==DMA_MAPPING_ERROR ))@@ -1063,13 +1026,10 @@ static size_t vduse_dev_max_mapping_size(union virtio_map token) }static const struct virtio_map_ops vduse_map_ops = { - .sync_single_for_device = vduse_dev_sync_single_for_device, - .sync_single_for_cpu = vduse_dev_sync_single_for_cpu, .map_page = vduse_dev_map_page, .unmap_page = vduse_dev_unmap_page, .alloc = vduse_dev_alloc_coherent, .free = vduse_dev_free_coherent, - .need_sync = vduse_dev_need_sync, .mapping_error = vduse_dev_mapping_error, .max_mapping_size = vduse_dev_max_mapping_size, };include/linux/virtio_config.h +0
-20@@ -157,16 +157,6 @@ struct virtio_config_ops {* size: the buffer size * dir: mapping direction * attrs: unmapping attributes -* @sync_single_for_cpu: sync a single buffer from device to cpu -* map: metadata for performing mapping -* map_handle: the mapping address to sync -* size: the size of the buffer -* dir: synchronization direction -* @sync_single_for_device: sync a single buffer from cpu to device -* map: metadata for performing mapping -* map_handle: the mapping address to sync -* size: the size of the buffer -* dir: synchronization direction * @alloc: alloc a coherent buffer mapping * map: metadata for performing mapping * size: the size of the buffer @@ -179,10 +169,6 @@ struct virtio_config_ops {* vaddr: virtual address of the buffer * map_handle: the mapping address that needs to be freed * attrs: unmapping attributes -* @need_sync: if the buffer needs synchronization -* map: metadata for performing mapping -* map_handle: the mapped address -* Returns whether the buffer needs synchronization * @mapping_error: if the mapping address is error * map: metadata for performing mapping * map_handle: the mapped address @@ -197,16 +183,10 @@ struct virtio_map_ops {void (*unmap_page)(union virtio_map map,dma_addr_t map_handle,size_t size,enum dma_data_direction dir,unsigned long attrs); -void (*sync_single_for_cpu)(union virtio_map map,dma_addr_t map_handle, -size_t size,enum dma_data_direction dir); -void (*sync_single_for_device)(union virtio_map map, -dma_addr_t map_handle,size_t size, -enum dma_data_direction dir);void *(*alloc)(union virtio_map map,size_t size,dma_addr_t *map_handle,gfp_t gfp);void (*free)(union virtio_map map,size_t size,void *vaddr,dma_addr_t map_handle,unsigned long attrs); -bool (*need_sync)(union virtio_map map,dma_addr_t map_handle);int (*mapping_error)(union virtio_map map,dma_addr_t map_handle);size_t (*max_mapping_size)(union virtio_map map); };
PATCH v2 3/12 vdpa: drop the VIRTIO_DEVICE_F_MASK example value
In reply to Alexander GrafThe comment above VIRTIO_DEVICE_F_MASK spells the mask out by hand as 0xfffffc000fffffff, which was correct while VIRTIO_TRANSPORT_F_END was
41. The bound is 42 now, so the macro expands to 0xfffff8000fffffff and whoever reads the comment to check a feature mask against it gets the wrong boundary between transport and per-device bits.
Correcting the number would leave a comment that has to be updated by hand every time the bound moves, and we already missed it once. Drop the expansion instead. The macro derives its value from the bound and needs no change.
--- drivers/vdpa/vdpa.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/vdpa/vdpa.c +3 -4
@@ -548,10 +548,9 @@ vdpa_nl_cmd_mgmtdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *cb)/* * Bitmask for all per-device features: feature bits VIRTIO_TRANSPORT_F_START -* through VIRTIO_TRANSPORT_F_END are unset, i.e. 0xfffffc000fffffff for -* all 64bit features. If the features are extended beyond 64 bits, or new -* "holes" are reserved for other type of features than per-device, this -* macro would have to be updated. +* through VIRTIO_TRANSPORT_F_END are unset. If the features are extended +* beyond 64 bits, or new "holes" are reserved for other type of features +* than per-device, this macro would have to be updated. */ #define VIRTIO_DEVICE_F_MASK (~0ULL << (VIRTIO_TRANSPORT_F_END + 1) | \ ((1ULL << VIRTIO_TRANSPORT_F_START) - 1))
Re: [PATCH v2 03/12] vdpa: drop the VIRTIO_DEVICE_F_MASK example value
Quoted 7 lines · Alexander Graf
The comment above
VIRTIO_DEVICE_F_MASKspells the mask out by hand as0xfffffc000fffffff, which was correct whileVIRTIO_TRANSPORT_F_ENDwas
41. The bound is 42 now, so the macro expands to0xfffff8000fffffffand whoever reads the comment to check a feature mask against it gets the wrong boundary between transport and per-device bits.
Quoted 7 lines
Correcting the number would leave a comment that has to be updated by hand every time the bound moves, and we already missed it once. Drop the expansion instead. The macro derives its value from the bound and needs no change.
Fixes 838bebb4c926 ("virtio: Define feature bit for administration virtqueue")Suggested-by Michael S. Tsirkin <[email protected]>
I'm not sure if a Fixes is needed though, as it will cause backports (which means work) that don't fix any code.
Quoted 25 lines
Assisted-by Kiro:claude-opus-5 checkpatch sparseSigned-off-by Alexander Graf <[email protected]>--- drivers/vdpa/vdpa.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)drivers/vdpa/vdpa.c +3
-4@@ -548,10 +548,9 @@ vdpa_nl_cmd_mgmtdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *cb)/* * Bitmask for all per-device features: feature bits VIRTIO_TRANSPORT_F_START -* through VIRTIO_TRANSPORT_F_END are unset, i.e. 0xfffffc000fffffff for -* all 64bit features. If the features are extended beyond 64 bits, or new -* "holes" are reserved for other type of features than per-device, this -* macro would have to be updated. +* through VIRTIO_TRANSPORT_F_END are unset. If the features are extended +* beyond 64 bits, or new "holes" are reserved for other type of features +* than per-device, this macro would have to be updated. */ #define VIRTIO_DEVICE_F_MASK (~0ULL << (VIRTIO_TRANSPORT_F_END + 1) | \ ((1ULL << VIRTIO_TRANSPORT_F_START) - 1))
PATCH v2 4/12 virtio_ring: return -ENOMEM when a packed ring mapping fails
In reply to Alexander GrafCommit f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails") moved virtqueue_add_split() and virtqueue_add_indirect_packed() to -ENOMEM, because virtio_queue_rq() maps -EIO to BLK_STS_IOERR and the request fails. We still return -EIO from virtqueue_add_packed(), and virtqueue_add_packed_in_order() copied that when it was added later.
Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring virtio_queue_rq() reports BLK_STS_RESOURCE and the block layer requeues the request. On a packed ring virtio_queue_rq() reports BLK_STS_IOERR instead and the error reaches the filesystem.
Return -ENOMEM from the packed unmap_release paths too. Both are reached from a single goto on a failed mapping, which is where vring_map_one_sg() already produces -ENOMEM.
That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
--- drivers/virtio/virtio_ring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +2 -2
@@ -1810,7 +1810,7 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; } static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
@@ -1966,7 +1966,7 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; }static bool virtqueue_kick_prepare_packed (struct vring_virtqueue *vq)
Re: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 17 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.
Quoted 5 lines
That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")
Quoted 29 lines
Assisted-by Kiro:claude-opus-5 checkpatch sparseSigned-off-by Alexander Graf <[email protected]>--- drivers/virtio/virtio_ring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/virtio/virtio_ring.c +2
-2@@ -1810,7 +1810,7 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; } static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,@@ -1966,7 +1966,7 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; }static bool virtqueue_kick_prepare_packed (struct vring_virtqueue *vq)
Eugenio Perez Martin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 19 lines · Eugenio Perez Martin
Quoted 17 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.
Quoted 8 lines
Quoted 5 lines
That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")Acked-by Eugenio Pérez <[email protected]>
Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
---
1. virtio_crypto (skcipher) — errno changes via AF_ALG socket
userspace: socket(AF_ALG, SOCK_SEQPACKET, 0)
bind(fd, {ALG_SET_KEY, "cbc(aes)"}, ...)
sendmsg(fd, plaintext, ...) ← submits data
recvmsg(fd, ciphertext, ...) ← triggers encryption,
waits for result
The kernel path:recvmsg()
→ skcipher_recvmsg() [crypto/algif_skcipher.c:180]
→ _skcipher_recvmsg() [crypto/algif_skcipher.c:84]
→ crypto_skcipher_encrypt(&areq->cra_u.skcipher_req)
→ virtio_crypto_skcipher_encrypt()
→ crypto_transfer_skcipher_request_to_engine() [enqueues req; engine kthread picks it up]
→ crypto_wait_req(err, &ctx->wait)
waits on completion
In the crypto engine kthread:crypto_pump_requests() [crypto/crypto_engine.c:~100]
→ op->do_one_request(engine, async_req)
= virtio_crypto_skcipher_crypt_req() [virtio_crypto_skcipher_algs.c:530]
→ __virtio_crypto_skcipher_do_req()
→ virtqueue_add_sgs(data_vq->vq, ...)
→ virtqueue_add_packed() ← changed here
Error propagation — before commit:virtqueue_add_packed() returns -EIO
__virtio_crypto_skcipher_do_req() returns -EIO (err propagated
directly)
virtio_crypto_skcipher_crypt_req() returns -EIO crypto_engine.c:155: crypto_request_complete(async_req, -EIO)
→ crypto_req_done(wait, -EIO)
→ wait->err = -EIO; complete(&wait->completion)
crypto_wait_req() returns -EIO
_skcipher_recvmsg() returns -EIO
skcipher_recvmsg() returns -EIO
userspace recvmsg() returns -1, errno = EIO (5)
Error propagation — after commit:virtqueue_add_packed() returns -ENOMEM
... same chain ...
userspace recvmsg() returns -1, errno = ENOMEM (12)
---
2. spi-virtio — errno changes via spidev ioctl
A userspace SPI application using /dev/spidevX.Y:
userspace: ioctl(fd, SPI_IOC_MESSAGE(n), &xfer)
→ spidev_ioctl() [drivers/spi/spidev.c]
→ spidev_message()
→ spi_sync(spi, &msg)
→ __spi_sync() [drivers/spi/spi.c:4687]
→ __spi_transfer_message_noqueue()
→ spi_transfer_one_message()
[drivers/spi/spi.c:~1603]
→ ctlr->transfer_one(ctlr, msg->spi, xfer)
= virtio_spi_transfer_one()[drivers/spi/spi-virtio.c:148]
→ virtqueue_add_sgs(priv->vq, ...)
→ virtqueue_add_packed() ←
changed here
Error propagation:
Inside virtio_spi_transfer_one() at line 251–253:msg_done:
if (ret)
ctrl->cur_msg->status = ret;
return ret;In spi_transfer_one_message() at line 1637–1655:
ret = ctlr->transfer_one(ctlr, msg->spi, xfer);
if (ret < 0) {
...goto out;
}// out: msg->status = ret (line 1708)
Back in __spi_sync() at line 4723: return message->status;
The errno from virtqueue_add_packed() propagates all the way to the ioctl() return value: before the commit, ioctl() returns -EIO; after, -ENOMEM. Any spidev userspace code that checks errno would observe the difference.
---
Claude also found another issue in gpio but the code has more ramifications and I couldn't verify it:
3. gpio-virtio — errno changes via GPIO character device ioctl
A userspace GPIO application using the character device API (/dev/gpiochipX):
userspace: ioctl(fd, GPIO_V2_GET_LINE_VALUES_IOCTL, &data)
→ linereq_ioctl() [drivers/gpio/gpiolib-cdev.c]
→ linereq_get_values()
→ gpiod_get_value_cansleep(line->desc)
→ gpio_chip_get_value() [drivers/gpio/gpiolib.c:3412]
→ gpiochip_get(gc, offset) [drivers/gpio/gpiolib.c:3394]
→ gc->get(gc, offset)
= virtio_gpio_get() [drivers/gpio/gpio-virtio.c:194]
→ virtio_gpio_req(vgpio, ...)
→ _virtio_gpio_req()
→virtqueue_add_sgs(vgpio->request_vq, ...)
→ virtqueue_add_packed() ←
changed here
_virtio_gpio_req() at line 100–104:
ret = virtqueue_add_sgs(vgpio->request_vq, sgs, 1, 1, line, GFP_KERNEL);
if (ret) {
...
goto out;
}// out: return ret
Apart from that, Claude also detected a change in behavior worth checking, regardless of whether this patch is applied. drivers/scsi/virtio_scsi.c:virtscsi_queuecommand will complete the command if the return of virtscsi_add_cmd is -EIO, but it will not do it if ret is different than 0 and -EIO.
Quoted 29 lines
Quoted 29 lines
Assisted-by Kiro:claude-opus-5 checkpatch sparseSigned-off-by Alexander Graf <[email protected]>--- drivers/virtio/virtio_ring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/virtio/virtio_ring.c +2
-2@@ -1810,7 +1810,7 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; } static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,@@ -1966,7 +1966,7 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, }END_USE (vq); -return -EIO ; +return -ENOMEM ; }static bool virtqueue_kick_prepare_packed (struct vring_virtqueue *vq)
Michael S. Tsirkin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Signature · 1 line
MST
Eugenio Perez Martin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 29 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Yes, but changing error codes returned from the kernel to userland always reminds me of this old thread so I wanted to give a heads up:
https://lkml.org/lkml/2012/12/23/75
Now I don't think we're in the same situation, though; probably no userland app checks the actual errno in these operations. However, the userland apps are not limited to VMMs; they include actual subsystem users that do not know the backend is a virtio device, so the base is large. If you still think that will not be a problem, I'm totally in :).
Thanks!
Michael S. Tsirkin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 45 lines · Eugenio Perez Martin
Quoted 29 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Yes, but changing error codes returned from the kernel to userland always reminds me of this old thread so I wanted to give a heads up:
Quoted 8 lines
Now I don't think we're in the same situation, though; probably no userland app checks the actual errno in these operations. However, the userland apps are not limited to VMMs; they include actual subsystem users that do not know the backend is a virtio device, so the base is large. If you still think that will not be a problem, I'm totally in :).
Thanks!
Well actual subsystem users for sure expect EIO on actual io errors no?
Eugenio Perez Martin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Eugenio Perez Martin
Quoted 29 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Yes, but changing error codes returned from the kernel to userland always reminds me of this old thread so I wanted to give a heads up:
https://lkml.org/lkml/2012/12/23/75
Now I don't think we're in the same situation, though; probably no userland app checks the actual errno in these operations. However, the userland apps are not limited to VMMs; they include actual subsystem users that do not know the backend is a virtio device, so the base is large. If you still think that will not be a problem, I'm totally in :).
Quoted 4 lines
Quoted 1 line
Thanks!
Well actual subsystem users for sure expect EIO on actual io errors no?
Yes, but code like drivers/scsi/virtio_scsi.c:virtscsi_queuecommand in the kernel explicitely checks for -EIO, and had a different code path to handle it than -ENOMEM. Even if both are valid, the visible behavior of the code changes. My fear here is that we have similar code in userland that we don't handle. I admit it is not very likely, and I actually prefer making both split and packed to return the same ERRNO, so if you're ok with the change I'm ok too.
Michael S. Tsirkin
maintainerRe: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 57 lines · Eugenio Perez Martin
Quoted 56 lines · Michael S. Tsirkin
Quoted 53 lines · Eugenio Perez Martin
Quoted 29 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Yes, but changing error codes returned from the kernel to userland always reminds me of this old thread so I wanted to give a heads up:
https://lkml.org/lkml/2012/12/23/75
Now I don't think we're in the same situation, though; probably no userland app checks the actual errno in these operations. However, the userland apps are not limited to VMMs; they include actual subsystem users that do not know the backend is a virtio device, so the base is large. If you still think that will not be a problem, I'm totally in :).
Thanks!
Quoted 7 lines
Quoted 2 lines
Well actual subsystem users for sure expect EIO on actual io errors no?
Yes, but code like drivers/scsi/virtio_scsi.c:virtscsi_queuecommand in the kernel explicitely checks for
-EIO, and had a different code path to handle it than-ENOMEM. Even if both are valid, the visible behavior of the code changes.
For the better.
Quoted 4 lines
My fear here is that we have similar
code in userland that we don't handle. I admit it is not very likely, and I actually prefer making both split and packed to return the same ERRNO, so if you're ok with the change I'm ok too.
Re: [PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails
Quoted 62 lines · Michael S. Tsirkin
Quoted 60 lines · Eugenio Perez Martin
Quoted 58 lines · Michael S. Tsirkin
Quoted 45 lines · Eugenio Perez Martin
Quoted 29 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 25 lines · Eugenio Perez Martin
Quoted 22 lines · Alexander Graf
Commit
f7728002c1c7("virtio_ring: fix return code on DMA mapping fails") movedvirtqueue_add_split()andvirtqueue_add_indirect_packed()to-ENOMEM, becausevirtio_queue_rq()maps-EIOtoBLK_STS_IOERRand the request fails. We still return-EIOfromvirtqueue_add_packed(), andvirtqueue_add_packed_in_order()copied that when it was added later.Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq()reportsBLK_STS_RESOURCEand the block layer requeues the request. On a packed ringvirtio_queue_rq()reportsBLK_STS_IOERRinstead and the error reaches the filesystem.Return
-ENOMEMfrom the packedunmap_releasepaths too. Both are reached from a single goto on a failed mapping, which is wherevring_map_one_sg()already produces-ENOMEM.That way every ring layout reports the same errno, and the block layer requeues the request instead of failing it.
Fixes f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")Fixes f6a15d854986 ("virtio_ring: add in order support")Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Acked-by Eugenio Pérez <[email protected]>Even if I'd like to see this merged, I'm having second thoughts because it introduces userland visible changes in some drivers. Are them acceptable?
I mean, fixing the kernel for the userspace is kinda what we do, right?
Yes, but changing error codes returned from the kernel to userland always reminds me of this old thread so I wanted to give a heads up:
Quoted 8 lines
Now I don't think we're in the same situation, though; probably no userland app checks the actual errno in these operations. However, the userland apps are not limited to VMMs; they include actual subsystem users that do not know the backend is a virtio device, so the base is large. If you still think that will not be a problem, I'm totally in :).
Thanks!
Well actual subsystem users for sure expect EIO on actual io errors no?
Quoted 7 lines
Quoted 4 lines
Yes, but code like drivers/scsi/virtio_scsi.c:virtscsi_queuecommand in the kernel explicitely checks for
-EIO, and had a different code path to handle it than-ENOMEM. Even if both are valid, the visible behavior of the code changes.For the better.
Yes, I agree on that :). So, if you're ok with the change,
Thanks!
PATCH v2 5/12 virtio: add the VIRTIO_F_DMB feature bit
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, define the feature bit. A device that negotiates it holds its virtqueues and the buffers they reference in a Device Memory Buffer, a shared memory region the device owns. The data DMA of such a device is routed through a per-device transparent IOMMU whose bus address space maps 1:1 onto that region, so every address the driver places in a virtqueue is an address in that space and the device can reach nothing the driver has not published there.
Define VIRTIO_F_DMB as 44 and raise VIRTIO_TRANSPORT_F_END to 45, so virtio_dev_probe() preserves the bit into finalize_features() where a transport can act on it. VIRTIO_DEVICE_F_MASK derives from the same bound and moves with it. vdpa reads the bound as VIRTIO_TRANSPORT_F_END + 1 while vring_transport_features() stops below it, so the mask drops bits 43 to 45 from the device_features vdpa reports over netlink, one bit beyond the last bit a transport can act on.
Define VIRTIO_DMB_MEM_TYPE_COHERENT too. A device reports the memory type of its region, and this is the one type that says a write by either side becomes visible to the other with no cache maintenance by the driver. Every transport reports the same type, so the name belongs here beside the feature bit and not in the header of one of them.
---
- Say bus address space rather than IOVA space
--- include/uapi/linux/virtio_config.h | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
include/uapi/linux/virtio_config.h +23 -1
@@ -52,7 +52,7 @@* rest are per-device feature bits. */ #define VIRTIO_TRANSPORT_F_START 28 -#define VIRTIO_TRANSPORT_F_END 42 +#define VIRTIO_TRANSPORT_F_END 45#ifndef VIRTIO_CONFIG_NO_LEGACY /* Do we get callbacks when the ring is completely used, even if we've
@@ -120,4 +120,26 @@*/ #define VIRTIO_F_ADMIN_VQ 41 +/* +* This feature indicates that the device offers a Device Memory Buffer: a +* shared memory region, owned by the device, that holds the virtqueues and +* the buffers they reference. When this feature is negotiated, the data DMA +* of the device is routed through a per-device transparent IOMMU whose bus +* address space maps 1:1 onto that region, so every address the driver +* places in a virtqueue is an address in that space and the device can +* reach nothing the driver has not published there. Interrupts are +* unaffected. +*/ +#define VIRTIO_F_DMB 44 + +/* +* The memory type of a Device Memory Buffer region, as the device reports it. +* VIRTIO_DMB_MEM_TYPE_COHERENT means a write by either side becomes visible to +* the other with no cache maintenance by the driver. Every other value is +* reserved. A device reports the type from the moment it offers +* VIRTIO_F_DMB, and a driver must not accept the feature unless the type is +* one it supports. +*/ +#define VIRTIO_DMB_MEM_TYPE_COHERENT 0 +#endif /* _UAPI_LINUX_VIRTIO_CONFIG_H */
PATCH v2 6/12 virtio_pci: read the device memory buffer registers
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, read the two registers a device uses to describe its Device Memory Buffer. Both are read-only le16 fields at the end of the common configuration structure. dmb_shm_id names the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that virtio_pci_find_shm_cap() has to find to map the region, and is valid once the feature is negotiated. dmb_mem_type describes the memory the region is made of: VIRTIO_DMB_MEM_TYPE_COHERENT means a write by either side becomes visible to the other with no cache maintenance by the driver. It is valid as soon as the device offers the feature, so a driver that does not support the value it reads can decline the offer instead of failing the device afterwards.
Add both fields, VIRTIO_PCI_COMMON_DMB_SHM_ID and VIRTIO_PCI_COMMON_DMB_MEM_TYPE for the offsets check_offsets() asserts them against, and vp_modern_get_dmb_shm_id() and vp_modern_get_dmb_mem_type() to read them. vp_modern_probe() capped the common cfg mapping at the end of admin_queue_num, exactly where dmb_shm_id starts, so extend it to the end of dmb_mem_type. vp_modern_map_capability() takes that as an upper bound, so a device with a shorter common cfg maps what it has and mdev->common_len records how much.
--- drivers/virtio/virtio_pci_modern_dev.c | 47 +++++++++++++++++++++++++- include/linux/virtio_pci_modern.h | 2 ++ include/uapi/linux/virtio_pci.h | 19 +++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-)
drivers/virtio/virtio_pci_modern_dev.c +46 -1
@@ -211,6 +211,10 @@ static inline void check_offsets(void) offsetof(struct virtio_pci_modern_common_cfg, admin_queue_index));BUILD_BUG_ON (VIRTIO_PCI_COMMON_ADM_Q_NUM != offsetof(struct virtio_pci_modern_common_cfg , admin_queue_num)); +BUILD_BUG_ON (VIRTIO_PCI_COMMON_DMB_SHM_ID != + offsetof(struct virtio_pci_modern_common_cfg , dmb_shm_id)); +BUILD_BUG_ON (VIRTIO_PCI_COMMON_DMB_MEM_TYPE != + offsetof(struct virtio_pci_modern_common_cfg , dmb_mem_type)); }/*
@@ -300,7 +304,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev) mdev->common =vp_modern_map_capability (mdev, common,sizeof (struct virtio_pci_common_cfg ),4 ,0 , offsetofend(struct virtio_pci_modern_common_cfg, - admin_queue_num), + dmb_mem_type), &mdev->common_len,NULL );if (!mdev->common) goto err_map_common;
@@ -752,6 +756,47 @@ u16 vp_modern_avq_index(struct virtio_pci_modern_device *mdev) }EXPORT_SYMBOL_GPL (vp_modern_avq_index); +/* +* vp_modern_get_dmb_shm_id - read the Device Memory Buffer shared memory id +* @mdev: the modern virtio-pci device +* +* The value identifies the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that +* describes the Device Memory Buffer region. Only valid once VIRTIO_F_DMB +* has been negotiated, and the caller has to have established that the common +* configuration structure is long enough to hold the field. +* +* Returns the shared memory id. +*/ +u16 vp_modern_get_dmb_shm_id (struct virtio_pci_modern_device *mdev) +{ +struct virtio_pci_modern_common_cfg __iomem *cfg; + + cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; +return vp_ioread16 (&cfg->dmb_shm_id); +} +EXPORT_SYMBOL_GPL (vp_modern_get_dmb_shm_id); + +/* +* vp_modern_get_dmb_mem_type - read the Device Memory Buffer memory type +* @mdev: the modern virtio-pci device +* +* The value describes the memory the Device Memory Buffer region is made of. +* VIRTIO_DMB_MEM_TYPE_COHERENT means the region is cache coherent. Valid +* whenever the device offers VIRTIO_F_DMB, and the caller has to have +* established that the common configuration structure is long enough to hold +* the field. +* +* Returns the memory type. +*/ +u16 vp_modern_get_dmb_mem_type (struct virtio_pci_modern_device *mdev) +{ +struct virtio_pci_modern_common_cfg __iomem *cfg; + + cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; +return vp_ioread16 (&cfg->dmb_mem_type); +} +EXPORT_SYMBOL_GPL (vp_modern_get_dmb_mem_type); +MODULE_VERSION ("0.1" );MODULE_DESCRIPTION ("Modern Virtio PCI Device" );MODULE_AUTHOR ("Jason Wang <[email protected]>" );
include/linux/virtio_pci_modern.h +2 -0
@@ -162,4 +162,6 @@ int vp_modern_get_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);void vp_modern_set_queue_reset (struct virtio_pci_modern_device *mdev,u16 index);u16 vp_modern_avq_num (struct virtio_pci_modern_device *mdev);u16 vp_modern_avq_index (struct virtio_pci_modern_device *mdev); +u16 vp_modern_get_dmb_shm_id (struct virtio_pci_modern_device *mdev); +u16 vp_modern_get_dmb_mem_type (struct virtio_pci_modern_device *mdev);#endif
include/uapi/linux/virtio_pci.h +19 -0
@@ -193,6 +193,23 @@ struct virtio_pci_modern_common_cfg {__le16 admin_queue_index;/* read-only */ __le16 admin_queue_num;/* read-only */ + +/* +* Reports the shmid of the Device Memory Buffer region. Valid once +* VIRTIO_F_DMB has been negotiated. +*/ +__le16 dmb_shm_id;/* read-only */ + +/* +* Reports the memory type of the Device Memory Buffer region. +* VIRTIO_DMB_MEM_TYPE_COHERENT means the region is cache coherent: a +* write by either side becomes visible to the other with no cache +* maintenance by the driver. Every other value is reserved, and a +* driver must not accept VIRTIO_F_DMB unless the value is one it +* supports. Valid whenever the device offers VIRTIO_F_DMB, so that a +* driver can read it before it accepts. +*/ +__le16 dmb_mem_type;/* read-only */ };/* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -235,6 +252,8 @@ struct virtio_pci_cfg_cap {#define VIRTIO_PCI_COMMON_Q_RESET 58#define VIRTIO_PCI_COMMON_ADM_Q_IDX 60#define VIRTIO_PCI_COMMON_ADM_Q_NUM 62 +#define VIRTIO_PCI_COMMON_DMB_SHM_ID 64 +#define VIRTIO_PCI_COMMON_DMB_MEM_TYPE 66#endif /* VIRTIO_PCI_NO_MODERN */
PATCH v2 7/12 virtio_pci: create virtqueues with the device's mapping token
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, create the modern PCI transport's virtqueues with the mapping token held on the virtio_device. Every mapping call for a virtqueue is handed a union virtio_map token, and vring_create_virtqueue() derives that token as {.dma_dev = vdev->dev.parent} without letting its callers choose.
Set vdev->vmap.dma_dev in virtio_pci_modern_probe() and create the virtqueues through vring_create_virtqueue_map() instead. That covers both the virtqueues from vp_modern_find_vqs() and the admin virtqueue, which go through the same setup_vq(). virtio_pci_probe() assigns vdev->dev.parent first, so the token holds the pointer vring_create_virtqueue() would have derived.
The legacy transport keeps its own vring_create_virtqueue() call:vp_legacy_get_features() reads 32 feature bits, so VIRTIO_F_DMB cannot reach it.
--- drivers/virtio/virtio_pci_modern.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
drivers/virtio/virtio_pci_modern.c +13 -4
@@ -715,10 +715,11 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, info->msix_vector = msix_vec;/* create the vring */ - vq =vring_create_virtqueue (index, num, -SMP_CACHE_BYTES , &vp_dev->vdev, - true, true, ctx, - notify, callback, name); + vq =vring_create_virtqueue_map (index, num, +SMP_CACHE_BYTES , &vp_dev->vdev, + true, true, ctx, + notify, callback, name, + vp_dev->vdev.vmap);if (!vq)return ERR_PTR (-ENOMEM );
@@ -1289,6 +1290,14 @@ int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev) vp_dev->isr = mdev->isr; vp_dev->vdev.id = mdev->id; +/* +* The mapping token every virtqueue of this device is created with. +* This is the same value vring_create_virtqueue() would derive from +* vdev->dev.parent, kept here so that a transport feature can replace +* it in one place. +*/ + vp_dev->vdev.vmap.dma_dev = &pci_dev->dev; +spin_lock_init (&vp_dev->admin_vq.lock);return 0 ; }
PATCH v2 8/12 virtio: add a device memory buffer region allocator
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, create a mechanism to allocate and map memory from the Device Memory Buffer (DMB). The DMB is a shared memory region a device exposes and owns. A device that negotiates the feature expects its virtqueues and all the buffers we hand it to live in that region, and its data DMA is routed through a per-device transparent IOMMU whose bus address space maps 1:1 onto that region.
Add virtio_dmb_init(), which locates the region by the shared memory id the device reports and builds a gen_pool over it at page granularity, and virtio_dmb_destroy() to tear that down. Add virtio_dmb_map_ops, a struct virtio_map_ops implementation that hands out allocations as addresses in that space: alloc() places a virtqueue area in the region, map_page() copies a buffer that lives elsewhere into it and copies it back on unmap. A new dmb member of union virtio_map carries the pool to those operations.
gen_pool tracks which pages are busy but not what an allocation is, so we keep one record per region page with the length and the bounced address of the allocation that starts there. The records are a single array allocated with the region, 16 bytes a page, which is a 256th of it.
The shared memory id is transport specific, so add a get_dmb_shm_id() callback to struct virtio_config_ops for a transport to report it. A transport that does not implement it must not accept VIRTIO_F_DMB. Add CONFIG_VIRTIO_DMB to enable this support, which defaults to y.
---
- Say bus address space rather than IOVA space
--- drivers/virtio/Kconfig | 17 + drivers/virtio/Makefile | 3 +- drivers/virtio/virtio_dmb.c | 798 ++++++++++++++++++++++++++++++++++ drivers/virtio/virtio_dmb.h | 28 ++ include/linux/virtio.h | 3 + include/linux/virtio_config.h | 8 + 6 files changed, 856 insertions(+), 1 deletion(-) create mode 100644 drivers/virtio/virtio_dmb.c create mode 100644 drivers/virtio/virtio_dmb.h
drivers/virtio/Kconfig +17 -0
@@ -188,6 +188,23 @@ config VIRTIO_DEBUG If unsure, say N. +config VIRTIO_DMB + bool "Device Memory Buffer support" + depends on VIRTIO + select GENERIC_ALLOCATOR + default y + help + Support devices that place their virtqueues and buffers in a shared + memory region they own, rather than in memory the driver allocates. + + Enabling this adds an allocator over that region and one record per + page of the region, both of which are set up only for a device that + negotiates the feature. When disabled no transport accepts the + feature, and a device offering it is driven as an ordinary one on a + platform that does not restrict memory access. + + If unsure, say Y. + config VIRTIO_RTC tristate "Virtio RTC driver" depends on VIRTIO
drivers/virtio/Makefile +2 -1
@@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o +virtio-dmb-$(CONFIG_VIRTIO_DMB) := virtio_dmb.o +obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o $(virtio-dmb-y) obj-$(CONFIG_VIRTIO_ANCHOR) += virtio_anchor.o obj-$(CONFIG_VIRTIO_PCI_LIB) += virtio_pci_modern_dev.o obj-$(CONFIG_VIRTIO_PCI_LIB_LEGACY) += virtio_pci_legacy_dev.o
drivers/virtio/virtio_dmb.c new file +798 -0
798 lines not on this page · raw message
drivers/virtio/virtio_dmb.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Device Memory Buffer support for virtio devices. +*/ +#ifndef _DRIVERS_VIRTIO_VIRTIO_DMB_H +#define _DRIVERS_VIRTIO_VIRTIO_DMB_H + +struct virtio_device ; + +#if IS_ENABLED (CONFIG_VIRTIO_DMB ) + +int virtio_dmb_init (struct virtio_device *vdev); +void virtio_dmb_destroy (struct virtio_device *vdev); + +#else + +static inline int virtio_dmb_init (struct virtio_device *vdev) +{ +return 0 ; +} + +static inline void virtio_dmb_destroy (struct virtio_device *vdev) +{ +} + +#endif /* CONFIG_VIRTIO_DMB */ + +#endif /* _DRIVERS_VIRTIO_VIRTIO_DMB_H */
include/linux/virtio.h +3 -0
@@ -44,12 +44,15 @@ struct virtqueue { };struct vduse_vq_group ; +struct virtio_dmb ;union virtio_map {/* Device that performs DMA */ struct device *dma_dev;/* VDUSE specific virtqueue group for doing map */ struct vduse_vq_group *group; +/* Device Memory Buffer holding the virtqueues and their buffers */ +struct virtio_dmb *dmb; }; int virtqueue_add_outbuf(struct virtqueue *vq,
include/linux/virtio_config.h +8 -0
@@ -96,6 +96,13 @@ struct virtqueue_info {* @set_vq_affinity: set the affinity for a virtqueue (optional). * @get_vq_affinity: get the affinity for a virtqueue (optional). * @get_shm_region: get a shared memory region based on the index. +* @get_dmb_shm_id: get the shared memory id of the Device Memory Buffer +* (optional). +* vdev: the device +* id: where to store the shared memory id +* Returns 0 on success or error status +* Only valid once VIRTIO_F_DMB has been negotiated. A transport that +* does not implement this must not accept VIRTIO_F_DMB. * @disable_vq_and_reset: reset a queue individually (optional). * vq: the virtqueue * Returns 0 on success or error status
@@ -135,6 +142,7 @@ struct virtio_config_ops { int index);bool (*get_shm_region)(struct virtio_device *vdev,struct virtio_shm_region *region,u8 id); +int (*get_dmb_shm_id)(struct virtio_device *vdev,u16 *id);int (*disable_vq_and_reset)(struct virtqueue *vq);int (*enable_vq_after_reset)(struct virtqueue *vq); };
PATCH v2 9/12 virtio: locate the device memory buffer after feature negotiation
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, locate a device's Device Memory Buffer once feature negotiation is complete. The shared memory id that locates the region may only be read after the device has confirmed FEATURES_OK, and finalize_features() runs before that.
virtio_features_ok() is the one place in the core that has just read FEATURES_OK back, so locate the region from there, and release it from virtio_dev_remove() and from the error paths of probe and restore. Restore and reset completion reach it too, where virtio_dmb_init() keeps the state of a device reporting the region it had, rebuilds it for a device reporting a different one, and refuses a device that moves its region while its driver has virtqueues.
Suspend and reset need no handling of their own. Neither virtio_device_freeze() nor virtio_reset_device() deletes a virtqueue, and a live virtqueue holds addresses inside the region, so we keep it across both.
--- drivers/virtio/virtio.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
drivers/virtio/virtio.c +14 -1
@@ -8,6 +8,8 @@#include <linux/of.h> #include <uapi/linux/virtio_ids.h> +#include "virtio_dmb.h" +/* Unique numbering for virtio devices. */ static DEFINE_IDA (virtio_index_ida);
@@ -231,7 +233,14 @@ static int virtio_features_ok(struct virtio_device *dev) status);return -ENODEV ; } -return 0 ; + +/* +* Negotiation is complete, so a Device Memory Buffer may now be +* located. Reached from probe, from resume and from reset +* completion, all of which have to end with the state matching what +* the device reports now. +*/ +return virtio_dmb_init (dev); }/**
@@ -361,6 +370,7 @@ static int virtio_dev_probe(struct device *_d) err:virtio_add_status (dev,VIRTIO_CONFIG_S_FAILED ); +virtio_dmb_destroy (dev);return err; }
@@ -377,6 +387,8 @@ static void virtio_dev_remove(struct device *_d)/* Driver should have reset device. */ WARN_ON_ONCE (dev->config->get_status (dev)); +virtio_dmb_destroy (dev); +/* Acknowledge the device's existence again. */ virtio_add_status (dev,VIRTIO_CONFIG_S_ACKNOWLEDGE );
@@ -650,6 +662,7 @@ static int virtio_device_restore_priv(struct virtio_device *dev, bool restore) err:virtio_add_status (dev,VIRTIO_CONFIG_S_FAILED ); +virtio_dmb_destroy (dev);return ret; }
PATCH v2 10/12 virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
In reply to Alexander GrafIn preparation to support VIRTIO_F_DMB, read the platform DMA bypass quirk from either feature. The data DMA of a device that negotiates DMB is routed through the IOMMU in front of the region it owns, so the driver has to map every address it publishes through virtio_map_ops. VIRTIO_F_ACCESS_PLATFORM asks a driver for the same thing, and virtio_has_dma_quirk() looks only at it.
virtio_features_ok() demands VIRTIO_F_ACCESS_PLATFORM of a device under restricted memory access, so accept VIRTIO_F_DMB there too.
--- drivers/virtio/virtio.c | 10 ++++++++-- include/linux/virtio_config.h | 13 ++++++++++++- 2 files changed, 20 insertions(+), 3 deletions(-)
drivers/virtio/virtio.c +8 -2
@@ -216,9 +216,15 @@ static int virtio_features_ok(struct virtio_device *dev)return -ENODEV ; } -if (!virtio_has_feature (dev,VIRTIO_F_ACCESS_PLATFORM )) { +/* +* VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_DMB both ensure that +* the device does not access guest memory directly, bypassing +* the platform DMA topology. +*/ +if (!virtio_has_feature (dev,VIRTIO_F_ACCESS_PLATFORM ) && + !virtio_has_feature (dev,VIRTIO_F_DMB )) {dev_warn (&dev->dev, -"device must provide VIRTIO_F_ACCESS_PLATFORM\n" ); +"device must provide VIRTIO_F_ACCESS_PLATFORM or VIRTIO_F_DMB\n" );return -ENODEV ; } }
include/linux/virtio_config.h +12 -1
@@ -273,8 +273,19 @@ static inline bool virtio_has_dma_quirk(const struct virtio_device *vdev)/* * Note the reverse polarity of the quirk feature (compared to most * other features), this is for compatibility with legacy systems. +* +* VIRTIO_F_ACCESS_PLATFORM says that the device is subject to the +* platform DMA topology and the driver maps every address it +* publishes through that topology. The device performs no direct +* guest memory access, so the quirk does not apply. +* +* VIRTIO_F_DMB says that the device holds the virtqueues and the +* buffers they reference in a region it owns, and routes its data +* DMA through the IOMMU in front of that region. The driver maps +* through that IOMMU, so the quirk does not apply either. */ -return !virtio_has_feature (vdev,VIRTIO_F_ACCESS_PLATFORM ); +return !(virtio_has_feature (vdev,VIRTIO_F_ACCESS_PLATFORM ) || +virtio_has_feature (vdev,VIRTIO_F_DMB )); } static inline
PATCH v2 11/12 virtio_pci: support VIRTIO_F_DMB
In reply to Alexander GrafLet a modern virtio-pci device place its virtqueues and the buffers they reference in a Device Memory Buffer of its own: accept VIRTIO_F_DMB from vp_transport_features(), and implement the get_dmb_shm_id config op on top of vp_modern_get_dmb_shm_id(). get_dmb_shm_id reports the shared memory id of the region that holds the buffer.
Take the offer only for a dmb_mem_type of VIRTIO_DMB_MEM_TYPE_COHERENT, the memory type that says a write by either side becomes visible to the other with no cache maintenance by the driver. The driver uses plain loads and stores on the region, so no other type will do. The field is valid from the moment the device offers the feature, so a type the driver does not support leaves the device driven as an ordinary one. The shmid is not readable that early, so a region we fail to locate after the accept fails virtio_features_ok() and probe sets the FAILED status bit.
Refuse a device whose common configuration is too short to hold dmb_mem_type, which would put that read outside what vp_modern_probe() mapped. The accept path measures the structure itself, because vp_check_common_size() runs on the features the driver has already accepted and so cannot cover a read that precedes it.
Two more things gate the accept. CONFIG_VIRTIO_DMB, so that a device offering the feature to a kernel built without it is driven as an ordinary device. And VIRTIO_F_VERSION_1, because virtio_features_ok() returns early without it, which would leave the feature negotiated and the region never built; this transport refuses such a device anyway.
---
- Explain in a comment how a transport feature survives negotiation
- Say ACCESS_PLATFORM is implied rather than dismissing it
--- drivers/virtio/virtio_pci_modern.c | 58 ++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
drivers/virtio/virtio_pci_modern.c +58 -0
@@ -364,6 +364,29 @@ static void vp_modern_avq_cleanup(struct virtio_device *vdev) } } +/* +* A device reports the memory type of its Device Memory Buffer region in +* dmb_mem_type, and the field is valid as soon as the device offers +* VIRTIO_F_DMB. VIRTIO_DMB_MEM_TYPE_COHERENT says that a write by either side +* becomes visible to the other with no cache maintenance by the driver, which +* is what plain loads and stores on the region need. A common configuration +* structure too short to hold the field reports no type at all, and reading it +* there would fall outside what vp_modern_probe() mapped. This is the only +* place that measures the structure against the field, because the read +* happens before vp_check_common_size() runs. +*/ +static bool vp_dmb_mem_type_supported (struct virtio_pci_device *vp_dev) +{ + size_t need = offsetofend(struct virtio_pci_modern_common_cfg, + dmb_mem_type); + +if (vp_dev->mdev.common_len < need) +return false; + +return vp_modern_get_dmb_mem_type (&vp_dev->mdev) == +VIRTIO_DMB_MEM_TYPE_COHERENT ; +} +static void vp_transport_features (struct virtio_device *vdev,u64 features) {struct virtio_pci_device *vp_dev =to_vp_device (vdev);
@@ -378,6 +401,30 @@ static void vp_transport_features(struct virtio_device *vdev, u64 features)if (features &BIT_ULL (VIRTIO_F_ADMIN_VQ ))__virtio_set_bit (vdev,VIRTIO_F_ADMIN_VQ ); + +/* +* vring_transport_features() clears every bit in the transport range +* that the ring itself does not consume, and vp_finalize_features() +* hands the features the device offered to this function afterwards. +* A transport feature therefore survives negotiation only if the +* transport sets it again here, the way VIRTIO_F_SR_IOV, +* VIRTIO_F_RING_RESET and VIRTIO_F_ADMIN_VQ do above, so this is where +* the driver accepts a Device Memory Buffer and declining refuses it. +* +* VIRTIO_F_VERSION_1 is required because the core locates and releases +* the region from virtio_features_ok(), which returns before it gets +* that far for a device without VERSION_1, so accepting the feature +* without it would leave the feature negotiated and the region never +* built. VIRTIO_F_ACCESS_PLATFORM is not required alongside it, +* because virtio_has_dma_quirk() already reads this feature as +* carrying what that one asserts. Whether the device asks for the +* stronger barriers is its own business and does not bear on this one. +*/ +if (IS_ENABLED (CONFIG_VIRTIO_DMB ) && + (features &BIT_ULL (VIRTIO_F_DMB )) && + (features &BIT_ULL (VIRTIO_F_VERSION_1 )) && +vp_dmb_mem_type_supported (vp_dev)) +__virtio_set_bit (vdev,VIRTIO_F_DMB ); } static int __vp_check_common_size_one_feature(struct virtio_device *vdev, u32 fbit,
@@ -878,6 +925,15 @@ static bool vp_get_shm_region(struct virtio_device *vdev,return true; } +static int vp_get_dmb_shm_id (struct virtio_device *vdev,u16 *id) +{ +struct virtio_pci_device *vp_dev =to_vp_device (vdev); + + *id =vp_modern_get_dmb_shm_id (&vp_dev->mdev); + +return 0 ; +} +/* * virtio_pci_admin_has_dev_parts - Checks whether the device parts * functionality is supported
@@ -1241,6 +1297,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = { .set_vq_affinity = vp_set_vq_affinity, .get_vq_affinity = vp_get_vq_affinity, .get_shm_region = vp_get_shm_region, + .get_dmb_shm_id = vp_get_dmb_shm_id, .disable_vq_and_reset = vp_modern_disable_vq_and_reset, .enable_vq_after_reset = vp_modern_enable_vq_after_reset, };
@@ -1261,6 +1318,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = { .set_vq_affinity = vp_set_vq_affinity, .get_vq_affinity = vp_get_vq_affinity, .get_shm_region = vp_get_shm_region, + .get_dmb_shm_id = vp_get_dmb_shm_id, .disable_vq_and_reset = vp_modern_disable_vq_and_reset, .enable_vq_after_reset = vp_modern_enable_vq_after_reset, };
[email protected]
AI reviewRe: [PATCH v2 11/12] virtio_pci: support VIRTIO_F_DMB
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Out-of-bounds MMIO read in vp_dmb_mem_type_supported due to mdev->common_len exceeding the actual mapped BAR size. --
--- Patch [11]: [PATCH v2 11/12] virtio_pci: support VIRTIO_F_DMB --- commit 2b2d786abf770566580dde39b790350db36f5d09
Author: Alexander Graf <[email protected]>
virtio_pci: support VIRTIO_F_DMB
Let a modern virtio-pci device place its virtqueues and the buffers they reference in a Device Memory Buffer of its own: accept VIRTIO_F_DMB from vp_transport_features(), and implement the get_dmb_shm_id config op on top of vp_modern_get_dmb_shm_id(). get_dmb_shm_id reports the shared memory id of the region that holds the buffer.
Quoted 4 lines
drivers/virtio/virtio_pci_modern.c +0
-0
[ ... ]
Quoted 14 lines
drivers/virtio/virtio_pci_modern.c +10
-0@@ -364,6 +364,29 @@ static void vp_modern_avq_cleanup(struct virtio_device *vdev) } } +/* +* A device reports the memory type of its Device Memory Buffer region in +* dmb_mem_type, and the field is valid as soon as the device offers +* VIRTIO_F_DMB. VIRTIO_DMB_MEM_TYPE_COHERENT says that a write by either side +* becomes visible to the other with no cache maintenance by the driver, which +* is what plain loads and stores on the region need. A common configuration +* structure too short to hold the field reports no type at all, and reading it +* there would fall outside what vp_modern_probe() mapped. This is the only +* place that measures the structure against the field, because the read +* happens before vp_check_common_size() runs.
drivers/virtio/virtio_pci_modern.c static void vp_modern_avq_cleanup(struct virtio_device *vdev) · line 386
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +10
-0+*/ +static bool vp_dmb_mem_type_supported (struct virtio_pci_device *vp_dev) +{ + size_t need = offsetofend(struct virtio_pci_modern_common_cfg, + dmb_mem_type); + +if (vp_dev->mdev.common_len < need) +return false; + + return vp_modern_get_dmb_mem_type(&vp_dev->mdev) ==
[Severity: High]
This isn't a bug introduced by this patch, but does this bounds check rely on an inflated common_len that could allow out-of-bounds MMIO reads?
If a buggy or malicious PCI device sets the capability length to 68 or more, but places it near the end of the BAR such that the remaining space is only between 56 (minlen) and 67 bytes, vp_modern_map_capability() sets mdev->common_len to the un-clamped capability length:
drivers/virtio/virtio_pci_modern_dev.c:vp_modern_map_capability() {
...
if (length > size)
length = size;
if (len)
*len = length; /* mdev->common_len receives the inflated length */
if (minlen + offset < minlen ||
minlen + offset > pci_resource_len(dev, bar)) {
...
}Since pci_iomap_range() will only map up to the physical end of the BAR, the actual mapped region is shorter than common_len.
Because common_len is inflated, the check (vp_dev->mdev.common_len < need) passes here, allowing vp_modern_get_dmb_mem_type() to perform a read which is physically outside the mapped PCI BAR.
Could this unmapped read trigger a PCIe Unsupported Request and Master Abort, potentially panicking the host system?
Quoted 5 lines
drivers/virtio/virtio_pci_modern.c +3
-0+VIRTIO_DMB_MEM_TYPE_COHERENT ; +} +static void vp_transport_features (struct virtio_device *vdev,u64 features) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
PATCH v2 12/12 virtio: expose device memory buffer occupancy over debugfs
In reply to Alexander GrafA driver allocates from a pool bounded by the region a Device Memory Buffer device reports, so running out of room in it is routine:vring_map_errno() reports the shortage as the -ENOSPC a full queue reports, and the driver retries and carries on. The allocator must not log that at any level a working device prints, or an undersized region turns into a log flood. That leaves an operator nothing but unexplained throughput loss to go on.
swiotlb can afford dev_warn_ratelimited() because exhaustion there is a misconfiguration, and it exports io_tlb_used and io_tlb_used_hiwater through debugfs, which a log line cannot do for a sampled quantity. Follow it, plus one file swiotlb has no need for:
dmb/pages pages the allocator can hand out
dmb/used_pages pages allocated now
dmb/used_pages_hiwater the largest used_pages has been
dmb/alloc_failed buffer mappings the pool had no room for
used_pages comes from the allocator's own free count, so no counter is maintained alongside it and no read can report a torn total. Writing 0 to used_pages_hiwater restarts the measurement from the occupancy now, so a peak never reads below the used_pages read alongside it.
used_pages_hiwater is the number to size a region against, because a burst that fills the pool between two samples of used_pages leaves no other trace.
---
- Rework description
--- drivers/virtio/virtio_dmb.c | 163 +++++++++++++++++++++++++++++++++++- 1 file changed, 162 insertions(+), 1 deletion(-)
drivers/virtio/virtio_dmb.c +162 -1
@@ -20,6 +20,8 @@*/ #include <linux/align.h> +#include <linux/atomic.h> +#include <linux/debugfs.h> #include <linux/dma-mapping.h> #include <linux/export.h> #include <linux/genalloc.h>
@@ -83,6 +85,10 @@ struct virtio_dmb_alloc {* @base_va: kernel address the pool starts at, inside the mapping * @base_off: region address the pool starts at * @nslots: pool size in PAGE_SIZE pages +* @used_hiwater: the largest occupancy has been since the last reset through +* debugfs, or since init, in pages; CONFIG_VIRTIO_DEBUG +* @alloc_failed: buffer mappings the pool had no room for; CONFIG_VIRTIO_DEBUG +* @debugfs_dir: directory holding this region's debugfs files * @shm_id: shared memory id the device reported for the region */ struct virtio_dmb {
@@ -98,6 +104,11 @@ struct virtio_dmb {void *base_va;u64 base_off;unsigned int nslots; +#ifdef CONFIG_VIRTIO_DEBUG +atomic_long_t used_hiwater; +atomic_long_t alloc_failed; +#endif +struct dentry *debugfs_dir;u16 shm_id; };
@@ -111,6 +122,52 @@ static size_t virtio_dmb_pool_size(const struct virtio_dmb *dmb)return (size_t )dmb->nslots <<PAGE_SHIFT ; } +#ifdef CONFIG_VIRTIO_DEBUG + +/* +* Pages allocated now. gen_pool_avail() sums its chunks' free counts under +* RCU, so this needs no counter of its own and no lock, and it is exact +* between two claims rather than approximate. +*/ +static unsigned long virtio_dmb_used (struct virtio_dmb *dmb) +{ +return (gen_pool_size (dmb->pool) -gen_pool_avail (dmb->pool)) + >>PAGE_SHIFT ; +} + +/* +* Raise the high-water mark to the occupancy a claim has just established. +* Two racing claims each observe a real total and the larger wins, so the +* figure is a occupancy the pool genuinely held rather than a sum of readings +* taken at different moments. +*/ +static void virtio_dmb_note_used (struct virtio_dmb *dmb) +{ +long old =atomic_long_read (&dmb->used_hiwater); +long now =virtio_dmb_used (dmb); + +while (now > old && + !atomic_long_try_cmpxchg (&dmb->used_hiwater, &old, now)) + ; +} + +static void virtio_dmb_inc_alloc_failed (struct virtio_dmb *dmb) +{ +atomic_long_inc (&dmb->alloc_failed); +} + +#else /* !CONFIG_VIRTIO_DEBUG */ + +static void virtio_dmb_note_used (struct virtio_dmb *dmb) +{ +} + +static void virtio_dmb_inc_alloc_failed (struct virtio_dmb *dmb) +{ +} + +#endif /* CONFIG_VIRTIO_DEBUG */ +/* Handle the driver publishes for the allocation starting at pool page @slot. */ static dma_addr_t virtio_dmb_handle (const struct virtio_dmb *dmb,unsigned int slot)
@@ -178,6 +235,8 @@ static void *virtio_dmb_claim(struct virtio_dmb *dmb, size_t len, rec->len = len; rec->src = src; +virtio_dmb_note_used (dmb); + *out = rec;return (void *)va; }
@@ -371,8 +430,22 @@ static dma_addr_t virtio_dmb_op_map_page(union virtio_map map,return DMA_MAPPING_ERROR ; va =virtio_dmb_claim (dmb, size, src, &slot, &rec); -if (!va) +if (!va) { +/* +* Counted here and not in virtio_dmb_claim(), which alloc() +* reaches as well. A virtqueue area that does not fit is a +* step of vring_alloc_queue_split()'s search for a size that +* does, so counting it would have a correctly sized region boot +* with a failure for every attempt but the last, in the one +* file whose purpose is to answer whether the region is too +* small for the traffic. A request over the per-mapping cap is +* not counted either: the cap is what max_mapping_size() +* advertises, so exceeding it is a caller bug rather than a +* property of the region. +*/ +virtio_dmb_inc_alloc_failed (dmb);return DMA_MAPPING_ERROR ; + }/* * Copy in whatever the direction is, and without honouring
@@ -448,6 +521,91 @@ static const struct virtio_map_ops virtio_dmb_map_ops = { .max_mapping_size = virtio_dmb_op_max_mapping_size, }; +#ifdef CONFIG_VIRTIO_DEBUG + +static int virtio_dmb_used_get (void *data,u64 *val) +{ +struct virtio_dmb *dmb = data; + + *val =virtio_dmb_used (dmb); + +return 0 ; +} +DEFINE_DEBUGFS_ATTRIBUTE (virtio_dmb_used_fops, virtio_dmb_used_get,NULL , +"%llu\n" ); + +static int virtio_dmb_hiwater_get (void *data,u64 *val) +{ +struct virtio_dmb *dmb = data; + + *val =atomic_long_read (&dmb->used_hiwater); + +return 0 ; +} + +/* +* Restart the measurement from the occupancy now, so that a peak never reads +* below the used_pages read alongside it. +*/ +static int virtio_dmb_hiwater_set (void *data,u64 val) +{ +struct virtio_dmb *dmb = data; + +if (val) +return -EINVAL ; + +atomic_long_set (&dmb->used_hiwater,virtio_dmb_used (dmb)); + +return 0 ; +} +DEFINE_DEBUGFS_ATTRIBUTE (virtio_dmb_hiwater_fops, virtio_dmb_hiwater_get, + virtio_dmb_hiwater_set,"%llu\n" ); + +static int virtio_dmb_alloc_failed_get (void *data,u64 *val) +{ +struct virtio_dmb *dmb = data; + + *val =atomic_long_read (&dmb->alloc_failed); + +return 0 ; +} +DEFINE_DEBUGFS_ATTRIBUTE (virtio_dmb_alloc_failed_fops, + virtio_dmb_alloc_failed_get,NULL ,"%llu\n" ); + +static void virtio_dmb_debugfs_init (struct virtio_dmb *dmb) +{ +struct dentry *dir; + + dir =debugfs_create_dir ("dmb" , dmb->vdev->debugfs_dir); + dmb->debugfs_dir = dir; + +debugfs_create_u32 ("pages" ,0400 , dir, &dmb->nslots); +debugfs_create_file ("used_pages" ,0400 , dir, dmb, + &virtio_dmb_used_fops); +debugfs_create_file ("used_pages_hiwater" ,0600 , dir, dmb, + &virtio_dmb_hiwater_fops); +debugfs_create_file ("alloc_failed" ,0400 , dir, dmb, + &virtio_dmb_alloc_failed_fops); +} + +static void virtio_dmb_debugfs_exit (struct virtio_dmb *dmb) +{ +debugfs_remove_recursive (dmb->debugfs_dir); + dmb->debugfs_dir =NULL ; +} + +#else /* !CONFIG_VIRTIO_DEBUG */ + +static void virtio_dmb_debugfs_init (struct virtio_dmb *dmb) +{ +} + +static void virtio_dmb_debugfs_exit (struct virtio_dmb *dmb) +{ +} + +#endif /* CONFIG_VIRTIO_DEBUG */ +/* * Whether the device still has virtqueues. No caller here can race an adder, * because every path that reaches this runs under the device lock, but the
@@ -515,6 +673,7 @@ void virtio_dmb_destroy(struct virtio_device *vdev)* virtqueue, and the refusal above establishes that none is left, which * is also why the records are empty by now. */ +virtio_dmb_debugfs_exit (dmb);gen_pool_destroy (dmb->pool);kvfree (dmb->allocs);memunmap (dmb->map_va);
@@ -773,6 +932,8 @@ int virtio_dmb_init(struct virtio_device *vdev) vdev->vmap.dmb = dmb; vdev->map = &virtio_dmb_map_ops; +virtio_dmb_debugfs_init (dmb); +dev_info (&vdev->dev,"device memory buffer %u at 0x%016llx, %u usable pages\n" , shm_id, region.addr, nslots);