virtio: alignment issues
Subsystems (18)
Michael S. Tsirkin
maintainerv8 cover virtio: alignment issues
This is an alternative to
vhost: force spec specified alignment on types
which is a bit safer as it does not change UAPI. I still think it's best to change the UAPI header as well, we can do that as a follow-up cleanup.
changes from v7:
typo fix
changes from v6:
add missing header includes all over the place
changes from v5:
ack for mellanox patch
fixup to remoteproc
changes from v4:
fixup to issues reported by kbuild
changes from v3:
tools/virtio fixes
a bunch more cleanups that now become possible
Changes from v2:
don't change struct name, instead add ifndef so kernel does not see the legacy UAPI version.
Jason, can you pls ack one of the approaches?
Matej Genci (1):
virtio: add VIRTIO_RING_NO_LEGACY
Michael S. Tsirkin (18):
tools/virtio: define aligned attribute
tools/virtio: make asm/barrier.h self contained tools/virtio: define __KERNEL__
virtgpu: pull in uaccess.h
virtio-rng: pull in slab.h
remoteproc: pull in slab.h
virtio_input: pull in slab.h
virtio: stop using legacy struct vring in kernel
vhost: force spec specified alignment on types
virtio: add legacy init/size APIs
virtio_ring: switch to virtio_legacy_init/size tools/virtio: switch to virtio_legacy_init/size
vop: switch to virtio_legacy_init/size
remoteproc: switch to virtio_legacy_init/size
mellanox: switch to virtio_legacy_init/size
vhost: option to fetch descriptors through an independent struct
vhost: use batched version by default
vhost: batching fetches
drivers/block/virtio_blk.c | 1 + drivers/char/hw_random/virtio-rng.c | 1 + drivers/gpu/drm/virtio/virtgpu_ioctl.c | 1 + drivers/misc/mic/vop/vop_main.c | 5 +- drivers/misc/mic/vop/vop_vringh.c | 8 +- drivers/platform/mellanox/mlxbf-tmfifo.c | 6 +- drivers/remoteproc/remoteproc_core.c | 2 +- drivers/remoteproc/remoteproc_sysfs.c | 1 + drivers/remoteproc/remoteproc_virtio.c | 2 +- drivers/vhost/test.c | 2 +- drivers/vhost/vhost.c | 271 +++++++++++++++-------- drivers/vhost/vhost.h | 23 +- drivers/virtio/virtio_input.c | 1 + drivers/virtio/virtio_pci_modern.c | 1 + drivers/virtio/virtio_ring.c | 15 +- include/linux/virtio.h | 1 - include/linux/virtio_ring.h | 46 ++++ include/linux/vringh.h | 1 + include/uapi/linux/virtio_ring.h | 30 ++- tools/virtio/Makefile | 2 +- tools/virtio/asm/barrier.h | 1 + tools/virtio/linux/compiler.h | 1 + tools/virtio/ringtest/virtio_ring_0_9.c | 6 +- tools/virtio/virtio_test.c | 6 +- tools/virtio/vringh_test.c | 18 +- 25 files changed, 311 insertions(+), 141 deletions(-)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 1/19 tools/virtio: define aligned attribute
--- tools/virtio/linux/compiler.h | 1 + 1 file changed, 1 insertion(+)
tools/virtio/linux/compiler.h +1 -0
@@ -7,4 +7,5 @@#define READ_ONCE (var) (*((volatile typeof(var) *)(&(var)))) +#define __aligned (x) __attribute((__aligned__(x)))#endif
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 2/19 tools/virtio: make asm/barrier.h self contained
In reply to Michael S. TsirkinWe are using abort() so let's include stdlib.h
--- tools/virtio/asm/barrier.h | 1 + 1 file changed, 1 insertion(+)
tools/virtio/asm/barrier.h +1 -0
@@ -1,4 +1,5 @@/* SPDX-License-Identifier: GPL-2.0 */ +#include <stdlib.h> #if defined(__i386__) || defined(__x86_64__)#define barrier () asm volatile("" ::: "memory")#define virt_mb () __sync_synchronize()
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 3/19 tools/virtio: define __KERNEL__
In reply to Michael S. TsirkinEven though we are building in userspace, we are emulating kernel bits.
--- tools/virtio/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/virtio/Makefile +1 -1
@@ -4,7 +4,7 @@ test: virtio_test vringh_test virtio_test: virtio_ring.o virtio_test.o vringh_test: vringh_test.o vringh.o virtio_ring.o -CFLAGS += -g -O2 -Werror -Wall -I. -I../include/ -I ../../usr/include/ -Wno-pointer-sign -fno-strict-overflow -fno-strict-aliasing -fno-common -MMD -U_FORTIFY_SOURCE -include ../../include/linux/kconfig.h +CFLAGS += -g -O2 -Werror -Wall -I. -I../include/ -I ../../usr/include/ -Wno-pointer-sign -fno-strict-overflow -fno-strict-aliasing -fno-common -MMD -U_FORTIFY_SOURCE -include ../../include/linux/kconfig.h -D__KERNEL__ vpath %.c ../../drivers/virtio ../../drivers/vhost mod: ${MAKE} -C `pwd`/../.. M=`pwd`/vhost_test V=${V}
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 4/19 virtio: add VIRTIO_RING_NO_LEGACY
In reply to Michael S. TsirkinFrom: Matej Genci <[email protected]>
Add macro to disable legacy vring functions.
--- drivers/virtio/virtio_pci_modern.c | 1 + include/uapi/linux/virtio_ring.h | 4 ++++ 2 files changed, 5 insertions(+)
drivers/virtio/virtio_pci_modern.c +1 -0
@@ -16,6 +16,7 @@#include <linux/delay.h> #define VIRTIO_PCI_NO_LEGACY +#define VIRTIO_RING_NO_LEGACY #include "virtio_pci_common.h" /*
include/uapi/linux/virtio_ring.h +4 -0
@@ -135,6 +135,8 @@ struct vring {#define VRING_USED_ALIGN_SIZE 4#define VRING_DESC_ALIGN_SIZE 16 +#ifndef VIRTIO_RING_NO_LEGACY +/* The standard layout for the ring is a continuous chunk of memory which looks * like this. We assume num is a power of 2. *
@@ -181,6 +183,8 @@ static inline unsigned vring_size(unsigned int num, unsigned long align) + sizeof(__virtio16) * 3 + sizeof(struct vring_used_elem) * num; } +#endif /* VIRTIO_RING_NO_LEGACY */ +/* The following is used with USED_EVENT_IDX and AVAIL_EVENT_IDX */ /* Assuming a given event_idx value from the other side, if * we have just incremented index from old to new_idx,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 5/19 virtgpu: pull in uaccess.h
In reply to Michael S. TsirkinIn preparation to virtio header changes, include uaccess.h directly as this file is using copy to/from user.
--- drivers/gpu/drm/virtio/virtgpu_ioctl.c | 1 + 1 file changed, 1 insertion(+)
drivers/gpu/drm/virtio/virtgpu_ioctl.c +1 -0
@@ -27,6 +27,7 @@#include <linux/file.h> #include <linux/sync_file.h> +#include <linux/uaccess.h> #include <drm/drm_file.h> #include <drm/virtgpu_drm.h>
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 6/19 virtio-rng: pull in slab.h
In reply to Michael S. TsirkinIn preparation to virtio header changes, include slab.h directly as this module is using it.
--- drivers/char/hw_random/virtio-rng.c | 1 + 1 file changed, 1 insertion(+)
drivers/char/hw_random/virtio-rng.c +1 -0
@@ -11,6 +11,7 @@#include <linux/virtio.h> #include <linux/virtio_rng.h> #include <linux/module.h> +#include <linux/slab.h> static DEFINE_IDA (rng_index_ida);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 7/19 remoteproc: pull in slab.h
In reply to Michael S. TsirkinIn preparation to virtio header changes, include slab.h directly as this module is using it.
--- drivers/remoteproc/remoteproc_sysfs.c | 1 + 1 file changed, 1 insertion(+)
drivers/remoteproc/remoteproc_sysfs.c +1 -0
@@ -4,6 +4,7 @@*/ #include <linux/remoteproc.h> +#include <linux/slab.h> #include "remoteproc_internal.h"
Signature · 1 line
MST
Re: [PATCH v8 07/19] remoteproc: pull in slab.h
Quoted 4 lines · Michael S. Tsirkin
In preparation to virtio header changes, include
slab.hdirectly as this module is using it.Signed-off-by Michael S. Tsirkin <[email protected]>
Regards,
Bjorn
Quoted 19 lines
--- drivers/remoteproc/remoteproc_sysfs.c | 1 + 1 file changed, 1 insertion(+)drivers/remoteproc/remoteproc_sysfs.c +1
-0@@ -4,6 +4,7 @@*/ #include <linux/remoteproc.h> +#include <linux/slab.h> #include "remoteproc_internal.h" Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 8/19 virtio_input: pull in slab.h
In reply to Michael S. TsirkinIn preparation to virtio header changes, include slab.h directly as this module is using it.
--- drivers/virtio/virtio_input.c | 1 + 1 file changed, 1 insertion(+)
drivers/virtio/virtio_input.c +1 -0
@@ -3,6 +3,7 @@#include <linux/virtio.h> #include <linux/virtio_config.h> #include <linux/input.h> +#include <linux/slab.h> #include <uapi/linux/virtio_ids.h> #include <uapi/linux/virtio_input.h>
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 9/19 virtio: stop using legacy struct vring in kernel
In reply to Michael S. Tsirkinstruct vring (in the uapi directory) and supporting APIs are kept around to solely avoid breaking old userspace builds. It's not actually part of the UAPI - it was kept in the UAPI header by mistake, and using it in kernel isn't necessary and prevents us from making changes safely.
In particular, the APIs actually assume the legacy layout.
Add an internal kernel-only struct vring and
switch everyone to use that.
--- drivers/block/virtio_blk.c | 1 + include/linux/virtio.h | 1 - include/linux/virtio_ring.h | 10 ++++++++++ include/linux/vringh.h | 1 + include/uapi/linux/virtio_ring.h | 26 ++++++++++++++++---------- 5 files changed, 28 insertions(+), 11 deletions(-)
drivers/block/virtio_blk.c +1 -0
@@ -15,6 +15,7 @@#include <linux/blk-mq.h> #include <linux/blk-mq-virtio.h> #include <linux/numa.h> +#include <uapi/linux/virtio_ring.h> #define PART_BITS 4#define VQ_NAME_LEN 16
include/linux/virtio.h +0 -1
@@ -9,7 +9,6 @@#include <linux/device.h> #include <linux/mod_devicetable.h> #include <linux/gfp.h> -#include <linux/vringh.h> /** * virtqueue - a queue to register buffers for sending or receiving.
include/linux/virtio_ring.h +10 -0
@@ -60,6 +60,16 @@ static inline void virtio_store_mb(bool weak_barriers,struct virtio_device ;struct virtqueue ; +struct vring { +unsigned int num; + +struct vring_desc *desc; + +struct vring_avail *avail; + +struct vring_used *used; +}; +/* * Creates a virtqueue and allocates the descriptor ring. If * may_reduce_num is set, then this may allocate a smaller ring than
include/linux/vringh.h +1 -0
@@ -11,6 +11,7 @@#ifndef _LINUX_VRINGH_H#define _LINUX_VRINGH_H#include <uapi/linux/virtio_ring.h> +#include <linux/virtio_ring.h> #include <linux/virtio_byteorder.h> #include <linux/uio.h> #include <linux/slab.h>
include/uapi/linux/virtio_ring.h +16 -10
@@ -118,16 +118,6 @@ struct vring_used {struct vring_used_elem ring[]; }; -struct vring { -unsigned int num; - -struct vring_desc *desc; - -struct vring_avail *avail; - -struct vring_used *used; -}; -/* Alignment requirements for vring elements. * When using pre-virtio 1.0 layout, these fall out naturally. */
@@ -166,6 +156,21 @@ struct vring {#define vring_used_event (vr) ((vr)->avail->ring[(vr)->num])#define vring_avail_event (vr) (*(__virtio16 *)&(vr)->used->ring[(vr)->num]) +#ifndef __KERNEL__ +/* +* The following definitions have been put in the UAPI header by mistake. We +* keep them around to avoid breaking old userspace builds. +*/ +struct vring { +unsigned int num; + +struct vring_desc *desc; + +struct vring_avail *avail; + +struct vring_used *used; +}; +static inline void vring_init (struct vring *vr,unsigned int num,void *p,unsigned long align) {
@@ -182,6 +187,7 @@ static inline unsigned vring_size(unsigned int num, unsigned long align) + align - 1) & ~(align - 1)) +sizeof (__virtio16) *3 +sizeof (struct vring_used_elem ) * num; } +#endif #endif /* VIRTIO_RING_NO_LEGACY */
Signature · 1 line
MST
Re: [PATCH v8 09/19] virtio: stop using legacy struct vring in kernel
Quoted 10 lines · Michael S. Tsirkin
struct vring (in the uapi directory) and supporting APIs are kept around to solely avoid breaking old userspace builds. It's not actually part of the UAPI - it was kept in the UAPI header by mistake, and using it in kernel isn't necessary and prevents us from making changes safely.
In particular, the APIs actually assume the legacy layout.Add an internal kernel-only struct vring and
switch everyone to use that.
Quoted 8 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/block/virtio_blk.c | 1 + include/linux/virtio.h | 1 - include/linux/virtio_ring.h | 10 ++++++++++ include/linux/vringh.h | 1 + include/uapi/linux/virtio_ring.h | 26 ++++++++++++++++---------- 5 files changed, 28 insertions(+), 11 deletions(-)
Michael S. Tsirkin
maintainerPATCH v8 10/19 vhost: force spec specified alignment on types
In reply to Michael S. TsirkinThe ring element addresses are passed between components with different alignments assumptions. Thus, if guest/userspace selects a pointer and host then gets and dereferences it, we might need to decrease the compiler-selected alignment to prevent compiler on the host from assuming pointer is aligned.
This actually triggers on ARM with -mabi=apcs-gnu - which is a deprecated configuration, but it seems safer to handle this generally.
I verified that the produced binary is exactly identical on x86.
--- drivers/vhost/vhost.h | 6 +++--- include/linux/virtio_ring.h | 24 +++++++++++++++++++++--- 2 files changed, 24 insertions(+), 6 deletions(-)
drivers/vhost/vhost.h +3 -3
@@ -67,9 +67,9 @@ struct vhost_virtqueue {/* The actual ring of buffers. */ struct mutex mutex;unsigned int num; -struct vring_desc __user *desc; -struct vring_avail __user *avail; -struct vring_used __user *used; +vring_desc_t __user *desc; +vring_avail_t __user *avail; +vring_used_t __user *used;const struct vhost_iotlb_map *meta_iotlb[VHOST_NUM_ADDRS ];struct file *kick;struct eventfd_ctx *call_ctx;
include/linux/virtio_ring.h +21 -3
@@ -60,14 +60,32 @@ static inline void virtio_store_mb(bool weak_barriers,struct virtio_device ;struct virtqueue ; +/* +* The ring element addresses are passed between components with different +* alignments assumptions. Thus, we might need to decrease the compiler-selected +* alignment, and so must use a typedef to make sure the __aligned attribute +* actually takes hold: +* +* https://gcc.gnu.org/onlinedocs//gcc/Common-Type-Attributes.html#Common-Type-Attributes +* +* When used on a struct, or struct member, the aligned attribute can only +* increase the alignment; in order to decrease it, the packed attribute must +* be specified as well. When used as part of a typedef, the aligned attribute +* can both increase and decrease alignment, and specifying the packed +* attribute generates a warning. +*/ +typedef struct vring_desc __aligned(VRING_DESC_ALIGN_SIZE) vring_desc_t ; +typedef struct vring_avail __aligned(VRING_AVAIL_ALIGN_SIZE) vring_avail_t ; +typedef struct vring_used __aligned(VRING_USED_ALIGN_SIZE) vring_used_t ; +struct vring {unsigned int num; -struct vring_desc *desc; +vring_desc_t *desc; -struct vring_avail *avail; +vring_avail_t *avail; -struct vring_used *used; +vring_used_t *used; };/*
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 11/19 virtio: add legacy init/size APIs
In reply to Michael S. Tsirkinvring_init/vring_size (in the uapi directory) are kept around to solely avoid breaking old userspace builds. It's not actually part of the UAPI - it was kept in the UAPI header by mistake, and using it in kernel isn't necessary and prevents us from making changes safely.
In particular, the APIs actually assume the legacy layout.
Add an internal kernel-only vring_legacy_init/vring_legacy_size.
--- include/linux/virtio_ring.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
include/linux/virtio_ring.h +18 -0
@@ -140,4 +140,22 @@ void vring_del_virtqueue(struct virtqueue *vq);void vring_transport_features (struct virtio_device *vdev);irqreturn_t vring_interrupt (int irq,void *_vq); + +static inline void vring_legacy_init (struct vring *vr,unsigned int num,void *p, +unsigned long align) +{ + vr->num = num; + vr->desc = p; + vr->avail = (struct vring_avail *)((char *)p + num *sizeof (struct vring_desc )); + vr->used = (void *)(((uintptr_t )&vr->avail->ring[num] +sizeof (__virtio16) + + align-1 ) & ~(align -1 )); +} + +static inline unsigned vring_legacy_size (unsigned int num,unsigned long align) +{ +return ((sizeof (struct vring_desc ) * num +sizeof (__virtio16) * (3 + num) + + align -1 ) & ~(align -1 )) + +sizeof (__virtio16) *3 +sizeof (struct vring_used_elem ) * num; +} +#endif /* _LINUX_VIRTIO_RING_H */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 12/19 virtio_ring: switch to virtio_legacy_init/size
In reply to Michael S. TsirkinThese are used for legacy ring format, switch to APIs that make this explicit.
--- drivers/virtio/virtio_ring.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/virtio/virtio_ring.c +9 -6
@@ -867,8 +867,10 @@ static struct virtqueue *vring_create_virtqueue_split( }/* TODO: allocate each queue chunk individually */ -for (; num &&vring_size (num, vring_align) >PAGE_SIZE ; num /=2 ) { - queue =vring_alloc_queue (vdev,vring_size (num, vring_align), +for (; num &&vring_legacy_size (num, vring_align) >PAGE_SIZE ; + num /=2 ) { + queue =vring_alloc_queue (vdev, +vring_legacy_size (num, vring_align), &dma_addr,GFP_KERNEL |__GFP_NOWARN|__GFP_ZERO);if (queue)
@@ -882,14 +884,15 @@ static struct virtqueue *vring_create_virtqueue_split(if (!queue) {/* Try to get a single page. You are my only hope! */ - queue =vring_alloc_queue (vdev,vring_size (num, vring_align), + queue =vring_alloc_queue (vdev, +vring_legacy_size (num, vring_align), &dma_addr,GFP_KERNEL |__GFP_ZERO); }if (!queue)return NULL ; - queue_size_in_bytes =vring_size (num, vring_align); -vring_init (&vring, num, queue, vring_align); + queue_size_in_bytes =vring_legacy_size (num, vring_align); +vring_legacy_init (&vring, num, queue, vring_align); vq =__vring_new_virtqueue (index, vring, vdev, weak_barriers, context, notify, callback, name);
@@ -2169,7 +2172,7 @@ struct virtqueue *vring_new_virtqueue(unsigned int index,if (virtio_has_feature (vdev,VIRTIO_F_RING_PACKED ))return NULL ; -vring_init (&vring, num, pages, vring_align); +vring_legacy_init (&vring, num, pages, vring_align);return __vring_new_virtqueue (index, vring, vdev, weak_barriers, context, notify, callback, name); }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 13/19 tools/virtio: switch to virtio_legacy_init/size
In reply to Michael S. TsirkinThese are used for legacy ring format, switch to APIs that make this explicit.
--- tools/virtio/ringtest/virtio_ring_0_9.c | 6 +++--- tools/virtio/virtio_test.c | 6 +++--- tools/virtio/vringh_test.c | 18 +++++++++--------- 3 files changed, 15 insertions(+), 15 deletions(-)
tools/virtio/ringtest/virtio_ring_0_9.c +3 -3
@@ -67,13 +67,13 @@ void alloc_ring(void)int i;void *p; - ret =posix_memalign (&p,0x1000 ,vring_size (ring_size,0x1000 )); + ret =posix_memalign (&p,0x1000 ,vring_legacy_size (ring_size,0x1000 ));if (ret) {perror ("Unable to allocate ring buffer.\n" );exit (3 ); } -memset (p,0 ,vring_size (ring_size,0x1000 )); -vring_init (&ring, ring_size, p,0x1000 ); +memset (p,0 ,vring_legacy_size (ring_size,0x1000 )); +vring_legacy_init (&ring, ring_size, p,0x1000 ); guest.avail_idx =0 ; guest.kicked_avail_idx =-1 ;
tools/virtio/virtio_test.c +3 -3
@@ -102,10 +102,10 @@ static void vq_info_add(struct vdev_info *dev, int num) info->idx = dev->nvqs; info->kick =eventfd (0 ,EFD_NONBLOCK ); info->call =eventfd (0 ,EFD_NONBLOCK ); - r =posix_memalign (&info->ring,4096 ,vring_size (num,4096 )); + r =posix_memalign (&info->ring,4096 ,vring_legacy_size (num,4096 ));assert (r >=0 ); -memset (info->ring,0 ,vring_size (num,4096 )); -vring_init (&info->vring, num, info->ring,4096 ); +memset (info->ring,0 ,vring_legacy_size (num,4096 )); +vring_legacy_init (&info->vring, num, info->ring,4096 ); info->vq = vring_new_virtqueue(info->idx, info->vring.num, 4096, &dev->vdev, true, false, info->ring,
tools/virtio/vringh_test.c +9 -9
@@ -151,7 +151,7 @@ static int parallel_test(u64 features,err (1 ,"Opening /tmp/vringh_test-file" );/* Extra room at the end for some data, and indirects */ - mapsize =vring_size (RINGSIZE ,ALIGN ) + mapsize =vring_legacy_size (RINGSIZE ,ALIGN ) +RINGSIZE *2 *sizeof (int ) +RINGSIZE *6 *sizeof (struct vring_desc ); mapsize = (mapsize +getpagesize () -1 ) & ~(getpagesize () -1 );
@@ -185,7 +185,7 @@ static int parallel_test(u64 features,close (to_guest[0 ]);close (to_host[1 ]); -vring_init (&vrh.vring,RINGSIZE , host_map,ALIGN ); +vring_legacy_init (&vrh.vring,RINGSIZE , host_map,ALIGN );vringh_init_user (&vrh, features,RINGSIZE , true, vrh.vring.desc, vrh.vring.avail, vrh.vring.used);CPU_SET (first_cpu, &cpu_set);
@@ -297,7 +297,7 @@ static int parallel_test(u64 features,unsigned int finished =0 ;/* We pass sg[]s pointing into here, but we need RINGSIZE+1 */ - data = guest_map +vring_size (RINGSIZE ,ALIGN ); + data = guest_map +vring_legacy_size (RINGSIZE ,ALIGN ); indirects = (void *)data + (RINGSIZE +1 ) *2 *sizeof (int );/* We are the guest. */
@@ -478,7 +478,7 @@ int main(int argc, char *argv[])if (posix_memalign (&__user_addr_min,PAGE_SIZE ,USER_MEM ) !=0 )abort (); __user_addr_max = __user_addr_min +USER_MEM ; -memset (__user_addr_min,0 ,vring_size (RINGSIZE ,ALIGN )); +memset (__user_addr_min,0 ,vring_legacy_size (RINGSIZE ,ALIGN ));/* Set up guest side. */ vq = vring_new_virtqueue(0, RINGSIZE, ALIGN, &vdev, true, false,
@@ -487,7 +487,7 @@ int main(int argc, char *argv[]) "guest vq");/* Set up host side. */ -vring_init (&vrh.vring,RINGSIZE , __user_addr_min,ALIGN ); +vring_legacy_init (&vrh.vring,RINGSIZE , __user_addr_min,ALIGN );vringh_init_user (&vrh, vdev.features,RINGSIZE , true, vrh.vring.desc, vrh.vring.avail, vrh.vring.used);
@@ -506,7 +506,7 @@ int main(int argc, char *argv[]) sgs[1 ] = &guest_sg[1 ];/* May allocate an indirect, so force it to allocate user addr */ - __kmalloc_fake = __user_addr_min +vring_size (RINGSIZE ,ALIGN ); + __kmalloc_fake = __user_addr_min +vring_legacy_size (RINGSIZE ,ALIGN ); err =virtqueue_add_sgs (vq, sgs,1 ,1 , &err,GFP_KERNEL );if (err)errx (1 ,"virtqueue_add_sgs: %i" , err);
@@ -556,7 +556,7 @@ int main(int argc, char *argv[])errx (1 ,"vringh_complete_user: %i" , err);/* Guest should see used token now. */ - __kfree_ignore_start = __user_addr_min +vring_size (RINGSIZE ,ALIGN ); + __kfree_ignore_start = __user_addr_min +vring_legacy_size (RINGSIZE ,ALIGN ); __kfree_ignore_end = __kfree_ignore_start +1 ; ret =virtqueue_get_buf (vq, &i);if (ret != &err)
@@ -575,7 +575,7 @@ int main(int argc, char *argv[]) ((char *)__user_addr_max -USER_MEM /4 )[i] = i;/* This will allocate an indirect, so force it to allocate user addr */ - __kmalloc_fake = __user_addr_min +vring_size (RINGSIZE ,ALIGN ); + __kmalloc_fake = __user_addr_min +vring_legacy_size (RINGSIZE ,ALIGN ); err =virtqueue_add_outbuf (vq, guest_sg,RINGSIZE , &err,GFP_KERNEL );if (err)errx (1 ,"virtqueue_add_outbuf (large): %i" , err);
@@ -680,7 +680,7 @@ int main(int argc, char *argv[])if (err)errx (1 ,"virtqueue_add_outbuf (indirect): %i" , err); -vring_init (&vring,RINGSIZE , __user_addr_min,ALIGN ); +vring_legacy_init (&vring,RINGSIZE , __user_addr_min,ALIGN );/* They're used in order, but double-check... */ assert (vring.desc[0 ].addr == (unsigned long )d);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 14/19 vop: switch to virtio_legacy_init/size
In reply to Michael S. TsirkinThese are used for legacy ring format, switch to APIs that make this explicit.
--- drivers/misc/mic/vop/vop_main.c | 5 +++-- drivers/misc/mic/vop/vop_vringh.c | 8 +++++--- 2 files changed, 8 insertions(+), 5 deletions(-)
drivers/misc/mic/vop/vop_main.c +3 -2
@@ -283,7 +283,7 @@ static struct virtqueue *vop_new_virtqueue(unsigned int index,bool weak_barriers = false;struct vring vring; -vring_init (&vring, num, pages,MIC_VIRTIO_RING_ALIGN ); +vring_legacy_init (&vring, num, pages,MIC_VIRTIO_RING_ALIGN ); vring.used = used; return __vring_new_virtqueue(index, vring, vdev, weak_barriers, context,
@@ -320,7 +320,8 @@ static struct virtqueue *vop_find_vq(struct virtio_device *dev,/* First assign the vring's allocated in host memory */ vqconfig =_vop_vq_config (vdev->desc) + index;memcpy_fromio (&config, vqconfig,sizeof (config)); - _vr_size =vring_size (le16_to_cpu (config.num),MIC_VIRTIO_RING_ALIGN ); + _vr_size =vring_legacy_size (le16_to_cpu (config.num), +MIC_VIRTIO_RING_ALIGN ); vr_size =PAGE_ALIGN (_vr_size +sizeof (struct _mic_vring_info )); va = vpdev->hw_ops->remap (vpdev,le64_to_cpu (config.address), vr_size);if (!va)
drivers/misc/mic/vop/vop_vringh.c +5 -3
@@ -296,7 +296,8 @@ static int vop_virtio_add_device(struct vop_vdev *vdev, num =le16_to_cpu (vqconfig[i].num);mutex_init (&vvr->vr_mutex); - vr_size =PAGE_ALIGN (vring_size (num,MIC_VIRTIO_RING_ALIGN ) + + vr_size =PAGE_ALIGN (vring_legacy_size (num, +MIC_VIRTIO_RING_ALIGN ) +sizeof (struct _mic_vring_info )); vr->va = (void *) __get_free_pages(GFP_KERNEL | __GFP_ZERO,
@@ -308,7 +309,8 @@ static int vop_virtio_add_device(struct vop_vdev *vdev, goto err; } vr->len = vr_size; - vr->info = vr->va +vring_size (num,MIC_VIRTIO_RING_ALIGN ); + vr->info = vr->va +vring_legacy_size (num, +MIC_VIRTIO_RING_ALIGN ); vr->info->magic =cpu_to_le32 (MIC_MAGIC + vdev->virtio_id + i); vr_addr =dma_map_single (&vpdev->dev, vr->va, vr_size,DMA_BIDIRECTIONAL );
@@ -321,7 +323,7 @@ static int vop_virtio_add_device(struct vop_vdev *vdev, } vqconfig[i].address =cpu_to_le64 (vr_addr); -vring_init (&vr->vr, num, vr->va,MIC_VIRTIO_RING_ALIGN ); +vring_legacy_init (&vr->vr, num, vr->va,MIC_VIRTIO_RING_ALIGN ); ret = vringh_init_kern(&vvr->vrh, *(u32 *)mic_vq_features(vdev->dd), num, false, vr->vr.desc, vr->vr.avail,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 15/19 remoteproc: switch to virtio_legacy_init/size
In reply to Michael S. TsirkinThese are used for legacy ring format, switch to APIs that make this explicit.
--- drivers/remoteproc/remoteproc_core.c | 2 +- drivers/remoteproc/remoteproc_virtio.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +1 -1
@@ -322,7 +322,7 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)struct rproc_mem_entry *mem;/* actual size of vring (in bytes) */ - size =PAGE_ALIGN (vring_size (rvring->len, rvring->align)); + size =PAGE_ALIGN (vring_legacy_size (rvring->len, rvring->align)); rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
drivers/remoteproc/remoteproc_virtio.c +1 -1
@@ -95,7 +95,7 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev, len = rvring->len;/* zero vring */ - size =vring_size (len, rvring->align); + size =vring_legacy_size (len, rvring->align);memset (addr,0 , size); dev_dbg(dev, "vring%d: va %pK qsz %d notifyid %d\n",
Signature · 1 line
MST
Re: [PATCH v8 15/19] remoteproc: switch to virtio_legacy_init/size
Quoted 4 lines · Michael S. Tsirkin
These are used for legacy ring format, switch to APIs that make this explicit.
Signed-off-by Michael S. Tsirkin <[email protected]>
Regards,
Bjorn
Quoted 34 lines
--- drivers/remoteproc/remoteproc_core.c | 2 +- drivers/remoteproc/remoteproc_virtio.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)drivers/remoteproc/remoteproc_core.c +1
-1@@ -322,7 +322,7 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)struct rproc_mem_entry *mem;/* actual size of vring (in bytes) */ - size =PAGE_ALIGN (vring_size (rvring->len, rvring->align)); + size =PAGE_ALIGN (vring_legacy_size (rvring->len, rvring->align)); rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;drivers/remoteproc/remoteproc_virtio.c +1
-1@@ -95,7 +95,7 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev, len = rvring->len;/* zero vring */ - size =vring_size (len, rvring->align); + size =vring_legacy_size (len, rvring->align);memset (addr,0 , size); dev_dbg(dev, "vring%d: va %pK qsz %d notifyid %d\n",Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 16/19 mellanox: switch to virtio_legacy_init/size
In reply to Michael S. TsirkinThese are used for legacy ring format, switch to APIs that make this explicit.
--- drivers/platform/mellanox/mlxbf-tmfifo.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/platform/mellanox/mlxbf-tmfifo.c +3 -3
@@ -214,7 +214,7 @@ static void mlxbf_tmfifo_free_vrings(struct mlxbf_tmfifo *fifo,for (i =0 ; i <ARRAY_SIZE (tm_vdev->vrings); i++) { vring = &tm_vdev->vrings[i];if (vring->va) { - size =vring_size (vring->num, vring->align); + size =vring_legacy_size (vring->num, vring->align);dma_free_coherent (tm_vdev->vdev.dev.parent, size, vring->va, vring->dma); vring->va =NULL ;
@@ -245,7 +245,7 @@ static int mlxbf_tmfifo_alloc_vrings(struct mlxbf_tmfifo *fifo, vring->vdev_id = tm_vdev->vdev.id.device; dev = &tm_vdev->vdev.dev; - size =vring_size (vring->num, vring->align); + size =vring_legacy_size (vring->num, vring->align); va =dma_alloc_coherent (dev->parent, size, &dma,GFP_KERNEL );if (!va) {mlxbf_tmfifo_free_vrings (fifo, tm_vdev);
@@ -935,7 +935,7 @@ static int mlxbf_tmfifo_virtio_find_vqs(struct virtio_device *vdev, vring = &tm_vdev->vrings[i];/* zero vring */ - size =vring_size (vring->num, vring->align); + size =vring_legacy_size (vring->num, vring->align);memset (vring->va,0 , size); vq = vring_new_virtqueue(i, vring->num, vring->align, vdev, false, false, vring->va,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 17/19 vhost: option to fetch descriptors through an independent struct
In reply to Michael S. TsirkinThe idea is to support multiple ring formats by converting to a format-independent array of descriptors.
This costs extra cycles, but we gain in ability
to fetch a batch of descriptors in one go, which
is good for code cache locality.
When used, this causes a minor performance degradation, it's been kept as simple as possible for ease of review. A follow-up patch gets us back the performance by adding batching.
To simplify benchmarking, I kept the old code around so one can switch back and forth between old and new code. This will go away in the final submission.
--- drivers/vhost/vhost.c | 297 +++++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 16 +++ 2 files changed, 312 insertions(+), 1 deletion(-)
drivers/vhost/vhost.c +296 -1
@@ -298,6 +298,7 @@ static void vhost_vq_reset(struct vhost_dev *dev, struct vhost_virtqueue *vq) { vq->num =1 ; + vq->ndescs =0 ; vq->desc =NULL ; vq->avail =NULL ; vq->used =NULL ;
@@ -368,6 +369,9 @@ static int vhost_worker(void *data)static void vhost_vq_free_iovecs (struct vhost_virtqueue *vq) { +kfree (vq->descs); + vq->descs =NULL ; + vq->max_descs =0 ;kfree (vq->indirect); vq->indirect =NULL ;kfree (vq->log);
@@ -384,6 +388,10 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev)for (i =0 ; i < dev->nvqs; ++i) { vq = dev->vqs[i]; + vq->max_descs = dev->iov_limit; + vq->descs =kmalloc_array (vq->max_descs, +sizeof (*vq->descs), +GFP_KERNEL ); vq->indirect =kmalloc_array (UIO_MAXIOV ,sizeof (*vq->indirect),GFP_KERNEL );
@@ -391,7 +399,7 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev) GFP_KERNEL); vq->heads =kmalloc_array (dev->iov_limit,sizeof (*vq->heads),GFP_KERNEL ); -if (!vq->indirect || !vq->log || !vq->heads) +if (!vq->indirect || !vq->log || !vq->heads || !vq->descs) goto err_nomem; }return 0 ;
@@ -2277,6 +2285,293 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, }EXPORT_SYMBOL_GPL (vhost_get_vq_desc); +static struct vhost_desc *peek_split_desc (struct vhost_virtqueue *vq) +{ +BUG_ON (!vq->ndescs); +return &vq->descs[vq->ndescs -1 ]; +} + +static void pop_split_desc (struct vhost_virtqueue *vq) +{ +BUG_ON (!vq->ndescs); + --vq->ndescs; +} + +#define VHOST_DESC_FLAGS (VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE | \ + VRING_DESC_F_NEXT) +static int push_split_desc (struct vhost_virtqueue *vq,struct vring_desc *desc,u16 id) +{ +struct vhost_desc *h; + +if (unlikely (vq->ndescs >= vq->max_descs)) +return -EINVAL ; + h = &vq->descs[vq->ndescs++]; + h->addr =vhost64_to_cpu (vq, desc->addr); + h->len =vhost32_to_cpu (vq, desc->len); + h->flags =vhost16_to_cpu (vq, desc->flags) &VHOST_DESC_FLAGS ; + h->id = id; + +return 0 ; +} + +static int fetch_indirect_descs (struct vhost_virtqueue *vq, +struct vhost_desc *indirect, +u16 head) +{ +struct vring_desc desc; +unsigned int i =0 , count, found =0 ; +u32 len = indirect->len; +struct iov_iter from; +int ret; + +/* Sanity check */ +if (unlikely (len %sizeof desc)) { +vq_err (vq,"Invalid length in indirect descriptor: " +"len 0x%llx not multiple of 0x%zx\n" , + (unsigned long long )len, +sizeof desc); +return -EINVAL ; + } + + ret =translate_desc (vq, indirect->addr, len, vq->indirect, +UIO_MAXIOV ,VHOST_ACCESS_RO ); +if (unlikely (ret <0 )) { +if (ret != -EAGAIN ) +vq_err (vq,"Translation failure %d in indirect.\n" , ret); +return ret; + } +iov_iter_init (&from,READ , vq->indirect, ret, len); + +/* We will use the result as an address to read from, so most +* architectures only need a compiler barrier here. */ +read_barrier_depends (); + + count = len /sizeof desc; +/* Buffers are chained via a 16 bit next field, so +* we can have at most 2^16 of these. */ +if (unlikely (count >USHRT_MAX +1 )) { +vq_err (vq,"Indirect buffer length too big: %d\n" , + indirect->len); +return -E2BIG ; + } +if (unlikely (vq->ndescs + count > vq->max_descs)) { +vq_err (vq,"Too many indirect + direct descs: %d + %d\n" , + vq->ndescs, indirect->len); +return -E2BIG ; + } + +do { +if (unlikely (++found > count)) { +vq_err (vq,"Loop detected: last one at %u " +"indirect size %u\n" , + i, count); +return -EINVAL ; + } +if (unlikely (!copy_from_iter_full (&desc,sizeof (desc), &from))) { +vq_err (vq,"Failed indirect descriptor: idx %d, %zx\n" , + i, (size_t )indirect->addr + i *sizeof desc); +return -EINVAL ; + } +if (unlikely (desc.flags &cpu_to_vhost16 (vq,VRING_DESC_F_INDIRECT ))) { +vq_err (vq,"Nested indirect descriptor: idx %d, %zx\n" , + i, (size_t )indirect->addr + i *sizeof desc); +return -EINVAL ; + } + +push_split_desc (vq, &desc, head); + }while ((i =next_desc (vq, &desc)) !=-1 ); +return 0 ; +} + +static int fetch_descs (struct vhost_virtqueue *vq) +{ +unsigned int i, head, found =0 ; +struct vhost_desc *last; +struct vring_desc desc; +__virtio16 avail_idx; +__virtio16 ring_head; +u16 last_avail_idx; +int ret; + +/* Check it isn't doing very strange things with descriptor numbers. */ + last_avail_idx = vq->last_avail_idx; + +if (vq->avail_idx == vq->last_avail_idx) { +if (unlikely (vhost_get_avail_idx (vq, &avail_idx))) { +vq_err (vq,"Failed to access avail idx at %p\n" , + &vq->avail->idx); +return -EFAULT ; + } + vq->avail_idx =vhost16_to_cpu (vq, avail_idx); + +if (unlikely ((u16)(vq->avail_idx - last_avail_idx) > vq->num)) { +vq_err (vq,"Guest moved used index from %u to %u" , + last_avail_idx, vq->avail_idx); +return -EFAULT ; + } + +/* If there's nothing new since last we looked, return +* invalid. +*/ +if (vq->avail_idx == last_avail_idx) +return vq->num; + +/* Only get avail ring entries after they have been +* exposed by guest. +*/ +smp_rmb (); + } + +/* Grab the next descriptor number they're advertising */ +if (unlikely (vhost_get_avail_head (vq, &ring_head, last_avail_idx))) { +vq_err (vq,"Failed to read head: idx %d address %p\n" , + last_avail_idx, + &vq->avail->ring[last_avail_idx % vq->num]); +return -EFAULT ; + } + + head =vhost16_to_cpu (vq, ring_head); + +/* If their number is silly, that's an error. */ +if (unlikely (head >= vq->num)) { +vq_err (vq,"Guest says index %u > %u is available" , + head, vq->num); +return -EINVAL ; + } + + i = head; +do { +if (unlikely (i >= vq->num)) { +vq_err (vq,"Desc index is %u > %u, head = %u" , + i, vq->num, head); +return -EINVAL ; + } +if (unlikely (++found > vq->num)) { +vq_err (vq,"Loop detected: last one at %u " +"vq size %u head %u\n" , + i, vq->num, head); +return -EINVAL ; + } + ret =vhost_get_desc (vq, &desc, i); +if (unlikely (ret)) { +vq_err (vq,"Failed to get descriptor: idx %d addr %p\n" , + i, vq->desc + i); +return -EFAULT ; + } + ret =push_split_desc (vq, &desc, head); +if (unlikely (ret)) { +vq_err (vq,"Failed to save descriptor: idx %d\n" , i); +return -EINVAL ; + } + }while ((i =next_desc (vq, &desc)) !=-1 ); + + last =peek_split_desc (vq); +if (unlikely (last->flags &VRING_DESC_F_INDIRECT )) { +pop_split_desc (vq); + ret =fetch_indirect_descs (vq, last, head); +if (unlikely (ret <0 )) { +if (ret != -EAGAIN ) +vq_err (vq,"Failure detected " +"in indirect descriptor at idx %d\n" , head); +return ret; + } + } + +/* Assume notifications from guest are disabled at this point, +* if they aren't we would need to update avail_event index. */ +BUG_ON (!(vq->used_flags &VRING_USED_F_NO_NOTIFY )); + +/* On success, increment avail index. */ + vq->last_avail_idx++; + +return 0 ; +} + +/* This looks in the virtqueue and for the first available buffer, and converts +* it to an iovec for convenient access. Since descriptors consist of some +* number of output then some number of input descriptors, it's actually two +* iovecs, but we pack them into one and note how many of each there were. +* +* This function returns the descriptor number found, or vq->num (which is +* never a valid descriptor number) if none was found. A negative code is +* returned on error. */ +int vhost_get_vq_desc_batch (struct vhost_virtqueue *vq, +struct iovec iov[],unsigned int iov_size, +unsigned int *out_num,unsigned int *in_num, +struct vhost_log *log,unsigned int *log_num) +{ +int ret =fetch_descs (vq); +int i; + +if (ret) +return ret; + +/* Now convert to IOV */ +/* When we start there are none of either input nor output. */ + *out_num = *in_num =0 ; +if (unlikely (log)) + *log_num =0 ; + +for (i =0 ; i < vq->ndescs; ++i) { +unsigned iov_count = *in_num + *out_num; +struct vhost_desc *desc = &vq->descs[i]; +int access; + +if (desc->flags & ~VHOST_DESC_FLAGS ) { +vq_err (vq,"Unexpected flags: 0x%x at descriptor id 0x%x\n" , + desc->flags, desc->id); + ret = -EINVAL ; + goto err; + } +if (desc->flags &VRING_DESC_F_WRITE ) + access =VHOST_ACCESS_WO ; +else + access =VHOST_ACCESS_RO ; + ret =translate_desc (vq, desc->addr, + desc->len, iov + iov_count, + iov_size - iov_count, access); +if (unlikely (ret <0 )) { +if (ret != -EAGAIN ) +vq_err (vq,"Translation failure %d descriptor idx %d\n" , + ret, i); + goto err; + } +if (access ==VHOST_ACCESS_WO ) { +/* If this is an input descriptor, +* increment that count. */ + *in_num += ret; +if (unlikely (log && ret)) { + log[*log_num].addr = desc->addr; + log[*log_num].len = desc->len; + ++*log_num; + } + }else { +/* If it's an output descriptor, they're all supposed +* to come before any input descriptors. */ +if (unlikely (*in_num)) { +vq_err (vq,"Descriptor has out after in: " +"idx %d\n" , i); + ret = -EINVAL ; + goto err; + } + *out_num += ret; + } + + ret = desc->id; + } + + vq->ndescs =0 ; + +return ret; + +err: +vhost_discard_vq_desc (vq,1 ); + vq->ndescs =0 ; + +return ret; +} +EXPORT_SYMBOL_GPL (vhost_get_vq_desc_batch); +/* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */ void vhost_discard_vq_desc (struct vhost_virtqueue *vq,int n) {
drivers/vhost/vhost.h +16 -0
@@ -60,6 +60,13 @@ enum vhost_uaddr_type { VHOST_NUM_ADDRS = 3, }; +struct vhost_desc { +u64 addr; +u32 len; +u16 flags;/* VRING_DESC_F_WRITE, VRING_DESC_F_NEXT */ +u16 id; +}; +/* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue {struct vhost_dev *dev;
@@ -71,6 +78,11 @@ struct vhost_virtqueue {vring_avail_t __user *avail;vring_used_t __user *used;const struct vhost_iotlb_map *meta_iotlb[VHOST_NUM_ADDRS ]; + +struct vhost_desc *descs; +int ndescs; +int max_descs; +struct file *kick;struct eventfd_ctx *call_ctx;struct eventfd_ctx *error_ctx;
@@ -175,6 +187,10 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *argbool vhost_vq_access_ok (struct vhost_virtqueue *vq);bool vhost_log_access_ok (struct vhost_dev *); +int vhost_get_vq_desc_batch (struct vhost_virtqueue *, +struct iovec iov[],unsigned int iov_count, +unsigned int *out_num,unsigned int *in_num, +struct vhost_log *log,unsigned int *log_num); int vhost_get_vq_desc(struct vhost_virtqueue *, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num,
Signature · 6 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerPATCH v8 18/19 vhost: use batched version by default
In reply to Michael S. TsirkinAs testing shows no performance change, switch to that now.
--- drivers/vhost/vhost.c | 251 +----------------------------------------- drivers/vhost/vhost.h | 4 - 2 files changed, 2 insertions(+), 253 deletions(-)
drivers/vhost/vhost.c +2 -249
@@ -2038,253 +2038,6 @@ static unsigned next_desc(struct vhost_virtqueue *vq, struct vring_desc *desc)return next; } -static int get_indirect (struct vhost_virtqueue *vq, -struct iovec iov[],unsigned int iov_size, -unsigned int *out_num,unsigned int *in_num, -struct vhost_log *log,unsigned int *log_num, -struct vring_desc *indirect) -{ -struct vring_desc desc; -unsigned int i =0 , count, found =0 ; -u32 len =vhost32_to_cpu (vq, indirect->len); -struct iov_iter from; -int ret, access; - -/* Sanity check */ -if (unlikely (len %sizeof desc)) { -vq_err (vq,"Invalid length in indirect descriptor: " -"len 0x%llx not multiple of 0x%zx\n" , - (unsigned long long )len, -sizeof desc); -return -EINVAL ; - } - - ret =translate_desc (vq,vhost64_to_cpu (vq, indirect->addr), len, vq->indirect, -UIO_MAXIOV ,VHOST_ACCESS_RO ); -if (unlikely (ret <0 )) { -if (ret != -EAGAIN ) -vq_err (vq,"Translation failure %d in indirect.\n" , ret); -return ret; - } -iov_iter_init (&from,READ , vq->indirect, ret, len); - -/* We will use the result as an address to read from, so most -* architectures only need a compiler barrier here. */ -read_barrier_depends (); - - count = len /sizeof desc; -/* Buffers are chained via a 16 bit next field, so -* we can have at most 2^16 of these. */ -if (unlikely (count >USHRT_MAX +1 )) { -vq_err (vq,"Indirect buffer length too big: %d\n" , - indirect->len); -return -E2BIG ; - } - -do { -unsigned iov_count = *in_num + *out_num; -if (unlikely (++found > count)) { -vq_err (vq,"Loop detected: last one at %u " -"indirect size %u\n" , - i, count); -return -EINVAL ; - } -if (unlikely (!copy_from_iter_full (&desc,sizeof (desc), &from))) { -vq_err (vq,"Failed indirect descriptor: idx %d, %zx\n" , - i, (size_t )vhost64_to_cpu (vq, indirect->addr) + i *sizeof desc); -return -EINVAL ; - } -if (unlikely (desc.flags &cpu_to_vhost16 (vq,VRING_DESC_F_INDIRECT ))) { -vq_err (vq,"Nested indirect descriptor: idx %d, %zx\n" , - i, (size_t )vhost64_to_cpu (vq, indirect->addr) + i *sizeof desc); -return -EINVAL ; - } - -if (desc.flags &cpu_to_vhost16 (vq,VRING_DESC_F_WRITE )) - access =VHOST_ACCESS_WO ; -else - access =VHOST_ACCESS_RO ; - - ret =translate_desc (vq,vhost64_to_cpu (vq, desc.addr), -vhost32_to_cpu (vq, desc.len), iov + iov_count, - iov_size - iov_count, access); -if (unlikely (ret <0 )) { -if (ret != -EAGAIN ) -vq_err (vq,"Translation failure %d indirect idx %d\n" , - ret, i); -return ret; - } -/* If this is an input descriptor, increment that count. */ -if (access ==VHOST_ACCESS_WO ) { - *in_num += ret; -if (unlikely (log && ret)) { - log[*log_num].addr =vhost64_to_cpu (vq, desc.addr); - log[*log_num].len =vhost32_to_cpu (vq, desc.len); - ++*log_num; - } - }else { -/* If it's an output descriptor, they're all supposed -* to come before any input descriptors. */ -if (unlikely (*in_num)) { -vq_err (vq,"Indirect descriptor " -"has out after in: idx %d\n" , i); -return -EINVAL ; - } - *out_num += ret; - } - }while ((i =next_desc (vq, &desc)) !=-1 ); -return 0 ; -} - -/* This looks in the virtqueue and for the first available buffer, and converts -* it to an iovec for convenient access. Since descriptors consist of some -* number of output then some number of input descriptors, it's actually two -* iovecs, but we pack them into one and note how many of each there were. -* -* This function returns the descriptor number found, or vq->num (which is -* never a valid descriptor number) if none was found. A negative code is -* returned on error. */ -int vhost_get_vq_desc (struct vhost_virtqueue *vq, -struct iovec iov[],unsigned int iov_size, -unsigned int *out_num,unsigned int *in_num, -struct vhost_log *log,unsigned int *log_num) -{ -struct vring_desc desc; -unsigned int i, head, found =0 ; -u16 last_avail_idx; -__virtio16 avail_idx; -__virtio16 ring_head; -int ret, access; - -/* Check it isn't doing very strange things with descriptor numbers. */ - last_avail_idx = vq->last_avail_idx; - -if (vq->avail_idx == vq->last_avail_idx) { -if (unlikely (vhost_get_avail_idx (vq, &avail_idx))) { -vq_err (vq,"Failed to access avail idx at %p\n" , - &vq->avail->idx); -return -EFAULT ; - } - vq->avail_idx =vhost16_to_cpu (vq, avail_idx); - -if (unlikely ((u16)(vq->avail_idx - last_avail_idx) > vq->num)) { -vq_err (vq,"Guest moved used index from %u to %u" , - last_avail_idx, vq->avail_idx); -return -EFAULT ; - } - -/* If there's nothing new since last we looked, return -* invalid. -*/ -if (vq->avail_idx == last_avail_idx) -return vq->num; - -/* Only get avail ring entries after they have been -* exposed by guest. -*/ -smp_rmb (); - } - -/* Grab the next descriptor number they're advertising, and increment -* the index we've seen. */ -if (unlikely (vhost_get_avail_head (vq, &ring_head, last_avail_idx))) { -vq_err (vq,"Failed to read head: idx %d address %p\n" , - last_avail_idx, - &vq->avail->ring[last_avail_idx % vq->num]); -return -EFAULT ; - } - - head =vhost16_to_cpu (vq, ring_head); - -/* If their number is silly, that's an error. */ -if (unlikely (head >= vq->num)) { -vq_err (vq,"Guest says index %u > %u is available" , - head, vq->num); -return -EINVAL ; - } - -/* When we start there are none of either input nor output. */ - *out_num = *in_num =0 ; -if (unlikely (log)) - *log_num =0 ; - - i = head; -do { -unsigned iov_count = *in_num + *out_num; -if (unlikely (i >= vq->num)) { -vq_err (vq,"Desc index is %u > %u, head = %u" , - i, vq->num, head); -return -EINVAL ; - } -if (unlikely (++found > vq->num)) { -vq_err (vq,"Loop detected: last one at %u " -"vq size %u head %u\n" , - i, vq->num, head); -return -EINVAL ; - } - ret =vhost_get_desc (vq, &desc, i); -if (unlikely (ret)) { -vq_err (vq,"Failed to get descriptor: idx %d addr %p\n" , - i, vq->desc + i); -return -EFAULT ; - } -if (desc.flags &cpu_to_vhost16 (vq,VRING_DESC_F_INDIRECT )) { - ret =get_indirect (vq, iov, iov_size, - out_num, in_num, - log, log_num, &desc); -if (unlikely (ret <0 )) { -if (ret != -EAGAIN ) -vq_err (vq,"Failure detected " -"in indirect descriptor at idx %d\n" , i); -return ret; - } -continue ; - } - -if (desc.flags &cpu_to_vhost16 (vq,VRING_DESC_F_WRITE )) - access =VHOST_ACCESS_WO ; -else - access =VHOST_ACCESS_RO ; - ret =translate_desc (vq,vhost64_to_cpu (vq, desc.addr), -vhost32_to_cpu (vq, desc.len), iov + iov_count, - iov_size - iov_count, access); -if (unlikely (ret <0 )) { -if (ret != -EAGAIN ) -vq_err (vq,"Translation failure %d descriptor idx %d\n" , - ret, i); -return ret; - } -if (access ==VHOST_ACCESS_WO ) { -/* If this is an input descriptor, -* increment that count. */ - *in_num += ret; -if (unlikely (log && ret)) { - log[*log_num].addr =vhost64_to_cpu (vq, desc.addr); - log[*log_num].len =vhost32_to_cpu (vq, desc.len); - ++*log_num; - } - }else { -/* If it's an output descriptor, they're all supposed -* to come before any input descriptors. */ -if (unlikely (*in_num)) { -vq_err (vq,"Descriptor has out after in: " -"idx %d\n" , i); -return -EINVAL ; - } - *out_num += ret; - } - }while ((i =next_desc (vq, &desc)) !=-1 ); - -/* On success, increment avail index. */ - vq->last_avail_idx++; - -/* Assume notifications from guest are disabled at this point, -* if they aren't we would need to update avail_event index. */ -BUG_ON (!(vq->used_flags &VRING_USED_F_NO_NOTIFY )); -return head; -} -EXPORT_SYMBOL_GPL (vhost_get_vq_desc); -static struct vhost_desc *peek_split_desc (struct vhost_virtqueue *vq) {BUG_ON (!vq->ndescs);
@@ -2495,7 +2248,7 @@ static int fetch_descs(struct vhost_virtqueue *vq)* This function returns the descriptor number found, or vq->num (which is * never a valid descriptor number) if none was found. A negative code is * returned on error. */ -int vhost_get_vq_desc_batch (struct vhost_virtqueue *vq, +int vhost_get_vq_desc (struct vhost_virtqueue *vq,struct iovec iov[],unsigned int iov_size,unsigned int *out_num,unsigned int *in_num,struct vhost_log *log,unsigned int *log_num)
@@ -2570,7 +2323,7 @@ int vhost_get_vq_desc_batch(struct vhost_virtqueue *vq,return ret; } -EXPORT_SYMBOL_GPL (vhost_get_vq_desc_batch); +EXPORT_SYMBOL_GPL (vhost_get_vq_desc);/* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */ void vhost_discard_vq_desc (struct vhost_virtqueue *vq,int n)
drivers/vhost/vhost.h +0 -4
@@ -187,10 +187,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *argbool vhost_vq_access_ok (struct vhost_virtqueue *vq);bool vhost_log_access_ok (struct vhost_dev *); -int vhost_get_vq_desc_batch (struct vhost_virtqueue *, -struct iovec iov[],unsigned int iov_count, -unsigned int *out_num,unsigned int *in_num, -struct vhost_log *log,unsigned int *log_num); int vhost_get_vq_desc(struct vhost_virtqueue *, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 19/19 vhost: batching fetches
In reply to Michael S. TsirkinWith this patch applied, new and old code perform identically.
Lots of extra optimizations are now possible, e.g. we can fetch multiple heads with copy_from/to_user now. We can get rid of maintaining the log array. Etc etc.
--- drivers/vhost/test.c | 2 +- drivers/vhost/vhost.c | 47 ++++++++++++++++++++++++++++++++++++++----- drivers/vhost/vhost.h | 5 ++++- 3 files changed, 47 insertions(+), 7 deletions(-)
drivers/vhost/test.c +1 -1
@@ -119,7 +119,7 @@ static int vhost_test_open(struct inode *inode, struct file *f) dev = &n->dev; vqs[VHOST_TEST_VQ ] = &n->vqs[VHOST_TEST_VQ ]; n->vqs[VHOST_TEST_VQ ].handle_kick = handle_vq_kick; -vhost_dev_init (dev, vqs,VHOST_TEST_VQ_MAX ,UIO_MAXIOV , +vhost_dev_init (dev, vqs,VHOST_TEST_VQ_MAX ,UIO_MAXIOV +64 ,VHOST_TEST_PKT_WEIGHT ,VHOST_TEST_WEIGHT ,NULL ); f->private_data = n;
drivers/vhost/vhost.c +42 -5
107 lines not on this page · raw message
drivers/vhost/vhost.h +4 -1
@@ -81,6 +81,7 @@ struct vhost_virtqueue {struct vhost_desc *descs;int ndescs; +int first_desc;int max_descs;struct file *kick;
@@ -229,7 +230,7 @@ void vhost_iotlb_map_free(struct vhost_iotlb *iotlb, struct vhost_iotlb_map *map);#define vq_err (vq, fmt, ...) do { \ - pr_debug(pr_fmt(fmt), ##__VA_ARGS__); \ + pr_err(pr_fmt(fmt), ##__VA_ARGS__); \ if ((vq)->error_ctx) \ eventfd_signal((vq)->error_ctx, 1);\ } while (0)
@@ -255,6 +256,8 @@ static inline void vhost_vq_set_backend(struct vhost_virtqueue *vq, void *private_data) { vq->private_data = private_data; + vq->ndescs =0 ; + vq->first_desc =0 ; }/**
Signature · 1 line
MST
----- Original Message -----
Quoted 21 lines
This is an alternative to
vhost: force spec specified alignment on types
which is a bit safer as it does not change UAPI. I still think it's best to change the UAPI header as well, we can do that as a follow-up cleanup.changes from v7:
typo fix
changes from v6:
add missing header includes all over the place
changes from v5:
ack for mellanox patch
fixup to remoteproc
changes from v4:
fixup to issues reported by kbuild
changes from v3:
tools/virtiofixes
a bunch more cleanups that now become possible
Quoted 5 lines
Changes from v2:
don't change struct name, instead add ifndef so kernel does not see the legacy UAPI version.Jason, can you pls ack one of the approaches?
Btw, this sereis includes the bathing optimization which I think should be done in another one.
Thanks