vDPA support
Subsystems (7)
v6 cover vDPA support
Hi all:
This is an update version of vDPA support in kernel.
vDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:
- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.
From a driver's perspective, depends on how and where the DMA translation is done, vDPA devices are split into two types:
- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.
To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.
This series introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method, the bus operations (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:
virtio drivers vhost drivers
| |
[virtio bus] [vhost uAPI]
| |
virtio device vhost device
virtio_vdpa drv vhost_vdpa drv
\ /
[vDPA bus]
|
vDPA device
hardware drv
|
[hardware bus]
|
vDPA hardwarevirtio_vdpa driver is a transport implementation for kernel virtio drivers on top of vDPA bus operations. An alternative is to refactor virtio bus which is sub-optimal since the bus and drivers are designed to be use by kernel subsystem, a non-trivial major refactoring is needed which may impact a brunches of drivers and devices implementation inside the kernel. Using a new transport may grealy simply both the design and changes.
vhost_vdpa driver is a new type of vhost device which allows userspace vhost drivers to use vDPA devices via vhost uAPI (with minor extension). This help to minimize the changes of existed vhost drivers for using vDPA devices.
With the abstraction of vDPA bus and vDPA bus operations, the difference and complexity of the under layer hardware is hidden from upper layer. The vDPA bus drivers on top can use a unified vdpa_config_ops to control different types of vDPA device.
Two drivers were implemented with the framework introduced in this series:
- Intel IFC VF driver which depends on the platform IOMMU for DMA translation
- VDPA simulator which is a software test device with an emulated onchip IOMMU
Future work:
- direct doorbell mapping support
- control virtqueue support
- dirty page tracking support
- direct interrupt support
- management API (devlink)
Please review.
Thanks
Changes from V5:
- include Intel IFCVF driver and vhost-vdpa drivers
- add the platform IOMMU support for vhost-vdpa
- check the return value of dev_set_name() (Jason)
- various tweaks and fixes
Changes from V4:
- use put_device() instead of kfree when fail to register virtio device (Jason)
- simplify the error handling when allocating vdpasim device (Jason)
- don't use device_for_each_child() during module exit (Jason)
- correct the error checking for vdpa_alloc_device() (Harpreet, Lingshan)
Changes from V3:
- various Kconfig fixes (Randy)
Changes from V2:
- release idr in the release function for put_device() unwind (Jason)
- don't panic when fail to register vdpa bus (Jason)
- use unsigned int instead of int for ida (Jason)
- fix the wrong commit log in virito_vdpa patches (Jason)
- make vdpa_sim depends on RUNTIME_TESTING_MENU (Michael)
- provide a bus release function for vDPA device (Jason)
- fix the wrong unwind when creating devices for vDPA simulator (Jason)
- move vDPA simulator to a dedicated directory (Lingshan)
- cancel the work before release vDPA simulator
Changes from V1:
- drop sysfs API, leave the management interface to future development (Michael)
- introduce incremental DMA ops (dma_map/dma_unmap) (Michael)
- introduce dma_device and use it instead of parent device for doing IOMMU or DMA from bus driver (Michael, Jason, Ling Shan, Tiwei)
- accept parent device and dma device when register vdpa device
- coding style and compile fixes (Randy)
- using vdpa_xxx instead of xxx_vdpa (Jason)
- ove vDPA accessors to header and make it static inline (Jason)
- split vdp_register_device() into two helpers vdpa_init_device() and vdpa_register_device() which allows intermediate step to be done (Jason)
- warn on invalidate queue state when fail to creating virtqueue (Jason)
- make to_virtio_vdpa_device() static (Jason)
- use kmalloc/kfree instead of devres for virtio vdpa device (Jason)
- avoid using cast in vdpa bus function (Jason)
- introduce module_vdpa_driver and fix module refcnt (Jason)
- fix returning freed address in vdapsim coherent DMA addr allocation (Dan)
- various other fixes and tweaks
V5: https://lkml.org/lkml/2020/2/26/58
V4: https://lkml.org/lkml/2020/2/20/59
V3: https://lkml.org/lkml/2020/2/19/1347
V2: https://lkml.org/lkml/2020/2/9/275
V1: https://lkml.org/lkml/2020/1/16/353
Jason Wang (6):
vhost: allow per device message handler
vhost: factor out IOTLB
vringh: IOTLB support
vDPA: introduce vDPA bus
virtio: introduce a vDPA based transport
vdpasim: vDPA device simulator
Tiwei Bie (1):
vhost: introduce vDPA-based backend
Zhu Lingshan (1):
virtio: Intel IFC VF driver for VDPA
MAINTAINERS | 2 + drivers/vhost/Kconfig | 17 + drivers/vhost/Kconfig.vringh | 1 + drivers/vhost/Makefile | 6 + drivers/vhost/iotlb.c | 177 +++++ drivers/vhost/net.c | 5 +- drivers/vhost/scsi.c | 2 +- drivers/vhost/vdpa.c | 883 ++++++++++++++++++++++++ drivers/vhost/vhost.c | 233 +++---- drivers/vhost/vhost.h | 45 +- drivers/vhost/vringh.c | 421 ++++++++++- drivers/vhost/vsock.c | 2 +- drivers/virtio/Kconfig | 15 + drivers/virtio/Makefile | 2 + drivers/virtio/vdpa/Kconfig | 35 + drivers/virtio/vdpa/Makefile | 4 + drivers/virtio/vdpa/ifcvf/Makefile | 3 + drivers/virtio/vdpa/ifcvf/ifcvf_base.c | 386 +++++++++++ drivers/virtio/vdpa/ifcvf/ifcvf_base.h | 133 ++++ drivers/virtio/vdpa/ifcvf/ifcvf_main.c | 494 +++++++++++++ drivers/virtio/vdpa/vdpa.c | 174 +++++ drivers/virtio/vdpa/vdpa_sim/Makefile | 2 + drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c | 646 +++++++++++++++++ drivers/virtio/virtio_vdpa.c | 397 +++++++++++ include/linux/vdpa.h | 232 +++++++ include/linux/vhost_iotlb.h | 47 ++ include/linux/vringh.h | 36 + include/uapi/linux/vhost.h | 24 + include/uapi/linux/vhost_types.h | 8 + 29 files changed, 4222 insertions(+), 210 deletions(-) create mode 100644 drivers/vhost/iotlb.c create mode 100644 drivers/vhost/vdpa.c create mode 100644 drivers/virtio/vdpa/Kconfig create mode 100644 drivers/virtio/vdpa/Makefile create mode 100644 drivers/virtio/vdpa/ifcvf/Makefile create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_base.c create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_base.h create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_main.c create mode 100644 drivers/virtio/vdpa/vdpa.c create mode 100644 drivers/virtio/vdpa/vdpa_sim/Makefile create mode 100644 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c create mode 100644 drivers/virtio/virtio_vdpa.c create mode 100644 include/linux/vdpa.h create mode 100644 include/linux/vhost_iotlb.h
Signature · 1 line
2.20.1
PATCH v6 1/8 vhost: allow per device message handler
This patch allow device to register its own message handler during vhost_dev_init(). vDPA device will use it to implement its own DMA mapping logic.
--- drivers/vhost/net.c | 3 ++- drivers/vhost/scsi.c | 2 +- drivers/vhost/vhost.c | 12 ++++++++++-- drivers/vhost/vhost.h | 6 +++++- drivers/vhost/vsock.c | 2 +- 5 files changed, 19 insertions(+), 6 deletions(-)
drivers/vhost/net.c +2 -1
@@ -1324,7 +1324,8 @@ static int vhost_net_open(struct inode *inode, struct file *f) }vhost_dev_init (dev, vqs,VHOST_NET_VQ_MAX ,UIO_MAXIOV +VHOST_NET_BATCH , -VHOST_NET_PKT_WEIGHT ,VHOST_NET_WEIGHT ); +VHOST_NET_PKT_WEIGHT ,VHOST_NET_WEIGHT , +NULL );vhost_poll_init (n->poll +VHOST_NET_VQ_TX , handle_tx_net,EPOLLOUT , dev);vhost_poll_init (n->poll +VHOST_NET_VQ_RX , handle_rx_net,EPOLLIN , dev);
drivers/vhost/scsi.c +1 -1
@@ -1628,7 +1628,7 @@ static int vhost_scsi_open(struct inode *inode, struct file *f) vs->vqs[i].vq.handle_kick = vhost_scsi_handle_kick; }vhost_dev_init (&vs->dev, vqs,VHOST_SCSI_MAX_VQ ,UIO_MAXIOV , -VHOST_SCSI_WEIGHT ,0 ); +VHOST_SCSI_WEIGHT ,0 ,NULL );vhost_scsi_init_inflight (vs,NULL );
drivers/vhost/vhost.c +10 -2
@@ -457,7 +457,9 @@ static size_t vhost_get_desc_size(struct vhost_virtqueue *vq,void vhost_dev_init (struct vhost_dev *dev,struct vhost_virtqueue **vqs,int nvqs, -int iov_limit,int weight,int byte_weight) +int iov_limit,int weight,int byte_weight, +int (*msg_handler)(struct vhost_dev *dev, +struct vhost_iotlb_msg *msg)) {struct vhost_virtqueue *vq;int i;
@@ -473,6 +475,7 @@ void vhost_dev_init(struct vhost_dev *dev, dev->iov_limit = iov_limit; dev->weight = weight; dev->byte_weight = byte_weight; + dev->msg_handler = msg_handler;init_llist_head (&dev->work_list);init_waitqueue_head (&dev->wait);INIT_LIST_HEAD (&dev->read_list);
@@ -1178,7 +1181,12 @@ ssize_t vhost_chr_write_iter(struct vhost_dev *dev, ret = -EINVAL ; goto done; } -if (vhost_process_iotlb_msg (dev, &msg)) { + +if (dev->msg_handler) + ret = dev->msg_handler (dev, &msg); +else + ret =vhost_process_iotlb_msg (dev, &msg); +if (ret) { ret = -EFAULT ; goto done; }
drivers/vhost/vhost.h +5 -1
@@ -174,11 +174,15 @@ struct vhost_dev {int weight;int byte_weight;u64 kcov_handle; +int (*msg_handler)(struct vhost_dev *dev, +struct vhost_iotlb_msg *msg); };bool vhost_exceeds_weight (struct vhost_virtqueue *vq,int pkts,int total_len);void vhost_dev_init (struct vhost_dev *,struct vhost_virtqueue **vqs, -int nvqs,int iov_limit,int weight,int byte_weight); +int nvqs,int iov_limit,int weight,int byte_weight, +int (*msg_handler)(struct vhost_dev *dev, +struct vhost_iotlb_msg *msg));long vhost_dev_set_owner (struct vhost_dev *dev);bool vhost_dev_has_owner (struct vhost_dev *dev);long vhost_dev_check_owner (struct vhost_dev *);
drivers/vhost/vsock.c +1 -1
@@ -621,7 +621,7 @@ static int vhost_vsock_dev_open(struct inode *inode, struct file *file)vhost_dev_init (&vsock->dev, vqs,ARRAY_SIZE (vsock->vqs),UIO_MAXIOV ,VHOST_VSOCK_PKT_WEIGHT , -VHOST_VSOCK_WEIGHT ); +VHOST_VSOCK_WEIGHT ,NULL ); file->private_data = vsock;spin_lock_init (&vsock->send_pkt_list_lock);
Signature · 1 line
2.20.1
PATCH v6 2/8 vhost: factor out IOTLB
In reply to Jason WangThis patch factors out IOTLB into a dedicated module in order to be reused by other modules like vringh. User may choose to enable the automatic retiring by specifying VHOST_IOTLB_FLAG_RETIRE flag to fit for the case of vhost device IOTLB implementation.
--- MAINTAINERS | 1 + drivers/vhost/Kconfig | 6 + drivers/vhost/Makefile | 3 + drivers/vhost/iotlb.c | 177 +++++++++++++++++++++++++++++ drivers/vhost/net.c | 2 +- drivers/vhost/vhost.c | 221 +++++++++++------------------------- drivers/vhost/vhost.h | 39 ++----- include/linux/vhost_iotlb.h | 47 ++++++++ 8 files changed, 315 insertions(+), 181 deletions(-) create mode 100644 drivers/vhost/iotlb.c create mode 100644 include/linux/vhost_iotlb.h
MAINTAINERS +1 -0
@@ -17768,6 +17768,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git S: Maintained F: drivers/vhost/ F: include/uapi/linux/vhost.h +F: include/linux/vhost_iotlb.h VIRTIO INPUT DRIVER M: Gerd Hoffmann <[email protected]>
drivers/vhost/Kconfig +6 -0
@@ -36,6 +36,7 @@ config VHOST_VSOCK config VHOST tristate + select VHOST_IOTLB ---help--- This option is selected by any driver which needs to access the core of vhost.
@@ -54,3 +55,8 @@ config VHOST_CROSS_ENDIAN_LEGACY adds some overhead, it is disabled by default. If unsure, say "N". + +config VHOST_IOTLB + tristate + help + Generic IOTLB implementation for vhost and vringh.
drivers/vhost/Makefile +3 -0
@@ -11,3 +11,6 @@ vhost_vsock-y := vsock.o obj-$(CONFIG_VHOST_RING) += vringh.o obj-$(CONFIG_VHOST) += vhost.o + +obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o +vhost_iotlb-y := iotlb.o
drivers/vhost/iotlb.c new file +177 -0
177 lines not on this page · raw message
drivers/vhost/net.c +1 -1
@@ -1595,7 +1595,7 @@ static long vhost_net_reset_owner(struct vhost_net *n)struct socket *tx_sock =NULL ;struct socket *rx_sock =NULL ;long err; -struct vhost_umem *umem; +struct vhost_iotlb *umem;mutex_lock (&n->dev.mutex); err =vhost_dev_check_owner (&n->dev);
drivers/vhost/vhost.c +69 -152
432 lines not on this page · raw message
drivers/vhost/vhost.h +11 -28
@@ -12,6 +12,7 @@#include <linux/virtio_config.h> #include <linux/virtio_ring.h> #include <linux/atomic.h> +#include <linux/vhost_iotlb.h> struct vhost_work ;typedef void (*vhost_work_fn_t )(struct vhost_work *work);
@@ -52,27 +53,6 @@ struct vhost_log {u64 len; }; -#define START (node) ((node)->start) -#define LAST (node) ((node)->last) - -struct vhost_umem_node { -struct rb_node rb; -struct list_head link; -__u64 start; -__u64 last; -__u64 size; -__u64 userspace_addr; -__u32 perm; -__u32 flags_padding; -__u64 __subtree_last; -}; - -struct vhost_umem { -struct rb_root_cached umem_tree; -struct list_head umem_list; -int numem; -}; -enum vhost_uaddr_type {VHOST_ADDR_DESC =0 ,VHOST_ADDR_AVAIL =1 ,
@@ -90,7 +70,7 @@ struct vhost_virtqueue {struct vring_desc __user *desc;struct vring_avail __user *avail;struct vring_used __user *used; -const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS ]; +const struct vhost_iotlb_map *meta_iotlb[VHOST_NUM_ADDRS ];struct file *kick;struct eventfd_ctx *call_ctx;struct eventfd_ctx *error_ctx;
@@ -128,8 +108,8 @@ struct vhost_virtqueue {struct iovec *indirect;struct vring_used_elem *heads;/* Protected by virtqueue mutex. */ -struct vhost_umem *umem; -struct vhost_umem *iotlb; +struct vhost_iotlb *umem; +struct vhost_iotlb *iotlb;void *private_data;u64 acked_features;u64 acked_backend_features;
@@ -164,8 +144,8 @@ struct vhost_dev {struct eventfd_ctx *log_ctx;struct llist_head work_list;struct task_struct *worker; -struct vhost_umem *umem; -struct vhost_umem *iotlb; +struct vhost_iotlb *umem; +struct vhost_iotlb *iotlb;spinlock_t iotlb_lock;struct list_head read_list;struct list_head pending_list;
@@ -186,8 +166,8 @@ void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs,long vhost_dev_set_owner (struct vhost_dev *dev);bool vhost_dev_has_owner (struct vhost_dev *dev);long vhost_dev_check_owner (struct vhost_dev *); -struct vhost_umem *vhost_dev_reset_owner_prepare (void ); -void vhost_dev_reset_owner (struct vhost_dev *,struct vhost_umem *); +struct vhost_iotlb *vhost_dev_reset_owner_prepare (void ); +void vhost_dev_reset_owner (struct vhost_dev *dev,struct vhost_iotlb *iotlb);void vhost_dev_cleanup (struct vhost_dev *);void vhost_dev_stop (struct vhost_dev *);long vhost_dev_ioctl (struct vhost_dev *,unsigned int ioctl,void __user *argp);
@@ -233,6 +213,9 @@ ssize_t vhost_chr_write_iter(struct vhost_dev *dev, struct iov_iter *from);int vhost_init_device_iotlb (struct vhost_dev *d,bool enabled); +void vhost_iotlb_map_free (struct vhost_iotlb *iotlb, +struct vhost_iotlb_map *map); +#define vq_err (vq, fmt, ...) do { \ pr_debug(pr_fmt(fmt), ##__VA_ARGS__); \ if ((vq)->error_ctx) \
include/linux/vhost_iotlb.h new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v6 3/8 vringh: IOTLB support
In reply to Jason WangThis patch implements the third memory accessor for vringh besides current kernel and userspace accessors. This idea is to allow vringh to do the address translation through an IOTLB which is implemented via vhost_map interval tree. Users should setup and IOVA to PA mapping in this IOTLB.
This allows us to:
- Using vringh to access virtqueues with vIOMMU
- Using vringh to implement software virtqueues for vDPA devices
--- drivers/vhost/Kconfig.vringh | 1 + drivers/vhost/vringh.c | 421 +++++++++++++++++++++++++++++++++-- include/linux/vringh.h | 36 +++ 3 files changed, 435 insertions(+), 23 deletions(-)
drivers/vhost/Kconfig.vringh +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VHOST_RING tristate + select VHOST_IOTLB ---help--- This option is selected by any driver which needs to access the host side of a virtio ring.
drivers/vhost/vringh.c +398 -23
557 lines not on this page · raw message
include/linux/vringh.h +36 -0
@@ -14,6 +14,8 @@#include <linux/virtio_byteorder.h> #include <linux/uio.h> #include <linux/slab.h> +#include <linux/dma-direction.h> +#include <linux/vhost_iotlb.h> #include <asm/barrier.h> /* virtio_ring with information needed for host access. */
@@ -39,6 +41,9 @@ struct vringh {/* The vring (note: it may contain user pointers!) */ struct vring vring; +/* IOTLB for this vring */ +struct vhost_iotlb *iotlb; +/* The function to call to notify the guest about added buffers */ void (*notify)(struct vringh *); };
@@ -248,4 +253,35 @@ static inline __virtio64 cpu_to_vringh64(const struct vringh *vrh, u64 val) {return __cpu_to_virtio64 (vringh_is_little_endian (vrh), val); } + +void vringh_set_iotlb (struct vringh *vrh,struct vhost_iotlb *iotlb); + +int vringh_init_iotlb (struct vringh *vrh,u64 features, +unsigned int num,bool weak_barriers, +struct vring_desc *desc, +struct vring_avail *avail, +struct vring_used *used); + +int vringh_getdesc_iotlb (struct vringh *vrh, +struct vringh_kiov *riov, +struct vringh_kiov *wiov, +u16 *head, +gfp_t gfp); + +ssize_t vringh_iov_pull_iotlb (struct vringh *vrh, +struct vringh_kiov *riov, +void *dst,size_t len); +ssize_t vringh_iov_push_iotlb (struct vringh *vrh, +struct vringh_kiov *wiov, +const void *src,size_t len); + +void vringh_abandon_iotlb (struct vringh *vrh,unsigned int num); + +int vringh_complete_iotlb (struct vringh *vrh,u16 head,u32 len); + +bool vringh_notify_enable_iotlb (struct vringh *vrh); +void vringh_notify_disable_iotlb (struct vringh *vrh); + +int vringh_need_notify_iotlb (struct vringh *vrh); +#endif /* _LINUX_VRINGH_H */
Signature · 1 line
2.20.1
kbuild test robot
build botRe: [PATCH V6 3/8] vringh: IOTLB support
I love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on vhost/linux-next]
[also build test ERROR on linux/master linus/master v5.6-rc6 next-20200317] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Jason-Wang/vDPA-support/20200318-191435 base: https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git linux-next config: h8300-randconfig-a001-20200318 (attached as .config) compiler: h8300-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=h8300
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
h8300-linux-ld: drivers/vhost/vringh.o: in function `iotlb_translate':
>> drivers/vhost/vringh.c:1079: undefined reference to `vhost_iotlb_itree_first'
vim +1079 drivers/vhost/vringh.c
1061
1062 static int iotlb_translate(const struct vringh *vrh,
1063 u64 addr, u64 len, struct bio_vec iov[],
1064 int iov_size, u32 perm)
1065 {
1066 struct vhost_iotlb_map *map;
1067 struct vhost_iotlb *iotlb = vrh->iotlb;
1068 int ret = 0;
1069 u64 s = 0;
1070
1071 while (len > s) {
1072 u64 size, pa, pfn;
1073
1074 if (unlikely(ret >= iov_size)) {
1075 ret = -ENOBUFS;
1076 break;
1077 }
1078
> 1079 map = vhost_iotlb_itree_first(iotlb, addr,
1080 addr + len - 1);
1081 if (!map || map->start > addr) {
1082 ret = -EINVAL;
1083 break;
1084 } else if (!(map->perm & perm)) {
1085 ret = -EPERM;
1086 break;
1087 }
1088
1089 size = map->size - addr + map->start;
1090 pa = map->addr + addr - map->start;
1091 pfn = pa >> PAGE_SHIFT;
1092 iov[ret].bv_page = pfn_to_page(pfn);
1093 iov[ret].bv_len = min(len - s, size);
1094 iov[ret].bv_offset = pa & (PAGE_SIZE - 1);
1095 s += size;
1096 addr += size;
1097 ++ret;
1098 }
1099
1100 return ret;
1101 }
1102
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kbuild test robot
build botRe: [PATCH V6 3/8] vringh: IOTLB support
In reply to Jason WangI love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on vhost/linux-next]
[also build test ERROR on linux/master linus/master v5.6-rc6 next-20200317] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Jason-Wang/vDPA-support/20200318-191435 base: https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git linux-next config: nds32-randconfig-a001-20200318 (attached as .config) compiler: nds32le-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=nds32
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
nds32le-linux-ld: arch/nds32/kernel/ex-entry.o: in function `common_exception_handler':
(.text+0xfe): undefined reference to `__trace_hardirqs_off'
(.text+0xfe): relocation truncated to fit: R_NDS32_25_PCREL_RELA against undefined symbol `__trace_hardirqs_off'
nds32le-linux-ld: arch/nds32/kernel/ex-exit.o: in function `no_work_pending':
(.text+0xce): undefined reference to `__trace_hardirqs_off'
nds32le-linux-ld: (.text+0xd2): undefined reference to `__trace_hardirqs_off'
nds32le-linux-ld: (.text+0xd6): undefined reference to `__trace_hardirqs_on'
nds32le-linux-ld: (.text+0xda): undefined reference to `__trace_hardirqs_on'
nds32le-linux-ld: drivers/vhost/vringh.o: in function `iotlb_translate.isra.0':
>> vringh.c:(.text+0xa68): undefined reference to `vhost_iotlb_itree_first' >> nds32le-linux-ld: vringh.c:(.text+0xa6c): undefined reference to `vhost_iotlb_itree_first'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH v6 4/8 vDPA: introduce vDPA bus
In reply to Jason WangvDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:
- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.
From a driver's perspective, depends on how and where the DMA translation is done, vDPA devices are split into two types:
- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.
To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.
This patch introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:
virtio drivers vhost drivers
| |
[virtio bus] [vhost uAPI]
| |
virtio device vhost device
virtio_vdpa drv vhost_vdpa drv
\ /
[vDPA bus]
|
vDPA device
hardware drv
|
[hardware bus]
|
vDPA hardwareWith the abstraction of vDPA bus and vDPA bus operations, the difference and complexity of the under layer hardware is hidden from upper layer. The vDPA bus drivers on top can use a unified vdpa_config_ops to control different types of vDPA device.
--- MAINTAINERS | 1 + drivers/virtio/Kconfig | 2 + drivers/virtio/Makefile | 1 + drivers/virtio/vdpa/Kconfig | 7 ++ drivers/virtio/vdpa/Makefile | 2 + drivers/virtio/vdpa/vdpa.c | 174 ++++++++++++++++++++++++++ include/linux/vdpa.h | 232 +++++++++++++++++++++++++++++++++++ 7 files changed, 419 insertions(+) create mode 100644 drivers/virtio/vdpa/Kconfig create mode 100644 drivers/virtio/vdpa/Makefile create mode 100644 drivers/virtio/vdpa/vdpa.c create mode 100644 include/linux/vdpa.h
MAINTAINERS +1 -0
@@ -17701,6 +17701,7 @@ F: tools/virtio/ F: drivers/net/virtio_net.c F: drivers/block/virtio_blk.c F: include/linux/virtio*.h +F: include/linux/vdpa.h F: include/uapi/linux/virtio_*.h F: drivers/crypto/virtio/ F: mm/balloon_compaction.c
drivers/virtio/Kconfig +2 -0
@@ -96,3 +96,5 @@ config VIRTIO_MMIO_CMDLINE_DEVICES If unsure, say 'N'. endif # VIRTIO_MENU + +source "drivers/virtio/vdpa/Kconfig"
drivers/virtio/Makefile +1 -0
@@ -6,3 +6,4 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o +obj-$(CONFIG_VDPA) += vdpa/
drivers/virtio/vdpa/Kconfig new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VDPA + tristate + help + Enable this module to support vDPA device that uses a + datapath which complies with virtio specifications with + vendor specific control path.
drivers/virtio/vdpa/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_VDPA) += vdpa.o
drivers/virtio/vdpa/vdpa.c new file +174 -0
174 lines not on this page · raw message
include/linux/vdpa.h new file +232 -0
232 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v6 5/8 virtio: introduce a vDPA based transport
In reply to Jason WangThis patch introduces a vDPA transport for virtio. This is used to use kernel virtio driver to drive the vDPA device that is capable of populating virtqueue directly.
A new virtio-vdpa driver will be registered to the vDPA bus, when a new virtio-vdpa device is probed, it will register the device with vdpa based config ops. This means it is a software transport between vDPA driver and vDPA device. The transport was implemented through bus_ops of vDPA parent.
--- drivers/virtio/Kconfig | 13 ++ drivers/virtio/Makefile | 1 + drivers/virtio/virtio_vdpa.c | 396 +++++++++++++++++++++++++++++++++++ 3 files changed, 410 insertions(+) create mode 100644 drivers/virtio/virtio_vdpa.c
drivers/virtio/Kconfig +13 -0
@@ -43,6 +43,19 @@ config VIRTIO_PCI_LEGACY If unsure, say Y. +config VIRTIO_VDPA + tristate "vDPA driver for virtio devices" + select VDPA + select VIRTIO + help + This driver provides support for virtio based paravirtual + device driver over vDPA bus. For this to be useful, you need + an appropriate vDPA device implementation that operates on a + physical device to allow the datapath of virtio to be + offloaded to hardware. + + If unsure, say M. + config VIRTIO_PMEM tristate "Support for virtio pmem driver" depends on VIRTIO
drivers/virtio/Makefile +1 -0
@@ -6,4 +6,5 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o +obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o obj-$(CONFIG_VDPA) += vdpa/
drivers/virtio/virtio_vdpa.c new file +396 -0
396 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v6 6/8 vhost: introduce vDPA-based backend
In reply to Jason WangFrom: Tiwei Bie <[email protected]>
This patch introduces a vDPA-based vhost backend. This backend is built on top of the same interface defined in virtio-vDPA and provides a generic vhost interface for userspace to accelerate the virtio devices in guest.
This backend is implemented as a vDPA device driver on top of the same ops used in virtio-vDPA. It will create char device entry named vhost-vdpa-$index for userspace to use. Userspace can use vhost ioctls on top of this char device to setup the backend.
Vhost ioctls are extended to make it type agnostic and behave like a virtio device, this help to eliminate type specific API like what vhost_net/scsi/vsock did:
- VHOST_VDPA_GET_DEVICE_ID: get the virtio device ID which is defined by virtio specification to differ from different type of devices
- VHOST_VDPA_GET_VRING_NUM: get the maximum size of virtqueue supported by the vDPA device
- VHSOT_VDPA_SET/GET_STATUS: set and get virtio status of vDPA device
- VHOST_VDPA_SET/GET_CONFIG: access virtio config space
- VHOST_VDPA_SET_VRING_ENABLE: enable a specific virtqueue
For memory mapping, IOTLB API is mandated for vhost-vDPA which means userspace drivers are required to use VHOST_IOTLB_UPDATE/VHOST_IOTLB_INVALIDATE to add or remove mapping for a specific userspace memory region.
The vhost-vDPA API is designed to be type agnostic, but it allows net device only in current stage. Due to the lacking of control virtqueue support, some features were filter out by vhost-vdpa.
We will enable more features and devices in the near future.
--- drivers/vhost/Kconfig | 11 + drivers/vhost/Makefile | 3 + drivers/vhost/vdpa.c | 883 +++++++++++++++++++++++++++++++ include/uapi/linux/vhost.h | 24 + include/uapi/linux/vhost_types.h | 8 + 5 files changed, 929 insertions(+) create mode 100644 drivers/vhost/vdpa.c
drivers/vhost/Kconfig +11 -0
@@ -34,6 +34,17 @@ config VHOST_VSOCK To compile this driver as a module, choose M here: the module will be called vhost_vsock. +config VHOST_VDPA + tristate "Vhost driver for vDPA-based backend" + depends on EVENTFD && VDPA + select VHOST + help + This kernel module can be loaded in host kernel to accelerate + guest virtio devices with the vDPA-based backends. + + To compile this driver as a module, choose M here: the module + will be called vhost_vdpa. + config VHOST tristate select VHOST_IOTLB
drivers/vhost/Makefile +3 -0
@@ -10,6 +10,9 @@ vhost_vsock-y := vsock.o obj-$(CONFIG_VHOST_RING) += vringh.o +obj-$(CONFIG_VHOST_VDPA) += vhost_vdpa.o +vhost_vdpa-y := vdpa.o + obj-$(CONFIG_VHOST) += vhost.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o
drivers/vhost/vdpa.c new file +883 -0
883 lines not on this page · raw message
include/uapi/linux/vhost.h +24 -0
@@ -116,4 +116,28 @@#define VHOST_VSOCK_SET_GUEST_CID _IOW(VHOST_VIRTIO, 0x60, __u64)#define VHOST_VSOCK_SET_RUNNING _IOW(VHOST_VIRTIO, 0x61, int) +/* VHOST_VDPA specific defines */ + +/* Get the device id. The device ids follow the same definition of +* the device id defined in virtio-spec. +*/ +#define VHOST_VDPA_GET_DEVICE_ID _IOR(VHOST_VIRTIO, 0x70, __u32) +/* Get and set the status. The status bits follow the same definition +* of the device status defined in virtio-spec. +*/ +#define VHOST_VDPA_GET_STATUS _IOR(VHOST_VIRTIO, 0x71, __u8) +#define VHOST_VDPA_SET_STATUS _IOW(VHOST_VIRTIO, 0x72, __u8) +/* Get and set the device config. The device config follows the same +* definition of the device config defined in virtio-spec. +*/ +#define VHOST_VDPA_GET_CONFIG _IOR(VHOST_VIRTIO, 0x73, \ + struct vhost_vdpa_config) +#define VHOST_VDPA_SET_CONFIG _IOW(VHOST_VIRTIO, 0x74, \ + struct vhost_vdpa_config) +/* Enable/disable the ring. */ +#define VHOST_VDPA_SET_VRING_ENABLE _IOW(VHOST_VIRTIO, 0x75, \ + struct vhost_vring_state) +/* Get the max ring size. */ +#define VHOST_VDPA_GET_VRING_NUM _IOR(VHOST_VIRTIO, 0x76, __u16) +#endif
include/uapi/linux/vhost_types.h +8 -0
@@ -119,6 +119,14 @@ struct vhost_scsi_target {unsigned short reserved; }; +/* VHOST_VDPA specific definitions */ + +struct vhost_vdpa_config { +__u32 off; +__u32 len; +__u8 buf[0 ]; +}; +/* Feature bits */ /* Log all write descriptors. Can be changed while device is active. */ #define VHOST_F_LOG_ALL 26
Signature · 1 line
2.20.1
PATCH v6 7/8 vdpasim: vDPA device simulator
In reply to Jason WangThis patch implements a software vDPA networking device. The datapath is implemented through vringh and workqueue. The device has an on-chip IOMMU which translates IOVA to PA. For kernel virtio drivers, vDPA simulator driver provides dma_ops. For vhost driers, set_map() methods of vdpa_config_ops is implemented to accept mappings from vhost.
Currently, vDPA device simulator will loopback TX traffic to RX. So the main use case for the device is vDPA feature testing, prototyping and development.
Note, there's no management API implemented, a vDPA device will be registered once the module is probed. We need to handle this in the future development.
--- drivers/virtio/vdpa/Kconfig | 18 + drivers/virtio/vdpa/Makefile | 1 + drivers/virtio/vdpa/vdpa_sim/Makefile | 2 + drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c | 646 ++++++++++++++++++++++++ 4 files changed, 667 insertions(+) create mode 100644 drivers/virtio/vdpa/vdpa_sim/Makefile create mode 100644 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c
drivers/virtio/vdpa/Kconfig +18 -0
@@ -5,3 +5,21 @@ config VDPA Enable this module to support vDPA device that uses a datapath which complies with virtio specifications with vendor specific control path. + +menuconfig VDPA_MENU + bool "VDPA drivers" + default n + +if VDPA_MENU + +config VDPA_SIM + tristate "vDPA device simulator" + select VDPA + depends on RUNTIME_TESTING_MENU && VHOST_RING + default n + help + vDPA networking device simulator which loop TX traffic back + to RX. This device is used for testing, prototyping and + development of vDPA. + +endif # VDPA_MENU
drivers/virtio/vdpa/Makefile +1 -0
@@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_VDPA) += vdpa.o +obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
drivers/virtio/vdpa/vdpa_sim/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_VDPA_SIM) += vdpa_sim.o
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +646 -0
646 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v6 8/8 virtio: Intel IFC VF driver for VDPA
In reply to Jason WangFrom: Zhu Lingshan <[email protected]>
This commit introduced two layers to drive IFC VF:
(1) ifcvf_base layer, which handles IFC VF NIC hardware operations and configurations.
(2) ifcvf_main layer, which complies to VDPA bus framework, implemented device operations for VDPA bus, handles device probe, bus attaching, vring operations, etc.
--- drivers/virtio/vdpa/Kconfig | 10 + drivers/virtio/vdpa/Makefile | 1 + drivers/virtio/vdpa/ifcvf/Makefile | 3 + drivers/virtio/vdpa/ifcvf/ifcvf_base.c | 386 +++++++++++++++++++ drivers/virtio/vdpa/ifcvf/ifcvf_base.h | 133 +++++++ drivers/virtio/vdpa/ifcvf/ifcvf_main.c | 494 +++++++++++++++++++++++++ drivers/virtio/virtio_vdpa.c | 1 + 7 files changed, 1028 insertions(+) create mode 100644 drivers/virtio/vdpa/ifcvf/Makefile create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_base.c create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_base.h create mode 100644 drivers/virtio/vdpa/ifcvf/ifcvf_main.c
drivers/virtio/vdpa/Kconfig +10 -0
@@ -22,4 +22,14 @@ config VDPA_SIM to RX. This device is used for testing, prototyping and development of vDPA. +config IFCVF + tristate "Intel IFC VF VDPA driver" + depends on VDPA + default n + help + This kernel module can drive Intel IFC VF NIC to offload + virtio dataplane traffic to hardware. + To compile this driver as a module, choose M here: the module will + be called ifcvf. + endif # VDPA_MENU
drivers/virtio/vdpa/Makefile +1 -0
@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_VDPA) += vdpa.o obj-$(CONFIG_VDPA_SIM) += vdpa_sim/ +obj-$(CONFIG_IFCVF) += ifcvf/
drivers/virtio/vdpa/ifcvf/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_IFCVF) += ifcvf.o +ifcvf-$(CONFIG_IFCVF) += ifcvf_main.o ifcvf_base.o
drivers/virtio/vdpa/ifcvf/ifcvf_base.c new file +386 -0
386 lines not on this page · raw message
drivers/virtio/vdpa/ifcvf/ifcvf_base.h new file +133 -0
133 lines not on this page · raw message
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +494 -0
494 lines not on this page · raw message
drivers/virtio/virtio_vdpa.c +1 -0
@@ -362,6 +362,7 @@ static int virtio_vdpa_probe(struct vdpa_device *vdpa) goto err;vdpa_set_drvdata (vdpa, vd_dev); +dev_info (vd_dev->vdev.dev.parent,"device attached to VDPA bus\n" );return 0 ;
Signature · 1 line
2.20.1
Re: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
drivers/virtio/vdpa/ifcvf/ifcvf_main.c line 351
Quoted 13 lines · Jason Wang
From: Zhu Lingshan <[email protected]>
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +12
-0+ +static int ifcvf_vdpa_attach (struct ifcvf_adapter *adapter) +{ +int ret; + + adapter->vdpa_dev =vdpa_alloc_device (adapter->dev, adapter->dev, + &ifc_vdpa_ops); +if (IS_ERR (adapter->vdpa_dev)) { +IFCVF_ERR (adapter->dev,"Failed to init ifcvf on vdpa bus" ); +put_device (&adapter->vdpa_dev->dev); +return -ENODEV ; + }
The point of having an alloc call is so that the drivers ifcvf_adaptor memory could be placed in the same struct - eg use container_of to flip between them, and have a kref for both memories.
It seem really weird to have an alloc followed immediately by register.
drivers/virtio/virtio_vdpa.c static int virtio_vdpa_probe(struct vdpa_device *vdpa) · line 367
Quoted 10 lines
drivers/virtio/virtio_vdpa.c +2
-0+++ b/drivers/virtio/virtio_vdpa.c@@ -362,6 +362,7 @@ static int virtio_vdpa_probe(struct vdpa_device *vdpa) goto err;vdpa_set_drvdata (vdpa, vd_dev); +dev_info (vd_dev->vdev.dev.parent,"device attached to VDPA bus\n" );return 0 ;
This hunk seems out of place
Jason
Re: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
Quoted 18 lines · Jason Gunthorpe
drivers/virtio/vdpa/ifcvf/ifcvf_main.c line 351
Quoted 13 lines · Jason Wang
From: Zhu Lingshan <[email protected]>
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +12
-0+ +static int ifcvf_vdpa_attach (struct ifcvf_adapter *adapter) +{ +int ret; + + adapter->vdpa_dev =vdpa_alloc_device (adapter->dev, adapter->dev, + &ifc_vdpa_ops); +if (IS_ERR (adapter->vdpa_dev)) { +IFCVF_ERR (adapter->dev,"Failed to init ifcvf on vdpa bus" ); +put_device (&adapter->vdpa_dev->dev); +return -ENODEV ; + }The point of having an alloc call is so that the drivers
ifcvf_adaptormemory could be placed in the same struct - eg usecontainer_ofto flip between them, and have a kref for both memories.
Quoted 2 lines
It seem really weird to have an alloc followed immediately by register.
Hi Jason,
Thanks for your comments, but I failed to understand this. In IFCVF driver, we call ifcvf_vdpa_attach() at the end at probe(), at this point, PCIE initialization almost done, then try to alloc vdpa device, if successful, we can register it to VDPA bus.
Are you suggesting to wait for anything else done?
THanks,
BR
Zhu Linghsan
Quoted 14 lines
drivers/virtio/virtio_vdpa.c static int virtio_vdpa_probe(struct vdpa_device *vdpa) · line 367
Quoted 11 lines
drivers/virtio/virtio_vdpa.c +2
-0+++ b/drivers/virtio/virtio_vdpa.c@@ -362,6 +362,7 @@ static int virtio_vdpa_probe(struct vdpa_device *vdpa) goto err;vdpa_set_drvdata (vdpa, vd_dev); +dev_info (vd_dev->vdev.dev.parent,"device attached to VDPA bus\n" );return 0 ;This hunk seems out of place
Jason
Re: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
In reply to Jason GunthorpeQuoted 18 lines · Jason Gunthorpe
drivers/virtio/vdpa/ifcvf/ifcvf_main.c line 351
Quoted 13 lines · Jason Wang
From: Zhu Lingshan <[email protected]>
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +12
-0+ +static int ifcvf_vdpa_attach (struct ifcvf_adapter *adapter) +{ +int ret; + + adapter->vdpa_dev =vdpa_alloc_device (adapter->dev, adapter->dev, + &ifc_vdpa_ops); +if (IS_ERR (adapter->vdpa_dev)) { +IFCVF_ERR (adapter->dev,"Failed to init ifcvf on vdpa bus" ); +put_device (&adapter->vdpa_dev->dev); +return -ENODEV ; + }The point of having an alloc call is so that the drivers
ifcvf_adaptormemory could be placed in the same struct - eg usecontainer_ofto flip between them, and have a kref for both memories.
Quoted 2 lines
It seem really weird to have an alloc followed immediately by register.
I admit the ifcvf_adapter is not correctly ref-counted. What you suggest should work. But it looks to me the following is more cleaner since the members of ifcvf_adapter are all related to PCI device not vDPA itself.
- keep the current layout of ifcvf_adapter
- merge vdpa_alloc_device() and vdpa_register_device()
- use devres to bind ifcvf_adapter refcnt/lifcycle to the under PCI device
If we go for the container_of method, we probably need
- accept a size of parent parent structure in vdpa_alloc_device() and mandate vdpa_device to be the first member of ifcvf_adapter
- we need provide a way to free resources of parent structure when we destroy vDPA device
What's your thought?
Quoted 6 lines
Quoted 6 lines
drivers/virtio/virtio_vdpa.c +1
-0+++ b/drivers/virtio/virtio_vdpa.c@@ -362,6 +362,7 @@ static int virtio_vdpa_probe(struct vdpa_device *vdpa) goto err;
Quoted 7 lines
drivers/virtio/virtio_vdpa.c static int virtio_vdpa_probe(struct vdpa_device *vdpa) · line 367
Quoted 4 lines
drivers/virtio/virtio_vdpa.c +1
-0vdpa_set_drvdata (vdpa, vd_dev); +dev_info (vd_dev->vdev.dev.parent,"device attached to VDPA bus\n" );return 0 ;This hunk seems out of place
Jason
Right, will fix.
Thanks
Re: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
Quoted 20 lines · Jason Wang
Quoted 18 lines · Jason Gunthorpe
drivers/virtio/vdpa/ifcvf/ifcvf_main.c line 351
Quoted 13 lines · Jason Wang
From: Zhu Lingshan <[email protected]>
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +12
-0+ +static int ifcvf_vdpa_attach (struct ifcvf_adapter *adapter) +{ +int ret; + + adapter->vdpa_dev =vdpa_alloc_device (adapter->dev, adapter->dev, + &ifc_vdpa_ops); +if (IS_ERR (adapter->vdpa_dev)) { +IFCVF_ERR (adapter->dev,"Failed to init ifcvf on vdpa bus" ); +put_device (&adapter->vdpa_dev->dev); +return -ENODEV ; + }The point of having an alloc call is so that the drivers
ifcvf_adaptormemory could be placed in the same struct - eg usecontainer_ofto flip between them, and have a kref for both memories.
Quoted 7 lines
Quoted 2 lines
It seem really weird to have an alloc followed immediately by register.
I admit the
ifcvf_adapteris not correctly ref-counted. What you suggest should work. But it looks to me the following is more cleaner since the members ofifcvf_adapterare all related to PCI device not vDPA itself.
I've done it both ways (eg tpm is as you describe, ib is using alloc).
I tend to prefer the alloc method today, allowing the driver memory to have a proper refcount makes the driver structure usable with RCU and allows simple solutions to some tricky cases. It is a bit hard to switch to this later..
Quoted 3 lines
- keep the current layout of
ifcvf_adapter
- mergevdpa_alloc_device()andvdpa_register_device()
- use devres to bindifcvf_adapterrefcnt/lifcycle to the under PCI device
This is almost what tpm does. Keep in mind the lifecycle with devm is just slightly past the driver remove call, so remove still must revoke all external references to the memory.
The merging alloc and register rarely works out, the register must be the very last thing done, and usually you need the subsystem pointer to do pre-registration setup in anything but the most trivial of subsystems and drivers.
Quoted 6 lines
If we go for the
container_ofmethod, we probably need- accept a size of parent parent structure in
vdpa_alloc_device()and mandatevdpa_deviceto be the first member ofifcvf_adapter
- we need provide a way to free resources of parent structure when we destroy vDPA device
Yep. netdev and rdma work this way with a free memory callback in the existing ops structures.
Jason
Re: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
Quoted 47 lines · Jason Gunthorpe
Quoted 19 lines · Jason Wang
Quoted 18 lines · Jason Gunthorpe
drivers/virtio/vdpa/ifcvf/ifcvf_main.c line 351
Quoted 13 lines · Jason Wang
From: Zhu Lingshan <[email protected]>
drivers/virtio/vdpa/ifcvf/ifcvf_main.c new file +12
-0+ +static int ifcvf_vdpa_attach (struct ifcvf_adapter *adapter) +{ +int ret; + + adapter->vdpa_dev =vdpa_alloc_device (adapter->dev, adapter->dev, + &ifc_vdpa_ops); +if (IS_ERR (adapter->vdpa_dev)) { +IFCVF_ERR (adapter->dev,"Failed to init ifcvf on vdpa bus" ); +put_device (&adapter->vdpa_dev->dev); +return -ENODEV ; + }The point of having an alloc call is so that the drivers
ifcvf_adaptormemory could be placed in the same struct - eg usecontainer_ofto flip between them, and have a kref for both memories.Quoted 6 lines
Quoted 2 lines
It seem really weird to have an alloc followed immediately by register.
I admit the
ifcvf_adapteris not correctly ref-counted. What you suggest should work. But it looks to me the following is more cleaner since the members ofifcvf_adapterare all related to PCI device not vDPA itself.I've done it both ways (eg tpm is as you describe, ib is using alloc).
I tend to prefer the alloc method today, allowing the driver memory to have a proper refcount makes the driver structure usable with RCU and allows simple solutions to some tricky cases. It is a bit hard to switch to this later..
Quoted 3 lines
- keep the current layout of
ifcvf_adapter
- mergevdpa_alloc_device()andvdpa_register_device()
- use devres to bindifcvf_adapterrefcnt/lifcycle to the under PCI deviceThis is almost what tpm does. Keep in mind the lifecycle with devm is just slightly past the driver remove call, so remove still must revoke all external references to the memory.
The merging alloc and register rarely works out, the register must be the very last thing done, and usually you need the subsystem pointer to do pre-registration setup in anything but the most trivial of subsystems and drivers.
Quoted 2 lines
If we go for the
container_ofmethod, we probably need
Quoted 8 lines
Quoted 4 lines
- accept a size of parent parent structure in
vdpa_alloc_device()and mandatevdpa_deviceto be the first member ofifcvf_adapter
- we need provide a way to free resources of parent structure when we destroy vDPA deviceYep. netdev and rdma work this way with a free memory callback in the existing ops structures.
Jason
Ok, I get your points now. Will go for way of container_of in next version.
Thanks
kbuild test robot
build botRe: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
In reply to Jason WangThank you for the patch! Yet something to improve:
Hi Jason,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vhost/linux-next]
[also build test ERROR on linux/master linus/master v5.6-rc6 next-20200317] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Jason-Wang/vDPA-support/20200318-191435 base: https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git linux-next config: c6x-allyesconfig (attached as .config) compiler: c6x-elf-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=c6x
If you fix the issue, kindly add following tag
All error/warnings (new ones prefixed by >>):
drivers/virtio/vdpa/ifcvf/ifcvf_main.c: In function 'ifcvf_probe':
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:409:30: error: implicit declaration of function 'pci_iomap_range'; did you mean 'pci_unmap_page'? [-Werror=implicit-function-declaration]
409 | vf->mem_resource[i].addr = pci_iomap_range(pdev, i, 0,
| ^~~~~~~~~~~~~~~
| pci_unmap_page
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:409:28: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
409 | vf->mem_resource[i].addr = pci_iomap_range(pdev, i, 0,
| ^
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:443:2: error: implicit declaration of function 'pci_free_irq_vectors'; did you mean 'pci_alloc_irq_vectors'? [-Werror=implicit-function-declaration]
443 | pci_free_irq_vectors(pdev);
| ^~~~~~~~~~~~~~~~~~~~
| pci_alloc_irq_vectors
drivers/virtio/vdpa/ifcvf/ifcvf_main.c: At top level:
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: warning: data definition has no type or storage class
491 | module_pci_driver(ifcvf_driver);
| ^~~~~~~~~~~~~~~~~
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: error: type defaults to 'int' in declaration of 'module_pci_driver' [-Werror=implicit-int]
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: warning: parameter names (without types) in function declaration
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:484:26: warning: 'ifcvf_driver' defined but not used [-Wunused-variable]
484 | static struct pci_driver ifcvf_driver = {
| ^~~~~~~~~~~~
cc1: some warnings being treated as errors
--
drivers/virtio/vdpa/ifcvf/ifcvf_base.c: In function 'ifcvf_init_hw':
>> drivers/virtio/vdpa/ifcvf/ifcvf_base.c:110:8: warning: 'pos' is used uninitialized in this function [-Wuninitialized]
110 | while (pos) {
| ^vim +409 drivers/virtio/vdpa/ifcvf/ifcvf_main.c
365
366 static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
367 {
368 struct device *dev = &pdev->dev;
369 struct ifcvf_adapter *adapter;
370 struct ifcvf_hw *vf;
371 int ret, i;
372
373 adapter = kzalloc(sizeof(struct ifcvf_adapter), GFP_KERNEL);
374 if (adapter == NULL) {
375 ret = -ENOMEM;
376 goto fail;
377 }
378
379 adapter->dev = dev;
380 pci_set_drvdata(pdev, adapter);
381 ret = pci_enable_device(pdev);
382 if (ret) {
383 IFCVF_ERR(adapter->dev, "Failed to enable device\n");
384 goto free_adapter;
385 }
386
387 ret = pci_request_regions(pdev, IFCVF_DRIVER_NAME);
388 if (ret) {
389 IFCVF_ERR(adapter->dev, "Failed to request MMIO region\n");
390 goto disable_device;
391 }
392
393 pci_set_master(pdev);
394 ret = ifcvf_init_msix(adapter);
395 if (ret) {
396 IFCVF_ERR(adapter->dev, "Failed to initialize MSI-X\n");
397 goto free_msix;
398 }
399
400 vf = &adapter->vf;
401 for (i = 0; i < IFCVF_PCI_MAX_RESOURCE; i++) {
402 vf->mem_resource[i].phys_addr = pci_resource_start(pdev, i);
403 vf->mem_resource[i].len = pci_resource_len(pdev, i);
404 if (!vf->mem_resource[i].len) {
405 vf->mem_resource[i].addr = NULL;
406 continue;
407 }
408
> 409 vf->mem_resource[i].addr = pci_iomap_range(pdev, i, 0,
410 vf->mem_resource[i].len);
411 if (!vf->mem_resource[i].addr) {
412 IFCVF_ERR(adapter->dev,
413 "Failed to map IO resource %d\n", i);
414 ret = -EINVAL;
415 goto free_msix;
416 }
417 }
418
419 if (ifcvf_init_hw(vf, pdev) < 0) {
420 ret = -EINVAL;
421 goto destroy_adapter;
422 }
423
424 ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
425 if (ret)
426 ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
427
428 if (ret) {
429 IFCVF_ERR(adapter->dev, "No usable DMA confiugration\n");
430 ret = -EINVAL;
431 goto destroy_adapter;
432 }
433
434 ret = ifcvf_vdpa_attach(adapter);
435 if (ret)
436 goto destroy_adapter;
437
438 return 0;
439
440 destroy_adapter:
441 ifcvf_destroy_adapter(adapter);
442 free_msix:
> 443 pci_free_irq_vectors(pdev);
444 pci_release_regions(pdev);
445 disable_device:
446 pci_disable_device(pdev);
447 free_adapter:
448 kfree(adapter);
449 fail:
450 return ret;
451 }
452
453 static void ifcvf_remove(struct pci_dev *pdev)
454 {
455 struct ifcvf_adapter *adapter = pci_get_drvdata(pdev);
456 struct ifcvf_hw *vf;
457 int i;
458
459 ifcvf_vdpa_detach(adapter);
460 vf = &adapter->vf;
461 for (i = 0; i < IFCVF_PCI_MAX_RESOURCE; i++) {
462 if (vf->mem_resource[i].addr) {
463 pci_iounmap(pdev, vf->mem_resource[i].addr);
464 vf->mem_resource[i].addr = NULL;
465 }
466 }
467
468 ifcvf_destroy_adapter(adapter);
469 pci_free_irq_vectors(pdev);
470 pci_release_regions(pdev);
471 pci_disable_device(pdev);
472 kfree(adapter);
473 }
474
475 static struct pci_device_id ifcvf_pci_ids[] = {
476 { PCI_DEVICE_SUB(IFCVF_VENDOR_ID,
477 IFCVF_DEVICE_ID,
478 IFCVF_SUBSYS_VENDOR_ID,
479 IFCVF_SUBSYS_DEVICE_ID) },
480 { 0 },
481 };
482 MODULE_DEVICE_TABLE(pci, ifcvf_pci_ids);
483
484 static struct pci_driver ifcvf_driver = {
485 .name = IFCVF_DRIVER_NAME,
486 .id_table = ifcvf_pci_ids,
487 .probe = ifcvf_probe,
488 .remove = ifcvf_remove,
489 };
490
> 491 module_pci_driver(ifcvf_driver);492
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kbuild test robot
build botRe: [PATCH V6 8/8] virtio: Intel IFC VF driver for VDPA
In reply to Jason WangThank you for the patch! Yet something to improve:
Hi Jason,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vhost/linux-next]
[also build test ERROR on linux/master linus/master v5.6-rc6 next-20200317] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Jason-Wang/vDPA-support/20200318-191435 base: https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git linux-next config: sh-allmodconfig (attached as .config) compiler: sh4-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=sh
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/virtio/vdpa/ifcvf/ifcvf_main.c: In function 'ifcvf_probe':
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:409:30: error: implicit declaration of function 'pci_iomap_range'; did you mean 'pci_unmap_page'? [-Werror=implicit-function-declaration]
409 | vf->mem_resource[i].addr = pci_iomap_range(pdev, i, 0,
| ^~~~~~~~~~~~~~~
| pci_unmap_page
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:409:28: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
409 | vf->mem_resource[i].addr = pci_iomap_range(pdev, i, 0,
| ^
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:443:2: error: implicit declaration of function 'pci_free_irq_vectors'; did you mean 'pci_alloc_irq_vectors'? [-Werror=implicit-function-declaration]
443 | pci_free_irq_vectors(pdev);
| ^~~~~~~~~~~~~~~~~~~~
| pci_alloc_irq_vectors
drivers/virtio/vdpa/ifcvf/ifcvf_main.c: In function 'ifcvf_remove':
>> drivers/virtio/vdpa/ifcvf/ifcvf_main.c:463:4: error: implicit declaration of function 'pci_iounmap'; did you mean 'pcim_iounmap'? [-Werror=implicit-function-declaration]
463 | pci_iounmap(pdev, vf->mem_resource[i].addr);
| ^~~~~~~~~~~
| pcim_iounmap
drivers/virtio/vdpa/ifcvf/ifcvf_main.c: At top level:
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: warning: data definition has no type or storage class
491 | module_pci_driver(ifcvf_driver);
| ^~~~~~~~~~~~~~~~~
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: error: type defaults to 'int' in declaration of 'module_pci_driver' [-Werror=implicit-int]
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:491:1: warning: parameter names (without types) in function declaration
drivers/virtio/vdpa/ifcvf/ifcvf_main.c:484:26: warning: 'ifcvf_driver' defined but not used [-Wunused-variable]
484 | static struct pci_driver ifcvf_driver = {
| ^~~~~~~~~~~~cc1: some warnings being treated as errors
vim +463 drivers/virtio/vdpa/ifcvf/ifcvf_main.c
452
453 static void ifcvf_remove(struct pci_dev *pdev)
454 {
455 struct ifcvf_adapter *adapter = pci_get_drvdata(pdev);
456 struct ifcvf_hw *vf;
457 int i;
458
459 ifcvf_vdpa_detach(adapter);
460 vf = &adapter->vf;
461 for (i = 0; i < IFCVF_PCI_MAX_RESOURCE; i++) {
462 if (vf->mem_resource[i].addr) {
> 463 pci_iounmap(pdev, vf->mem_resource[i].addr);
464 vf->mem_resource[i].addr = NULL;
465 }
466 }
467
468 ifcvf_destroy_adapter(adapter);
469 pci_free_irq_vectors(pdev);
470 pci_release_regions(pdev);
471 pci_disable_device(pdev);
472 kfree(adapter);
473 }
474
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org