Enhance VHOST to enable SoC-to-SoC communication
Subsystems (12)
cover Enhance VHOST to enable SoC-to-SoC communication
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components
UseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+-----v------+ +------v-------+
| Linux | | Linux |
| Endpoint | | Root Complex |
| <-----------------> |
| | | |
| SOC1 | | SOC2 |
+------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+
| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP | |
| | CONTROLLER1 | | CONTROLLER2 | |
| | <-----------------------------------> | |
| | | | | |
| | | | | |
| | | SoC With Multiple EP Instances | | |
| | | (Configured using NTB Function) | | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+
| RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X |
+-------^--------+ +-----^-----+ +-----^------+ +----^-----+
| | | |
| | | |
| | | |
+-----------v-----------------v--------------v--------------v----------+
| VHOST CORE |
+--------^---------------^--------------------^------------------^-----+
| | | |
| | | |
| | | |
+--------v-------+ +----v------+ +----------v----------+ +----v-----+
| PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X |
+----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
[1] -> https://lore.kernel.org/r/[email protected]
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation
Documentation/driver-api/ntb.rst | 11 +
Documentation/rpmsg.txt | 56 +
drivers/ntb/Kconfig | 18 +
drivers/ntb/Makefile | 2 +
drivers/ntb/ntb_vhost.c | 776 +++++++++++
drivers/ntb/ntb_virtio.c | 853 ++++++++++++
drivers/ntb/ntb_virtio.h | 56 +
drivers/pci/endpoint/functions/Kconfig | 11 +
drivers/pci/endpoint/functions/Makefile | 1 +
.../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++
drivers/rpmsg/Kconfig | 10 +
drivers/rpmsg/Makefile | 3 +-
drivers/rpmsg/rpmsg_cfs.c | 394 ++++++
drivers/rpmsg/rpmsg_core.c | 7 +
drivers/rpmsg/rpmsg_internal.h | 136 ++
drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++
drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++-
drivers/vhost/Kconfig | 1 +
drivers/vhost/Makefile | 2 +-
drivers/vhost/net.c | 10 +-
drivers/vhost/scsi.c | 24 +-
drivers/vhost/test.c | 17 +-
drivers/vhost/vdpa.c | 2 +-
drivers/vhost/vhost.c | 730 ++++++++++-
drivers/vhost/vhost_cfs.c | 341 +++++
drivers/vhost/vringh.c | 332 +++++
drivers/vhost/vsock.c | 20 +-
drivers/virtio/Kconfig | 9 +
drivers/virtio/Makefile | 1 +
drivers/virtio/virtio_pci_common.c | 25 +-
drivers/virtio/virtio_pci_epf.c | 670 ++++++++++
include/linux/mod_devicetable.h | 6 +
include/linux/rpmsg.h | 6 +
{drivers/vhost => include/linux}/vhost.h | 132 +-
include/linux/virtio.h | 3 +
include/linux/virtio_config.h | 42 +
include/linux/vringh.h | 46 +
samples/rpmsg/rpmsg_client_sample.c | 32 +-
tools/virtio/virtio_test.c | 2 +-
39 files changed, 7083 insertions(+), 183 deletions(-)
create mode 100644 drivers/ntb/ntb_vhost.c
create mode 100644 drivers/ntb/ntb_virtio.c
create mode 100644 drivers/ntb/ntb_virtio.h
create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
create mode 100644 drivers/rpmsg/rpmsg_cfs.c
create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
create mode 100644 drivers/vhost/vhost_cfs.c
create mode 100644 drivers/virtio/virtio_pci_epf.c
rename {drivers/vhost => include/linux}/vhost.h (66%)Signature · 1 line
2.17.1
RFC PATCH 1/22 vhost: Make _feature_ bits a property of vhost device
No functional change intended. The feature bits defined in virtio specification is associated with virtio device and not virtqueue. In order to correctly reflect this in the vhost backend, remove "acked_features" from struct vhost_virtqueue and add "features" in struct vhost_dev. This will also make it look symmetrical to virtio in guest.
--- drivers/vhost/net.c | 7 ++++--- drivers/vhost/scsi.c | 22 ++++++++-------------- drivers/vhost/test.c | 14 +++++--------- drivers/vhost/vhost.c | 33 +++++++++++++++++++++------------ drivers/vhost/vhost.h | 6 +++--- drivers/vhost/vsock.c | 18 ++++++------------ 6 files changed, 47 insertions(+), 53 deletions(-)
drivers/vhost/net.c +4 -3
@@ -1137,9 +1137,9 @@ static void handle_rx(struct vhost_net *net) vhost_hlen = nvq->vhost_hlen; sock_hlen = nvq->sock_hlen; - vq_log =unlikely (vhost_has_feature (vq,VHOST_F_LOG_ALL )) ? + vq_log =unlikely (vhost_has_feature (&net->dev,VHOST_F_LOG_ALL )) ? vq->log :NULL ; - mergeable =vhost_has_feature (vq,VIRTIO_NET_F_MRG_RXBUF ); + mergeable =vhost_has_feature (&net->dev,VIRTIO_NET_F_MRG_RXBUF ); do { sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
@@ -1633,6 +1633,7 @@ static int vhost_net_set_backend_features(struct vhost_net *n, u64 features)static int vhost_net_set_features (struct vhost_net *n,u64 features) {size_t vhost_hlen, sock_hlen, hdr_len; +struct vhost_dev *vdev = &n->dev;int i; hdr_len = (features & ((1ULL << VIRTIO_NET_F_MRG_RXBUF) |
@@ -1658,9 +1659,9 @@ static int vhost_net_set_features(struct vhost_net *n, u64 features) goto out_unlock; } + vdev->features = features;for (i =0 ; i <VHOST_NET_VQ_MAX ; ++i) {mutex_lock (&n->vqs[i].vq.mutex); - n->vqs[i].vq.acked_features = features; n->vqs[i].vhost_hlen = vhost_hlen; n->vqs[i].sock_hlen = sock_hlen;mutex_unlock (&n->vqs[i].vq.mutex);
drivers/vhost/scsi.c +8 -14
@@ -921,7 +921,7 @@ vhost_scsi_handle_vq(struct vhost_scsi *vs, struct vhost_virtqueue *vq)int ret, prot_bytes, c =0 ;u16 lun;u8 task_attr; -bool t10_pi =vhost_has_feature (vq,VIRTIO_SCSI_F_T10_PI ); +bool t10_pi =vhost_has_feature (&vs->dev,VIRTIO_SCSI_F_T10_PI );void *cdb;mutex_lock (&vq->mutex);
@@ -1573,26 +1573,20 @@ vhost_scsi_clear_endpoint(struct vhost_scsi *vs,static int vhost_scsi_set_features (struct vhost_scsi *vs,u64 features) { -struct vhost_virtqueue *vq; -int i; +struct vhost_dev *vdev = &vs->dev;if (features & ~VHOST_SCSI_FEATURES )return -EOPNOTSUPP ; -mutex_lock (&vs->dev.mutex); +mutex_lock (&vdev->mutex);if ((features & (1 <<VHOST_F_LOG_ALL )) && - !vhost_log_access_ok (&vs->dev)) { -mutex_unlock (&vs->dev.mutex); + !vhost_log_access_ok (vdev)) { +mutex_unlock (&vdev->mutex);return -EFAULT ; } -for (i =0 ; i <VHOST_SCSI_MAX_VQ ; i++) { - vq = &vs->vqs[i].vq; -mutex_lock (&vq->mutex); - vq->acked_features = features; -mutex_unlock (&vq->mutex); - } -mutex_unlock (&vs->dev.mutex); + vdev->features = features; +mutex_unlock (&vdev->mutex);return 0 ; }
@@ -1789,7 +1783,7 @@ vhost_scsi_do_plug(struct vhost_scsi_tpg *tpg, vq = &vs->vqs[VHOST_SCSI_VQ_EVT ].vq;mutex_lock (&vq->mutex); -if (vhost_has_feature (vq,VIRTIO_SCSI_F_HOTPLUG )) +if (vhost_has_feature (&vs->dev,VIRTIO_SCSI_F_HOTPLUG ))vhost_scsi_send_evt (vs, tpg, lun,VIRTIO_SCSI_T_TRANSPORT_RESET , reason);mutex_unlock (&vq->mutex);
drivers/vhost/test.c +5 -9
@@ -247,19 +247,15 @@ static long vhost_test_reset_owner(struct vhost_test *n)static int vhost_test_set_features (struct vhost_test *n,u64 features) { -struct vhost_virtqueue *vq; +struct vhost_dev *vdev = &n->dev; -mutex_lock (&n->dev.mutex); +mutex_lock (&vdev->mutex);if ((features & (1 <<VHOST_F_LOG_ALL )) && - !vhost_log_access_ok (&n->dev)) { -mutex_unlock (&n->dev.mutex); + !vhost_log_access_ok (vdev)) { +mutex_unlock (&vdev->mutex);return -EFAULT ; } - vq = &n->vqs[VHOST_TEST_VQ ]; -mutex_lock (&vq->mutex); - vq->acked_features = features; -mutex_unlock (&vq->mutex); -mutex_unlock (&n->dev.mutex); +mutex_unlock (&vdev->mutex);return 0 ; }
drivers/vhost/vhost.c +21 -12
@@ -104,12 +104,14 @@ static long vhost_get_vring_endian(struct vhost_virtqueue *vq, u32 idx,static void vhost_init_is_le (struct vhost_virtqueue *vq) { +struct vhost_dev *vdev = vq->dev; +/* Note for legacy virtio: user_be is initialized at reset time * according to the host endianness. If userspace does not set an * explicit endianness, the default behavior is native endian, as * expected by legacy virtio. */ - vq->is_le =vhost_has_feature (vq,VIRTIO_F_VERSION_1 ) || !vq->user_be; + vq->is_le =vhost_has_feature (vdev,VIRTIO_F_VERSION_1 ) || !vq->user_be; }#else static void vhost_disable_cross_endian (struct vhost_virtqueue *vq)
@@ -129,7 +131,9 @@ static long vhost_get_vring_endian(struct vhost_virtqueue *vq, u32 idx,static void vhost_init_is_le (struct vhost_virtqueue *vq) { - vq->is_le =vhost_has_feature (vq,VIRTIO_F_VERSION_1 ) +struct vhost_dev *vdev = vq->dev; + + vq->is_le =vhost_has_feature (vdev,VIRTIO_F_VERSION_1 ) ||virtio_legacy_is_little_endian (); }#endif /* CONFIG_VHOST_CROSS_ENDIAN_LEGACY */
@@ -310,7 +314,6 @@ static void vhost_vq_reset(struct vhost_dev *dev, vq->log_used = false; vq->log_addr =-1ull ; vq->private_data =NULL ; - vq->acked_features =0 ; vq->acked_backend_features =0 ; vq->log_base =NULL ; vq->error_ctx =NULL ;
@@ -428,8 +431,9 @@ EXPORT_SYMBOL_GPL(vhost_exceeds_weight);static size_t vhost_get_avail_size (struct vhost_virtqueue *vq,unsigned int num) { +struct vhost_dev *vdev = vq->dev;size_t event__maybe_unused = -vhost_has_feature (vq,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ; +vhost_has_feature (vdev,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ;return sizeof (*vq->avail) +sizeof (*vq->avail->ring) * num + event;
@@ -438,8 +442,9 @@ static size_t vhost_get_avail_size(struct vhost_virtqueue *vq,static size_t vhost_get_used_size (struct vhost_virtqueue *vq,unsigned int num) { +struct vhost_dev *vdev = vq->dev;size_t event__maybe_unused = -vhost_has_feature (vq,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ; +vhost_has_feature (vdev,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ;return sizeof (*vq->used) +sizeof (*vq->used->ring) * num + event;
@@ -468,6 +473,7 @@ void vhost_dev_init(struct vhost_dev *dev, dev->iotlb =NULL ; dev->mm =NULL ; dev->worker =NULL ; + dev->features =0 ; dev->iov_limit = iov_limit; dev->weight = weight; dev->byte_weight = byte_weight;
@@ -738,14 +744,15 @@ static inline void __user *vhost_vq_meta_fetch(struct vhost_virtqueue *vq,static bool memory_access_ok (struct vhost_dev *d,struct vhost_iotlb *umem,int log_all) { +bool log;int i; + log = log_all ||vhost_has_feature (d,VHOST_F_LOG_ALL ); +for (i =0 ; i < d->nvqs; ++i) {bool ok; -bool log;mutex_lock (&d->vqs[i]->mutex); - log = log_all ||vhost_has_feature (d->vqs[i],VHOST_F_LOG_ALL );/* If ring is inactive, will check when it's enabled. */ if (d->vqs[i]->private_data) ok = vq_memory_access_ok(d->vqs[i]->log_base,
@@ -1329,8 +1336,10 @@ EXPORT_SYMBOL_GPL(vhost_log_access_ok);static bool vq_log_access_ok (struct vhost_virtqueue *vq,void __user *log_base) { +struct vhost_dev *vdev = vq->dev; +return vq_memory_access_ok (log_base, vq->umem, -vhost_has_feature (vq,VHOST_F_LOG_ALL )) && +vhost_has_feature (vdev,VHOST_F_LOG_ALL )) && (!vq->log_used ||log_access_ok (log_base, vq->log_addr,vhost_get_used_size (vq, vq->num))); }
@@ -2376,11 +2385,11 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)* interrupts. */ smp_mb (); -if (vhost_has_feature (vq,VIRTIO_F_NOTIFY_ON_EMPTY ) && +if (vhost_has_feature (dev,VIRTIO_F_NOTIFY_ON_EMPTY ) &&unlikely (vq->avail_idx == vq->last_avail_idx))return true; -if (!vhost_has_feature (vq,VIRTIO_RING_F_EVENT_IDX )) { +if (!vhost_has_feature (dev,VIRTIO_RING_F_EVENT_IDX )) {__virtio16 flags;if (vhost_get_avail_flags (vq, &flags)) {vq_err (vq,"Failed to get flags" );
@@ -2459,7 +2468,7 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)if (!(vq->used_flags &VRING_USED_F_NO_NOTIFY ))return false; vq->used_flags &= ~VRING_USED_F_NO_NOTIFY ; -if (!vhost_has_feature (vq,VIRTIO_RING_F_EVENT_IDX )) { +if (!vhost_has_feature (dev,VIRTIO_RING_F_EVENT_IDX )) { r =vhost_update_used_flags (vq);if (r) { vq_err(vq, "Failed to enable notification at %p: %d\n",
@@ -2496,7 +2505,7 @@ void vhost_disable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)if (vq->used_flags &VRING_USED_F_NO_NOTIFY )return ; vq->used_flags |=VRING_USED_F_NO_NOTIFY ; -if (!vhost_has_feature (vq,VIRTIO_RING_F_EVENT_IDX )) { +if (!vhost_has_feature (dev,VIRTIO_RING_F_EVENT_IDX )) { r =vhost_update_used_flags (vq);if (r) vq_err(vq, "Failed to enable notification at %p: %d\n",
drivers/vhost/vhost.h +3 -3
@@ -111,7 +111,6 @@ struct vhost_virtqueue {struct vhost_iotlb *umem;struct vhost_iotlb *iotlb;void *private_data; -u64 acked_features;u64 acked_backend_features;/* Log write descriptors */ void __user *log_base;
@@ -140,6 +139,7 @@ struct vhost_dev {struct mm_struct *mm;struct mutex mutex;struct vhost_virtqueue **vqs; +u64 features;int nvqs;struct eventfd_ctx *log_ctx;struct llist_head work_list;
@@ -258,9 +258,9 @@ static inline void *vhost_vq_get_backend(struct vhost_virtqueue *vq)return vq->private_data; } -static inline bool vhost_has_feature (struct vhost_virtqueue *vq,int bit) +static inline bool vhost_has_feature (struct vhost_dev *vdev,int bit) { -return vq->acked_features & (1ULL << bit); +return vdev->features & (1ULL << bit); }static inline bool vhost_backend_has_feature (struct vhost_virtqueue *vq,int bit)
drivers/vhost/vsock.c +6 -12
@@ -757,26 +757,20 @@ static int vhost_vsock_set_cid(struct vhost_vsock *vsock, u64 guest_cid)static int vhost_vsock_set_features (struct vhost_vsock *vsock,u64 features) { -struct vhost_virtqueue *vq; -int i; +struct vhost_dev *vdev = &vsock->dev;if (features & ~VHOST_VSOCK_FEATURES )return -EOPNOTSUPP ; -mutex_lock (&vsock->dev.mutex); +mutex_lock (&vdev->mutex);if ((features & (1 <<VHOST_F_LOG_ALL )) && - !vhost_log_access_ok (&vsock->dev)) { -mutex_unlock (&vsock->dev.mutex); + !vhost_log_access_ok (vdev)) { +mutex_unlock (&vdev->mutex);return -EFAULT ; } -for (i =0 ; i <ARRAY_SIZE (vsock->vqs); i++) { - vq = &vsock->vqs[i]; -mutex_lock (&vq->mutex); - vq->acked_features = features; -mutex_unlock (&vq->mutex); - } -mutex_unlock (&vsock->dev.mutex); + vdev->features = features; +mutex_unlock (&vdev->mutex);return 0 ; }
Signature · 1 line
2.17.1
RFC PATCH 2/22 vhost: Introduce standard Linux driver model in VHOST
In reply to Kishon Vijay Abraham IIntroduce standard driver model in VHOST. This will facilitate using multiple VHOST drivers (like net, scsi etc.,) over different VHOST devices using MMIO (like PCIe or NTB), using kernel pointers (like platform devices) or using userspace pointers.
---
drivers/vhost/net.c | 3 +-
drivers/vhost/scsi.c | 2 +-
drivers/vhost/test.c | 3 +-
drivers/vhost/vdpa.c | 2 +-
drivers/vhost/vhost.c | 157 ++++++++++++++++++++++-
drivers/vhost/vsock.c | 2 +-
include/linux/mod_devicetable.h | 6 +
{drivers/vhost => include/linux}/vhost.h | 22 +++-
tools/virtio/virtio_test.c | 2 +-
9 files changed, 190 insertions(+), 9 deletions(-)
rename {drivers/vhost => include/linux}/vhost.h (93%)drivers/vhost/net.c +2 -1
@@ -18,6 +18,7 @@#include <linux/slab.h> #include <linux/sched/clock.h> #include <linux/sched/signal.h> +#include <linux/vhost.h> #include <linux/vmalloc.h> #include <linux/net.h>
@@ -33,7 +34,7 @@#include <net/sock.h> #include <net/xdp.h> -#include "vhost.h" +#include <uapi/linux/vhost.h> static int experimental_zcopytx =0 ;module_param (experimental_zcopytx, int,0444 );
drivers/vhost/scsi.c +1 -1
@@ -47,7 +47,7 @@#include <linux/llist.h> #include <linux/bitmap.h> -#include "vhost.h" +#include <uapi/linux/vhost.h> #define VHOST_SCSI_VERSION "v0.1"#define VHOST_SCSI_NAMELEN 256
drivers/vhost/test.c +2 -1
@@ -14,9 +14,10 @@#include <linux/workqueue.h> #include <linux/file.h> #include <linux/slab.h> +#include <linux/vhost.h> +#include <uapi/linux/vhost.h> #include "test.h" -#include "vhost.h" /* Max number of bytes transferred before requeueing the job. * Using this limit prevents one virtqueue from starving others. */
drivers/vhost/vdpa.c +1 -1
@@ -22,7 +22,7 @@#include <linux/vhost.h> #include <linux/virtio_net.h> -#include "vhost.h" +#include <uapi/linux/vhost.h> enum { VHOST_VDPA_FEATURES =
drivers/vhost/vhost.c +155 -2
183 lines not on this page · raw message
drivers/vhost/vsock.c +1 -1
@@ -17,7 +17,7 @@#include <linux/hashtable.h> #include <net/af_vsock.h> -#include "vhost.h" +#include <uapi/linux/vhost.h> #define VHOST_VSOCK_DEFAULT_HOST_CID 2/* Max number of bytes transferred before requeueing the job.
include/linux/mod_devicetable.h +6 -0
@@ -430,6 +430,12 @@ struct virtio_device_id { };#define VIRTIO_DEV_ANY_ID 0xffffffff +/* VHOST */ +struct vhost_device_id { +__u32 device; +__u32 vendor; +}; +/* * For Hyper-V devices we use the device guid as the id. */
include/linux/vhost.h renamed from drivers/vhost/vhost.h +21 -1
@@ -3,7 +3,6 @@#define _VHOST_H#include <linux/eventfd.h> -#include <linux/vhost.h> #include <linux/mm.h> #include <linux/mutex.h> #include <linux/poll.h>
@@ -13,6 +12,7 @@#include <linux/virtio_ring.h> #include <linux/atomic.h> #include <linux/vhost_iotlb.h> +#include <uapi/linux/vhost.h> struct vhost_work ;typedef void (*vhost_work_fn_t )(struct vhost_work *work);
@@ -135,7 +135,20 @@ struct vhost_msg_node {struct list_head node; }; +struct vhost_driver { +struct device_driver driver; +struct vhost_device_id *id_table; +int (*probe)(struct vhost_dev *dev); +int (*remove)(struct vhost_dev *dev); +}; + +#define to_vhost_driver (drv) (container_of((drv), struct vhost_driver, driver)) +struct vhost_dev { +struct device dev; +struct vhost_driver *driver; +struct vhost_device_id id; +int index;struct mm_struct *mm;struct mutex mutex;struct vhost_virtqueue **vqs;
@@ -158,6 +171,13 @@ struct vhost_dev { struct vhost_iotlb_msg *msg); }; +#define to_vhost_dev (d) container_of((d), struct vhost_dev, dev) + +int vhost_register_driver (struct vhost_driver *driver); +void vhost_unregister_driver (struct vhost_driver *driver); +int vhost_register_device (struct vhost_dev *vdev); +void vhost_unregister_device (struct vhost_dev *vdev); +bool vhost_exceeds_weight (struct vhost_virtqueue *vq,int pkts,int total_len); void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs, int nvqs, int iov_limit, int weight, int byte_weight,
tools/virtio/virtio_test.c +1 -1
@@ -13,9 +13,9 @@#include <fcntl.h> #include <stdbool.h> #include <linux/virtio_types.h> -#include <linux/vhost.h> #include <linux/virtio.h> #include <linux/virtio_ring.h> +#include <uapi/linux/vhost.h> #include "../../drivers/vhost/test.h" /* Unused */
Signature · 1 line
2.17.1
RFC PATCH 3/22 vhost: Add ops for the VHOST driver to configure VHOST device
In reply to Kishon Vijay Abraham IAdd "vhost_config_ops" in *struct vhost_driver* for the VHOST driver to configure VHOST device and add facility for VHOST device to notify VHOST driver (whenever VIRTIO sets a new status or finalize features). This is in preparation to use the same vhost_driver across different VHOST devices (like PCIe or NTB or platform device).
--- drivers/vhost/vhost.c | 185 ++++++++++++++++++++++++++++++++++++++++++ include/linux/vhost.h | 57 +++++++++++++ 2 files changed, 242 insertions(+)
drivers/vhost/vhost.c +185 -0
@@ -2558,6 +2558,190 @@ struct vhost_msg_node *vhost_dequeue_msg(struct vhost_dev *dev, }EXPORT_SYMBOL_GPL (vhost_dequeue_msg); +/** +* vhost_create_vqs() - Invoke vhost_config_ops to create virtqueue +* @vdev: Vhost device that provides create_vqs() callback to create virtqueue +* @nvqs: Number of vhost virtqueues to be created +* @num_bufs: The number of buffers that should be supported by the vhost +* virtqueue (number of descriptors in the vhost virtqueue) +* @vqs: Pointers to all the created vhost virtqueues +* @callback: Callback function associated with the virtqueue +* @names: Names associated with each virtqueue +* +* Wrapper that invokes vhost_config_ops to create virtqueue. +*/ +int vhost_create_vqs (struct vhost_dev *vdev,unsigned int nvqs, +unsigned int num_bufs,struct vhost_virtqueue *vqs[], +vhost_vq_callback_t *callbacks[], +const char *const names[]) +{ +int ret; + +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->create_vqs) +return -EINVAL ; + +mutex_lock (&vdev->mutex); + ret = vdev->ops->create_vqs (vdev, nvqs, num_bufs, vqs, callbacks, + names); +mutex_unlock (&vdev->mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (vhost_create_vqs); + +/* vhost_del_vqs - Invoke vhost_config_ops to delete the created virtqueues +* @vdev: Vhost device that provides del_vqs() callback to delete virtqueue +* +* Wrapper that invokes vhost_config_ops to delete all the virtqueues +* associated with the vhost device. +*/ +void vhost_del_vqs (struct vhost_dev *vdev) +{ +if (IS_ERR_OR_NULL (vdev)) +return ; + +if (!vdev->ops && !vdev->ops->del_vqs) +return ; + +mutex_lock (&vdev->mutex); + vdev->ops->del_vqs (vdev); +mutex_unlock (&vdev->mutex); +} +EXPORT_SYMBOL_GPL (vhost_del_vqs); + +/* vhost_write - Invoke vhost_config_ops to write data to buffer provided +* by remote virtio driver +* @vdev: Vhost device that provides write() callback to write data +* @dst: Buffer address in the remote device provided by the remote virtio +* driver +* @src: Buffer address in the local device provided by the vhost client driver +* @len: Length of the data to be copied from @src to @dst +* +* Wrapper that invokes vhost_config_ops to write data to buffer provided by +* remote virtio driver from buffer provided by vhost client driver. +*/ +int vhost_write (struct vhost_dev *vdev,u64 vhost_dst,void *src,int len) +{ +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->write) +return -EINVAL ; + +return vdev->ops->write (vdev, vhost_dst, src, len); +} +EXPORT_SYMBOL_GPL (vhost_write); + +/* vhost_read - Invoke vhost_config_ops to read data from buffers provided by +* remote virtio driver +* @vdev: Vhost device that provides read() callback to read data +* @dst: Buffer address in the local device provided by the vhost client driver +* @src: Buffer address in the remote device provided by the remote virtio +* driver +* @len: Length of the data to be copied from @src to @dst +* +* Wrapper that invokes vhost_config_ops to read data from buffers provided by +* remote virtio driver to the address provided by vhost client driver. +*/ +int vhost_read (struct vhost_dev *vdev,void *dst,u64 vhost_src,int len) +{ +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->read) +return -EINVAL ; + +return vdev->ops->read (vdev, dst, vhost_src, len); +} +EXPORT_SYMBOL_GPL (vhost_read); + +/* vhost_set_status - Invoke vhost_config_ops to set vhost device status +* @vdev: Vhost device that provides set_status() callback to set device status +* @status: Vhost device status configured by vhost client driver +* +* Wrapper that invokes vhost_config_ops to set vhost device status. +*/ +int vhost_set_status (struct vhost_dev *vdev,u8 status) +{ +int ret; + +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->set_status) +return -EINVAL ; + +mutex_lock (&vdev->mutex); + ret = vdev->ops->set_status (vdev, status); +mutex_unlock (&vdev->mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (vhost_set_status); + +/* vhost_get_status - Invoke vhost_config_ops to get vhost device status +* @vdev: Vhost device that provides get_status() callback to get device status +* +* Wrapper that invokes vhost_config_ops to get vhost device status. +*/ +u8 vhost_get_status (struct vhost_dev *vdev) +{ +u8 status; + +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->get_status) +return -EINVAL ; + +mutex_lock (&vdev->mutex); + status = vdev->ops->get_status (vdev); +mutex_unlock (&vdev->mutex); + +return status; +} +EXPORT_SYMBOL_GPL (vhost_get_status); + +/* vhost_set_features - Invoke vhost_config_ops to set vhost device features +* @vdev: Vhost device that provides set_features() callback to set device +* features +* +* Wrapper that invokes vhost_config_ops to set device features. +*/ +int vhost_set_features (struct vhost_dev *vdev,u64 device_features) +{ +int ret; + +if (IS_ERR_OR_NULL (vdev)) +return -EINVAL ; + +if (!vdev->ops && !vdev->ops->set_features) +return -EINVAL ; + +mutex_lock (&vdev->mutex); + ret = vdev->ops->set_features (vdev, device_features); +mutex_unlock (&vdev->mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (vhost_set_features); + +/* vhost_register_notifier - Register notifier to receive notification from +* vhost device +* @vdev: Vhost device from which notification has to be received. +* @nb: Notifier block holding the callback function +* +* Invoked by vhost client to receive notification from vhost device. +*/ +int vhost_register_notifier (struct vhost_dev *vdev,struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&vdev->notifier, nb); +} +EXPORT_SYMBOL_GPL (vhost_register_notifier); +static inline int vhost_id_match (const struct vhost_dev *vdev,const struct vhost_device_id *id) {
@@ -2669,6 +2853,7 @@ int vhost_register_device(struct vhost_dev *vdev)device_initialize (dev);dev_set_name (dev,"vhost%u" , ret); +BLOCKING_INIT_NOTIFIER_HEAD (&vdev->notifier); ret =device_add (dev);if (ret) {
include/linux/vhost.h +57 -0
80 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 4/22 vringh: Add helpers to access vring in MMIO
In reply to Kishon Vijay Abraham IAdd helpers to access vring in memory mapped IO. This would require using IO accessors to access vring and the addresses populated by virtio driver (in the descriptor) should be directly given to the vhost client driver. Even if the vhost runs in 32 bit system, it can access 64 bit address provided by the virtio if the vhost device supports translation.
This is in preparation for adding VHOST devices (PCIe Endpoint or Host in NTB) to access vrings created by VIRTIO devices (PCIe RC or Host in NTB) over memory mapped IO.
--- drivers/vhost/vringh.c | 332 +++++++++++++++++++++++++++++++++++++++++ include/linux/vringh.h | 46 ++++++ 2 files changed, 378 insertions(+)
drivers/vhost/vringh.c +332 -0
@@ -5,6 +5,7 @@* Since these may be in userspace, we use (inline) accessors. */ #include <linux/compiler.h> +#include <linux/io.h> #include <linux/module.h> #include <linux/vringh.h> #include <linux/virtio_ring.h>
@@ -188,6 +189,32 @@ static int move_to_indirect(const struct vringh *vrh,return 0 ; } +static int resize_mmiovec (struct vringh_mmiov *iov,gfp_t gfp) +{ +unsigned int flag, new_num = (iov->max_num & ~VRINGH_IOV_ALLOCATED ) *2 ; +struct mmiovec *new; + +if (new_num <8 ) + new_num =8 ; + + flag = (iov->max_num &VRINGH_IOV_ALLOCATED ); +if (flag) { + new =krealloc (iov->iov, new_num *sizeof (struct iovec ), gfp); + }else { + new =kmalloc_array (new_num,sizeof (struct iovec ), gfp); +if (new) { +memcpy (new, iov->iov, + iov->max_num *sizeof (struct iovec )); + flag =VRINGH_IOV_ALLOCATED ; + } + } +if (!new) +return -ENOMEM ; + iov->iov = new; + iov->max_num = (new_num | flag); +return 0 ; +} +static int resize_iovec (struct vringh_kiov *iov,gfp_t gfp) {struct kvec *new;
@@ -261,6 +288,142 @@ static int slow_copy(struct vringh *vrh, void *dst, const void *src,return 0 ; } +static inline int +__vringh_mmiov (struct vringh *vrh,u16 i,struct vringh_mmiov *riov, +struct vringh_mmiov *wiov, +bool (*rcheck)(struct vringh *vrh,u64 addr,size_t *len, +struct vringh_range *range, +bool (*getrange)(struct vringh *,u64 , +struct vringh_range *)), +bool (*getrange)(struct vringh *,u64 ,struct vringh_range *), +gfp_t gfp, +int (*copy)(const struct vringh *vrh, +void *dst,const void *src,size_t len)) +{ +int err, count =0 , up_next, desc_max; +struct vring_desc desc, *descs; +struct vringh_range range = {-1ULL ,0 }, slowrange; +bool slow = false; + +/* We start traversing vring's descriptor table. */ + descs = vrh->vring.desc; + desc_max = vrh->vring.num; + up_next =-1 ; + +if (riov) { + riov->i =0 ; + riov->used =0 ; + }else if (wiov) { + wiov->i =0 ; + wiov->used =0 ; + }else { +/* You must want something! */ +WARN_ON (1 ); + } + +for (;;) { +u64 addr; +struct vringh_mmiov *iov; +size_t len; + +if (unlikely (slow)) + err =slow_copy (vrh, &desc, &descs[i], rcheck, getrange, + &slowrange, copy); +else + err =copy (vrh, &desc, &descs[i],sizeof (desc)); +if (unlikely (err)) + goto fail; + +if (unlikely (desc.flags & +cpu_to_vringh16 (vrh,VRING_DESC_F_INDIRECT ))) { +/* VRING_DESC_F_INDIRECT is not supported */ + err = -EINVAL ; + goto fail; + } + +if (count++ == vrh->vring.num) { +vringh_bad ("Descriptor loop in %p" , descs); + err = -ELOOP ; + goto fail; + } + +if (desc.flags &cpu_to_vringh16 (vrh,VRING_DESC_F_WRITE )) { + iov = wiov; + }else { + iov = riov; +if (unlikely (wiov && wiov->i)) { +vringh_bad ("Readable desc %p after writable" , + &descs[i]); + err = -EINVAL ; + goto fail; + } + } + +if (!iov) { +vringh_bad ("Unexpected %s desc" , + !wiov ?"writable" :"readable" ); + err = -EPROTO ; + goto fail; + } + +again: +/* Make sure it's OK, and get offset. */ + len =vringh32_to_cpu (vrh, desc.len); +if (!rcheck (vrh,vringh64_to_cpu (vrh, desc.addr), &len, &range, + getrange)) { + err = -EINVAL ; + goto fail; + } + addr =vringh64_to_cpu (vrh, desc.addr) + range.offset; + +if (unlikely (iov->used == (iov->max_num & ~VRINGH_IOV_ALLOCATED ))) { + err =resize_mmiovec (iov, gfp); +if (err) + goto fail; + } + + iov->iov[iov->used].iov_base = addr; + iov->iov[iov->used].iov_len = len; + iov->used++; + +if (unlikely (len !=vringh32_to_cpu (vrh, desc.len))) { + desc.len = +cpu_to_vringh32 (vrh, +vringh32_to_cpu (vrh, desc.len) + - len); + desc.addr = +cpu_to_vringh64 (vrh, +vringh64_to_cpu (vrh, desc.addr) + + len); + goto again; + } + +if (desc.flags &cpu_to_vringh16 (vrh,VRING_DESC_F_NEXT )) { + i =vringh16_to_cpu (vrh, desc.next); + }else { +/* Just in case we need to finish traversing above. */ +if (unlikely (up_next >0 )) { + i =return_from_indirect (vrh, &up_next, + &descs, &desc_max); + slow = false; + }else { +break ; + } + } + +if (i >= desc_max) { +vringh_bad ("Chained index %u > %u" , i, desc_max); + err = -EINVAL ; + goto fail; + } + } + +return 0 ; + +fail: +return err; +} + static inline int __vringh_iov(struct vringh *vrh, u16 i, struct vringh_kiov *riov,
@@ -833,6 +996,175 @@ int vringh_need_notify_user(struct vringh *vrh) }EXPORT_SYMBOL (vringh_need_notify_user); +/* MMIO access helpers */ +static inline int getu16_mmio (const struct vringh *vrh, +u16 *val,const __virtio16 *p) +{ + *val =vringh16_to_cpu (vrh,readw (p)); +return 0 ; +} + +static inline int putu16_mmio (const struct vringh *vrh,__virtio16 *p,u16 val) +{ +writew (cpu_to_vringh16 (vrh, val), p); +return 0 ; +} + +static inline int copydesc_mmio (const struct vringh *vrh, +void *dst,const void *src,size_t len) +{ +memcpy_fromio (dst, src, len); +return 0 ; +} + +static inline int putused_mmio (const struct vringh *vrh, +struct vring_used_elem *dst, +const struct vring_used_elem *src, +unsigned int num) +{ +memcpy_toio (dst, src, num *sizeof (*dst)); +return 0 ; +} + +/** +* vringh_init_mmio - initialize a vringh for a MMIO vring. +* @vrh: the vringh to initialize. +* @features: the feature bits for this ring. +* @num: the number of elements. +* @weak_barriers: true if we only need memory barriers, not I/O. +* @desc: the userpace descriptor pointer. +* @avail: the userpace avail pointer. +* @used: the userpace used pointer. +* +* Returns an error if num is invalid. +*/ +int vringh_init_mmio (struct vringh *vrh,u64 features, +unsigned int num,bool weak_barriers, +struct vring_desc *desc, +struct vring_avail *avail, +struct vring_used *used) +{ +/* Sane power of 2 please! */ +if (!num || num >0xffff || (num & (num -1 ))) { +vringh_bad ("Bad ring size %u" , num); +return -EINVAL ; + } + + vrh->little_endian = (features & (1ULL <<VIRTIO_F_VERSION_1 )); + vrh->event_indices = (features & (1 <<VIRTIO_RING_F_EVENT_IDX )); + vrh->weak_barriers = weak_barriers; + vrh->completed =0 ; + vrh->last_avail_idx =0 ; + vrh->last_used_idx =0 ; + vrh->vring.num = num; + vrh->vring.desc = desc; + vrh->vring.avail = avail; + vrh->vring.used = used; +return 0 ; +} +EXPORT_SYMBOL (vringh_init_mmio); + +/** +* vringh_getdesc_mmio - get next available descriptor from MMIO ring. +* @vrh: the MMIO vring. +* @riov: where to put the readable descriptors (or NULL) +* @wiov: where to put the writable descriptors (or NULL) +* @head: head index we received, for passing to vringh_complete_mmio(). +* @gfp: flags for allocating larger riov/wiov. +* +* Returns 0 if there was no descriptor, 1 if there was, or -errno. +* +* Note that on error return, you can tell the difference between an +* invalid ring and a single invalid descriptor: in the former case, +* *head will be vrh->vring.num. You may be able to ignore an invalid +* descriptor, but there's not much you can do with an invalid ring. +* +* Note that you may need to clean up riov and wiov, even on error! +*/ +int vringh_getdesc_mmio (struct vringh *vrh, +struct vringh_mmiov *riov, +struct vringh_mmiov *wiov, +u16 *head, +gfp_t gfp) +{ +int err; + + err =__vringh_get_head (vrh, getu16_mmio, &vrh->last_avail_idx); +if (err <0 ) +return err; + +/* Empty... */ +if (err == vrh->vring.num) +return 0 ; + + *head = err; + err =__vringh_mmiov (vrh, *head, riov, wiov, no_range_check,NULL , + gfp, copydesc_mmio); +if (err) +return err; + +return 1 ; +} +EXPORT_SYMBOL (vringh_getdesc_mmio); + +/** +* vringh_complete_mmio - we've finished with descriptor, publish it. +* @vrh: the vring. +* @head: the head as filled in by vringh_getdesc_mmio. +* @len: the length of data we have written. +* +* You should check vringh_need_notify_mmio() after one or more calls +* to this function. +*/ +int vringh_complete_mmio (struct vringh *vrh,u16 head,u32 len) +{ +struct vring_used_elem used; + + used.id =cpu_to_vringh32 (vrh, head); + used.len =cpu_to_vringh32 (vrh, len); + +return __vringh_complete (vrh, &used,1 , putu16_mmio, putused_mmio); +} +EXPORT_SYMBOL (vringh_complete_mmio); + +/** +* vringh_notify_enable_mmio - we want to know if something changes. +* @vrh: the vring. +* +* This always enables notifications, but returns false if there are +* now more buffers available in the vring. +*/ +bool vringh_notify_enable_mmio (struct vringh *vrh) +{ +return __vringh_notify_enable (vrh, getu16_mmio, putu16_mmio); +} +EXPORT_SYMBOL (vringh_notify_enable_mmio); + +/** +* vringh_notify_disable_mmio - don't tell us if something changes. +* @vrh: the vring. +* +* This is our normal running state: we disable and then only enable when +* we're going to sleep. +*/ +void vringh_notify_disable_mmio (struct vringh *vrh) +{ +__vringh_notify_disable (vrh, putu16_mmio); +} +EXPORT_SYMBOL (vringh_notify_disable_mmio); + +/** +* vringh_need_notify_mmio - must we tell the other side about used buffers? +* @vrh: the vring we've called vringh_complete_mmio() on. +* +* Returns -errno or 0 if we don't need to tell the other side, 1 if we do. +*/ +int vringh_need_notify_mmio (struct vringh *vrh) +{ +return __vringh_need_notify (vrh, getu16_mmio); +} +EXPORT_SYMBOL (vringh_need_notify_mmio); +/* Kernelspace access helpers. */ static inline int getu16_kern (const struct vringh *vrh,u16 *val,const __virtio16 *p)
include/linux/vringh.h +46 -0
58 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 5/22 vhost: Add MMIO helpers for operations on vhost virtqueue
In reply to Kishon Vijay Abraham IAdd helpers for VHOST drivers to read descriptor data from vhost_virtqueue for IN transfers or write descriptor data to vhost_virtqueue for OUT transfers respectively. Also add helpers to enable callback, disable callback and notify remote virtio for events on virtqueue.
This adds helpers only for virtqueue in MMIO (helpers for virtqueue in kernel space and user space can be added later).
--- drivers/vhost/Kconfig | 1 + drivers/vhost/vhost.c | 292 ++++++++++++++++++++++++++++++++++++++++++ include/linux/vhost.h | 22 ++++ 3 files changed, 315 insertions(+)
drivers/vhost/Kconfig +1 -0
@@ -24,6 +24,7 @@ config VHOST_DPN config VHOST tristate + select VHOST_RING select VHOST_IOTLB help This option is selected by any driver which needs to access
drivers/vhost/vhost.c +292 -0
298 lines not on this page · raw message
include/linux/vhost.h +22 -0
@@ -10,6 +10,7 @@#include <linux/uio.h> #include <linux/virtio_config.h> #include <linux/virtio_ring.h> +#include <linux/vringh.h> #include <linux/atomic.h> #include <linux/vhost_iotlb.h> #include <uapi/linux/vhost.h>
@@ -60,9 +61,20 @@ enum vhost_uaddr_type { VHOST_NUM_ADDRS = 3, }; +enum vhost_type { +VHOST_TYPE_UNKNOWN , +VHOST_TYPE_USER , +VHOST_TYPE_KERN , +VHOST_TYPE_MMIO , +}; +/* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue {struct vhost_dev *dev; +enum vhost_type type; +struct vringh vringh; +void (*callback)(struct vhost_virtqueue *vq); +void (*notify)(struct vhost_virtqueue *vq);/* The actual ring of buffers. */ struct mutex mutex;
@@ -235,6 +247,16 @@ u8 vhost_get_status(struct vhost_dev *vdev);int vhost_register_notifier (struct vhost_dev *vdev,struct notifier_block *nb); +u64 vhost_virtqueue_get_outbuf (struct vhost_virtqueue *vq,u16 *head,int *len); +u64 vhost_virtqueue_get_inbuf (struct vhost_virtqueue *vq,u16 *head,int *len); +void vhost_virtqueue_put_buf (struct vhost_virtqueue *vq,u16 head,int len); + +void vhost_virtqueue_disable_cb (struct vhost_virtqueue *vq); +bool vhost_virtqueue_enable_cb (struct vhost_virtqueue *vq); +void vhost_virtqueue_notify (struct vhost_virtqueue *vq); +void vhost_virtqueue_kick (struct vhost_virtqueue *vq); +void vhost_virtqueue_callback (struct vhost_virtqueue *vq); +bool vhost_exceeds_weight (struct vhost_virtqueue *vq,int pkts,int total_len); void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs, int nvqs, int iov_limit, int weight, int byte_weight,
Signature · 1 line
2.17.1
RFC PATCH 6/22 vhost: Introduce configfs entry for configuring VHOST
In reply to Kishon Vijay Abraham ICreate a configfs entry for each entry populated in
"struct vhost_device_id" in the VHOST driver and create a configfs entry for each VHOST device. This is used to link VHOST driver to VHOST device (by assigning deviceID and vendorID to the VHOST device) and register VHOST device, thereby letting VHOST client driver to be selected in the userspace.
--- drivers/vhost/Makefile | 2 +- drivers/vhost/vhost.c | 63 +++++++ drivers/vhost/vhost_cfs.c | 354 ++++++++++++++++++++++++++++++++++++++ include/linux/vhost.h | 11 ++ 4 files changed, 429 insertions(+), 1 deletion(-) create mode 100644 drivers/vhost/vhost_cfs.c
drivers/vhost/Makefile +1 -1
@@ -13,7 +13,7 @@ obj-$(CONFIG_VHOST_RING) += vringh.o obj-$(CONFIG_VHOST_VDPA) += vhost_vdpa.o vhost_vdpa-y := vdpa.o -obj-$(CONFIG_VHOST) += vhost.o +obj-$(CONFIG_VHOST) += vhost.o vhost_cfs.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o vhost_iotlb-y := iotlb.o
drivers/vhost/vhost.c +63 -0
81 lines not on this page · raw message
drivers/vhost/vhost_cfs.c new file +354 -0
354 lines not on this page · raw message
include/linux/vhost.h +11 -0
@@ -2,6 +2,7 @@#ifndef _VHOST_H#define _VHOST_H +#include <linux/configfs.h> #include <linux/eventfd.h> #include <linux/mm.h> #include <linux/mutex.h>
@@ -181,6 +182,7 @@ struct vhost_config_ops {struct vhost_driver {struct device_driver driver;struct vhost_device_id *id_table; +struct config_group *group;int (*probe)(struct vhost_dev *dev);int (*remove)(struct vhost_dev *dev); };
@@ -233,6 +235,15 @@ void vhost_unregister_driver(struct vhost_driver *driver);int vhost_register_device (struct vhost_dev *vdev);void vhost_unregister_device (struct vhost_dev *vdev); +struct config_group *vhost_cfs_add_driver_group (const char *name); +void vhost_cfs_remove_driver_group (struct config_group *group); +struct config_group * +vhost_cfs_add_driver_item (struct config_group *driver_group,u32 vendor, +u32 device); +void vhost_cfs_remove_driver_item (struct config_group *group); +struct config_group *vhost_cfs_add_device_item (struct vhost_dev *vdev); +void vhost_cfs_remove_device_item (struct config_group *group); + int vhost_create_vqs(struct vhost_dev *vdev, unsigned int nvqs, unsigned int num_bufs, struct vhost_virtqueue *vqs[], vhost_vq_callback_t *callbacks[],
Signature · 1 line
2.17.1
RFC PATCH 7/22 virtio_pci: Use request_threaded_irq() instead of request_irq()
In reply to Kishon Vijay Abraham ISome of the virtio drivers (like virtio_rpmsg_bus.c) use sleeping functions like mutex_*() in the virtqueue callback. Use request_threaded_irq() instead of request_irq() in order for the virtqueue callbacks to be executed in thread context instead of interrupt context.
--- drivers/virtio/virtio_pci_common.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_common.c +13 -12
@@ -140,9 +140,9 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v = vp_dev->msix_used_vectors;snprintf (vp_dev->msix_names[v],sizeof *vp_dev->msix_names,"%s-config" , name); - err =request_irq (pci_irq_vector (vp_dev->pci_dev, v), - vp_config_changed,0 , vp_dev->msix_names[v], - vp_dev); + err =request_threaded_irq (pci_irq_vector (vp_dev->pci_dev, v),0 , + vp_config_changed,0 , vp_dev->msix_names[v], + vp_dev);if (err) goto error; ++vp_dev->msix_used_vectors;
@@ -159,9 +159,9 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v = vp_dev->msix_used_vectors;snprintf (vp_dev->msix_names[v],sizeof *vp_dev->msix_names,"%s-virtqueues" , name); - err =request_irq (pci_irq_vector (vp_dev->pci_dev, v), - vp_vring_interrupt,0 , vp_dev->msix_names[v], - vp_dev); + err =request_threaded_irq (pci_irq_vector (vp_dev->pci_dev, v), +0 , vp_vring_interrupt,0 , + vp_dev->msix_names[v], vp_dev);if (err) goto error; ++vp_dev->msix_used_vectors;
@@ -336,10 +336,11 @@ static int vp_find_vqs_msix(struct virtio_device *vdev, unsigned nvqs,sizeof *vp_dev->msix_names,"%s-%s" , dev_name(&vp_dev->vdev.dev), names[i]); - err =request_irq (pci_irq_vector (vp_dev->pci_dev, msix_vec), - vring_interrupt,0 , - vp_dev->msix_names[msix_vec], - vqs[i]); + err =request_threaded_irq (pci_irq_vector (vp_dev->pci_dev, + msix_vec), +0 , vring_interrupt,0 , + vp_dev->msix_names[msix_vec], + vqs[i]);if (err) goto error_find; }
@@ -361,8 +362,8 @@ static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned nvqs,if (!vp_dev->vqs)return -ENOMEM ; - err =request_irq (vp_dev->pci_dev->irq, vp_interrupt,IRQF_SHARED , -dev_name (&vdev->dev), vp_dev); + err =request_threaded_irq (vp_dev->pci_dev->irq,0 , vp_interrupt, +IRQF_SHARED ,dev_name (&vdev->dev), vp_dev);if (err) goto out_del_vqs;
Signature · 1 line
2.17.1
RFC PATCH 8/22 rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
In reply to Kishon Vijay Abraham ISince rpmsg_recv_done() reads messages in a while loop, disable callbacks until the while loop exits. This helps to get rid of the annoying "uhm, incoming signal, but no used buffer ?" message.
--- drivers/rpmsg/virtio_rpmsg_bus.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
drivers/rpmsg/virtio_rpmsg_bus.c +14 -0
@@ -777,6 +777,7 @@ static void rpmsg_recv_done(struct virtqueue *rvq)return ; } +virtqueue_disable_cb (rvq);while (msg) { err =rpmsg_recv_single (vrp, dev, msg, len);if (err)
@@ -786,6 +787,19 @@ static void rpmsg_recv_done(struct virtqueue *rvq) msg =virtqueue_get_buf (rvq, &len); } +virtqueue_enable_cb (rvq); + +/* +* Try to read message one more time in case a new message is submitted +* after virtqueue_get_buf() inside the while loop but before enabling +* callbacks +*/ + msg =virtqueue_get_buf (rvq, &len); +if (msg) { + err =rpmsg_recv_single (vrp, dev, msg, len); +if (!err) + msgs_received++; + }dev_dbg (dev,"Received %u messages\n" , msgs_received);
Signature · 1 line
2.17.1
RFC PATCH 9/22 rpmsg: Introduce configfs entry for configuring rpmsg
In reply to Kishon Vijay Abraham ICreate a configfs entry for each "struct rpmsg_device_id" populated in rpmsg client driver and create a configfs entry for each rpmsg device. This will be used to bind a rpmsg client driver to a rpmsg device in order to create a new rpmsg channel.
This is used for creating channel for VHOST based rpmsg bus (channels are created in VIRTIO based bus during namespace announcement).
--- drivers/rpmsg/Makefile | 2 +- drivers/rpmsg/rpmsg_cfs.c | 394 +++++++++++++++++++++++++++++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 16 ++ include/linux/rpmsg.h | 5 + 5 files changed, 423 insertions(+), 1 deletion(-) create mode 100644 drivers/rpmsg/rpmsg_cfs.c
drivers/rpmsg/Makefile +1 -1
@@ -1,5 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_RPMSG) += rpmsg_core.o +obj-$(CONFIG_RPMSG) += rpmsg_core.o rpmsg_cfs.o obj-$(CONFIG_RPMSG_CHAR) += rpmsg_char.o obj-$(CONFIG_RPMSG_MTK_SCP) += mtk_rpmsg.o obj-$(CONFIG_RPMSG_QCOM_GLINK_RPM) += qcom_glink_rpm.o
drivers/rpmsg/rpmsg_cfs.c new file +394 -0
394 lines not on this page · raw message
drivers/rpmsg/rpmsg_core.c +7 -0
@@ -563,8 +563,15 @@ EXPORT_SYMBOL(rpmsg_unregister_device);*/ int __register_rpmsg_driver (struct rpmsg_driver *rpdrv,struct module *owner) { +const struct rpmsg_device_id *ids = rpdrv->id_table; rpdrv->drv.bus = &rpmsg_bus; rpdrv->drv.owner = owner; + +while (ids && ids->name[0 ]) { +rpmsg_cfs_add_channel_group (ids->name); + ids++; + } +return driver_register (&rpdrv->drv); }EXPORT_SYMBOL (__register_rpmsg_driver);
drivers/rpmsg/rpmsg_internal.h +16 -0
@@ -68,6 +68,18 @@ struct rpmsg_endpoint_ops { poll_table *wait); }; +/** +* struct rpmsg_virtproc_ops - indirection table for rpmsg_virtproc operations +* @create_channel: Create a new rpdev channel +* @delete_channel: Delete the rpdev channel +* @owner: Owner of the module holding the ops +*/ +struct rpmsg_virtproc_ops { +struct device *(*create_channel)(struct device *dev,const char *name); +void (*delete_channel)(struct device *dev); +struct module *owner; +}; +int rpmsg_register_device (struct rpmsg_device *rpdev);int rpmsg_unregister_device (struct device *parent,struct rpmsg_channel_info *chinfo);
@@ -75,6 +87,10 @@ int rpmsg_unregister_device(struct device *parent,struct device *rpmsg_find_device (struct device *parent,struct rpmsg_channel_info *chinfo); +struct config_group * +rpmsg_cfs_add_virtproc_group (struct device *dev, +const struct rpmsg_virtproc_ops *ops); +/** * rpmsg_chrdev_register_device() - register chrdev device based on rpdev * @rpdev: prepared rpdev to be used for creating endpoints
include/linux/rpmsg.h +5 -0
@@ -135,6 +135,7 @@ int rpmsg_trysend_offchannel(struct rpmsg_endpoint *ept, u32 src, u32 dst,__poll_t rpmsg_poll (struct rpmsg_endpoint *ept,struct file *filp,poll_table *wait); +struct config_group *rpmsg_cfs_add_channel_group (const char *name);#else static inline int register_rpmsg_device (struct rpmsg_device *dev)
@@ -242,6 +243,10 @@ static inline __poll_t rpmsg_poll(struct rpmsg_endpoint *ept, return 0; } +static inline struct config_group *rpmsg_cfs_add_channel_group (const char *name) +{ +return NULL ; +}#endif /* IS_ENABLED(CONFIG_RPMSG) */ /* use a macro to avoid include chaining to get THIS_MODULE */
Signature · 1 line
2.17.1
RFC PATCH 10/22 rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
In reply to Kishon Vijay Abraham IAdd support to send address service notification message to the backend rpmsg device. This informs the backend rpmsg device about the address allocated to the channel that is created in response to the name service announce message from backend rpmsg device. This is in preparation to add backend rpmsg device using VHOST framework in Linux.
--- drivers/rpmsg/virtio_rpmsg_bus.c | 92 +++++++++++++++++++++++++++++--- 1 file changed, 85 insertions(+), 7 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +85 -7
@@ -71,6 +71,7 @@ struct virtproc_info {/* The feature bitmap for virtio rpmsg */ #define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_AS 1/* RP supports address service notifications */ /** * struct rpmsg_hdr - common header for all rpmsg messages
@@ -110,6 +111,26 @@ struct rpmsg_ns_msg {u32 flags; } __packed; +/** +* struct rpmsg_as_msg - dynamic address service announcement message +* @name: name of the created channel +* @dst: destination address to be used by the backend rpdev +* @src: source address of the backend rpdev (the one that sent name service +* announcement message) +* @flags: indicates whether service is created or destroyed +* +* This message is sent (by virtio_rpmsg_bus) when a new channel is created +* in response to name service announcement message by backend rpdev to create +* a new channel. This sends the allocated source address for the channel +* (destination address for the backend rpdev) to the backend rpdev. +*/ +struct rpmsg_as_msg { +char name[RPMSG_NAME_SIZE ]; +u32 dst; +u32 src; +u32 flags; +} __packed; +/** * enum rpmsg_ns_flags - dynamic name service announcement flags *
@@ -119,6 +140,19 @@ struct rpmsg_ns_msg {enum rpmsg_ns_flags {RPMSG_NS_CREATE =0 ,RPMSG_NS_DESTROY =1 , +RPMSG_AS_ANNOUNCE =2 , +}; + +/** +* enum rpmsg_as_flags - dynamic address service announcement flags +* +* @RPMSG_AS_ASSIGN: address has been assigned to the newly created channel +* @RPMSG_AS_FREE: assigned address is freed from the channel and no longer can +* be used +*/ +enum rpmsg_as_flags { +RPMSG_AS_ASSIGN =1 , +RPMSG_AS_FREE =2 , };/**
@@ -164,6 +198,9 @@ struct virtio_rpmsg_channel {/* Address 53 is reserved for advertising remote services */ #define RPMSG_NS_ADDR (53) +/* Address 54 is reserved for advertising address services */ +#define RPMSG_AS_ADDR (54) +static void virtio_rpmsg_destroy_ept (struct rpmsg_endpoint *ept);static int virtio_rpmsg_send (struct rpmsg_endpoint *ept,void *data,int len); static int virtio_rpmsg_sendto(struct rpmsg_endpoint *ept, void *data, int len,
@@ -329,9 +366,11 @@ static int virtio_rpmsg_announce_create(struct rpmsg_device *rpdev)struct device *dev = &rpdev->dev;int err =0 ; +if (!rpdev->ept || !rpdev->announce) +return err; +/* need to tell remote processor's name service about this channel ? */ -if (rpdev->announce && rpdev->ept && -virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_NS )) { +if (virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_NS )) {struct rpmsg_ns_msg nsm;strncpy (nsm.name, rpdev->id.name,RPMSG_NAME_SIZE );
@@ -343,6 +382,23 @@ static int virtio_rpmsg_announce_create(struct rpmsg_device *rpdev)dev_err (dev,"failed to announce service %d\n" , err); } +/* +* need to tell remote processor's address service about the address allocated +* to this channel +*/ +if (virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_AS )) { +struct rpmsg_as_msg asmsg; + +strncpy (asmsg.name, rpdev->id.name,RPMSG_NAME_SIZE ); + asmsg.dst = rpdev->src; + asmsg.src = rpdev->dst; + asmsg.flags =RPMSG_AS_ASSIGN ; + + err =rpmsg_sendto (rpdev->ept, &asmsg,sizeof (asmsg),RPMSG_AS_ADDR ); +if (err) +dev_err (dev,"failed to announce service %d\n" , err); + } +return err; }
@@ -353,9 +409,28 @@ static int virtio_rpmsg_announce_destroy(struct rpmsg_device *rpdev)struct device *dev = &rpdev->dev;int err =0 ; +if (!rpdev->ept || !rpdev->announce) +return err; + +/* +* need to tell remote processor's address service that we're freeing +* the address allocated to this channel +*/ +if (virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_AS )) { +struct rpmsg_as_msg asmsg; + +strncpy (asmsg.name, rpdev->id.name,RPMSG_NAME_SIZE ); + asmsg.dst = rpdev->src; + asmsg.src = rpdev->dst; + asmsg.flags =RPMSG_AS_FREE ; + + err =rpmsg_sendto (rpdev->ept, &asmsg,sizeof (asmsg),RPMSG_AS_ADDR ); +if (err) +dev_err (dev,"failed to announce service %d\n" , err); + } +/* tell remote processor's name service we're removing this channel */ -if (rpdev->announce && rpdev->ept && -virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_NS )) { +if (virtio_has_feature (vrp->vdev,VIRTIO_RPMSG_F_NS )) {struct rpmsg_ns_msg nsm;strncpy (nsm.name, rpdev->id.name,RPMSG_NAME_SIZE );
@@ -390,7 +465,8 @@ static void virtio_rpmsg_release_device(struct device *dev)* channels. */ static struct rpmsg_device *rpmsg_create_channel (struct virtproc_info *vrp, -struct rpmsg_channel_info *chinfo) +struct rpmsg_channel_info *chinfo, +bool announce) {struct virtio_rpmsg_channel *vch;struct rpmsg_device *rpdev;
@@ -424,7 +500,8 @@ static struct rpmsg_device *rpmsg_create_channel(struct virtproc_info *vrp,* rpmsg server channels has predefined local address (for now), * and their existence needs to be announced remotely */ - rpdev->announce = rpdev->src !=RPMSG_ADDR_ANY ; +if (rpdev->src !=RPMSG_ADDR_ANY || announce) + rpdev->announce = true;strncpy (rpdev->id.name, chinfo->name,RPMSG_NAME_SIZE );
@@ -873,7 +950,7 @@ static int rpmsg_ns_cb(struct rpmsg_device *rpdev, void *data, int len,if (ret)dev_err (dev,"rpmsg_destroy_channel failed: %d\n" , ret); }else { - newch =rpmsg_create_channel (vrp, &chinfo); + newch =rpmsg_create_channel (vrp, &chinfo, msg->flags &RPMSG_AS_ANNOUNCE );if (!newch)dev_err (dev,"rpmsg_create_channel failed\n" ); }
@@ -1042,6 +1119,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_AS , };static struct virtio_driver virtio_ipc_driver = {
Signature · 1 line
2.17.1
RFC PATCH 11/22 rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
In reply to Kishon Vijay Abraham INo functional change intended. Move generic rpmsg structures like "struct rpmsg_hdr", "struct rpmsg_ns_msg" and "struct rpmsg_as_msg", its associated flags and generic macros to rpmsg_internal.h. This is in preparation to add VHOST based vhost_rpmsg_bus.c which will use the same structures and macros.
--- drivers/rpmsg/rpmsg_internal.h | 120 +++++++++++++++++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 120 ------------------------------- 2 files changed, 120 insertions(+), 120 deletions(-)
drivers/rpmsg/rpmsg_internal.h +120 -0
@@ -15,6 +15,126 @@#include <linux/rpmsg.h> #include <linux/poll.h> +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_AS 1/* RP supports address service notifications */ + +/** +* struct rpmsg_hdr - common header for all rpmsg messages +* @src: source address +* @dst: destination address +* @reserved: reserved for future use +* @len: length of payload (in bytes) +* @flags: message flags +* @data: @len bytes of message payload data +* +* Every message sent(/received) on the rpmsg bus begins with this header. +*/ +struct rpmsg_hdr { +u32 src; +u32 dst; +u32 reserved; +u16 len; +u16 flags; +u8 data[0 ]; +} __packed; + +/** +* struct rpmsg_ns_msg - dynamic name service announcement message +* @name: name of remote service that is published +* @addr: address of remote service that is published +* @flags: indicates whether service is created or destroyed +* +* This message is sent across to publish a new service, or announce +* about its removal. When we receive these messages, an appropriate +* rpmsg channel (i.e device) is created/destroyed. In turn, the ->probe() +* or ->remove() handler of the appropriate rpmsg driver will be invoked +* (if/as-soon-as one is registered). +*/ +struct rpmsg_ns_msg { +char name[RPMSG_NAME_SIZE ]; +u32 addr; +u32 flags; +} __packed; + +/** +* struct rpmsg_as_msg - dynamic address service announcement message +* @name: name of the created channel +* @dst: destination address to be used by the backend rpdev +* @src: source address of the backend rpdev (the one that sent name service +* announcement message) +* @flags: indicates whether service is created or destroyed +* +* This message is sent (by virtio_rpmsg_bus) when a new channel is created +* in response to name service announcement message by backend rpdev to create +* a new channel. This sends the allocated source address for the channel +* (destination address for the backend rpdev) to the backend rpdev. +*/ +struct rpmsg_as_msg { +char name[RPMSG_NAME_SIZE ]; +u32 dst; +u32 src; +u32 flags; +} __packed; + +/** +* enum rpmsg_ns_flags - dynamic name service announcement flags +* +* @RPMSG_NS_CREATE: a new remote service was just created +* @RPMSG_NS_DESTROY: a known remote service was just destroyed +*/ +enum rpmsg_ns_flags { +RPMSG_NS_CREATE =0 , +RPMSG_NS_DESTROY =1 , +RPMSG_AS_ANNOUNCE =2 , +}; + +/** +* enum rpmsg_as_flags - dynamic address service announcement flags +* +* @RPMSG_AS_ASSIGN: address has been assigned to the newly created channel +* @RPMSG_AS_FREE: assigned address is freed from the channel and no longer can +* be used +*/ +enum rpmsg_as_flags { +RPMSG_AS_ASSIGN =1 , +RPMSG_AS_FREE =2 , +}; + +/* +* We're allocating buffers of 512 bytes each for communications. The +* number of buffers will be computed from the number of buffers supported +* by the vring, upto a maximum of 512 buffers (256 in each direction). +* +* Each buffer will have 16 bytes for the msg header and 496 bytes for +* the payload. +* +* This will utilize a maximum total space of 256KB for the buffers. +* +* We might also want to add support for user-provided buffers in time. +* This will allow bigger buffer size flexibility, and can also be used +* to achieve zero-copy messaging. +* +* Note that these numbers are purely a decision of this driver - we +* can change this without changing anything in the firmware of the remote +* processor. +*/ +#define MAX_RPMSG_NUM_BUFS (512) +#define MAX_RPMSG_BUF_SIZE (512) + +/* +* Local addresses are dynamically allocated on-demand. +* We do not dynamically assign addresses from the low 1024 range, +* in order to reserve that address range for predefined services. +*/ +#define RPMSG_RESERVED_ADDRESSES (1024) + +/* Address 53 is reserved for advertising remote services */ +#define RPMSG_NS_ADDR (53) + +/* Address 54 is reserved for advertising address services */ +#define RPMSG_AS_ADDR (54) +#define to_rpmsg_device (d) container_of(d, struct rpmsg_device, dev)#define to_rpmsg_driver (d) container_of(d, struct rpmsg_driver, drv)
drivers/rpmsg/virtio_rpmsg_bus.c +0 -120
132 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 12/22 virtio: Add ops to allocate and free buffer
In reply to Kishon Vijay Abraham IAdd ops to allocate and free buffer in struct virtio_config_ops. Certain vhost devices can have restriction on the range of memory it can access on the virtio. The virtio drivers attached to such vhost devices reserves memory that can be accessed by vhost. This function allocates buffer for such reserved region.
For instance when virtio-vhost is used by two hosts connected to NTB, the vhost can access only memory exposed by memory windows and the size of the memory window can be limited. Here the NTB virtio driver can reserve a small region (few MBs) and provide buffer address from this pool whenever requested by virtio client driver.
--- include/linux/virtio_config.h | 42 +++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
include/linux/virtio_config.h +42 -0
@@ -65,6 +65,9 @@ struct irq_affinity;* the caller can then copy. * @set_vq_affinity: set the affinity for a virtqueue (optional). * @get_vq_affinity: get the affinity for a virtqueue (optional). +* @alloc_buffer: Allocate and provide buffer addresses that can be +* accessed by both virtio and vhost +* @free_buffer: Free the allocated buffer address */ typedef void vq_callback_t (struct virtqueue *);struct virtio_config_ops {
@@ -88,6 +91,9 @@ struct virtio_config_ops { const struct cpumask *cpu_mask);const struct cpumask *(*get_vq_affinity)(struct virtio_device *vdev,int index); +void * (*alloc_buffer)(struct virtio_device *vdev,size_t size); +void (*free_buffer)(struct virtio_device *vdev,void *addr, +size_t size); };/* If driver didn't advertise the feature, it will never appear. */
@@ -232,6 +238,42 @@ const char *virtio_bus_name(struct virtio_device *vdev)return vdev->config->bus_name (vdev); } +/** +* virtio_alloc_buffer - Allocate buffer from the reserved memory +* @vdev: Virtio device which manages the reserved memory +* @size: Size of the buffer to be allocated +* +* Certain vhost devices can have restriction on the range of memory +* it can access on the virtio. The virtio drivers attached to +* such vhost devices reserves memory that can be accessed by +* vhost. This function allocates buffer for such reserved region. +*/ +static inline void * +virtio_alloc_buffer (struct virtio_device *vdev,size_t size) +{ +if (!vdev->config->alloc_buffer) +return NULL ; + +return vdev->config->alloc_buffer (vdev, size); +} + +/** +* virtio_free_buffer - Free the allocated buffer +* @vdev: Virtio device which manages the reserved memory +* @addr: Address returned by virtio_alloc_buffer() +* @size: Size of the buffer that has to be freed +* +* Free the allocated buffer address given by virtio_alloc_buffer(). +*/ +static inline void +virtio_free_buffer (struct virtio_device *vdev,void *addr,size_t size) +{ +if (!vdev->config->free_buffer) +return ; + +return vdev->config->free_buffer (vdev, addr, size); +} +/** * virtqueue_set_affinity - setting affinity for a virtqueue * @vq: the virtqueue
Signature · 1 line
2.17.1
RFC PATCH 13/22 rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
In reply to Kishon Vijay Abraham IUse virtio_alloc_buffer() and virtio_free_buffer() to allocate and free memory buffer respectively. Only if buffer allocation using virtio_alloc_buffer() try using dma_alloc_coherent(). This is required for devices like NTB to use rpmsg for communicating with other host.
--- drivers/rpmsg/virtio_rpmsg_bus.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +22 -10
@@ -882,13 +882,16 @@ static int rpmsg_probe(struct virtio_device *vdev) total_buf_space = vrp->num_bufs * vrp->buf_size; -/* allocate coherent memory for the buffers */ - bufs_va =dma_alloc_coherent (vdev->dev.parent, - total_buf_space, &vrp->bufs_dma, -GFP_KERNEL ); + bufs_va =virtio_alloc_buffer (vdev, total_buf_space);if (!bufs_va) { - err = -ENOMEM ; - goto vqs_del; +/* allocate coherent memory for the buffers */ + bufs_va =dma_alloc_coherent (vdev->dev.parent, + total_buf_space, &vrp->bufs_dma, +GFP_KERNEL ); +if (!bufs_va) { + err = -ENOMEM ; + goto vqs_del; + } } dev_dbg(&vdev->dev, "buffers: va %pK, dma %pad\n",
@@ -951,8 +954,13 @@ static int rpmsg_probe(struct virtio_device *vdev)return 0 ; free_coherent: -dma_free_coherent (vdev->dev.parent, total_buf_space, - bufs_va, vrp->bufs_dma); +if (!vrp->bufs_dma) { +virtio_free_buffer (vdev, bufs_va, total_buf_space); + }else { +dma_free_coherent (vdev->dev.parent, total_buf_space, + bufs_va, vrp->bufs_dma); + } + vqs_del: vdev->config->del_vqs (vrp->vdev); free_vrp:
@@ -986,8 +994,12 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev); -dma_free_coherent (vdev->dev.parent, total_buf_space, - vrp->rbufs, vrp->bufs_dma); +if (!vrp->bufs_dma) { +virtio_free_buffer (vdev, vrp->rbufs, total_buf_space); + }else { +dma_free_coherent (vdev->dev.parent, total_buf_space, + vrp->rbufs, vrp->bufs_dma); + }kfree (vrp); }
Signature · 1 line
2.17.1
RFC PATCH 14/22 rpmsg: Add VHOST based remote processor messaging bus
In reply to Kishon Vijay Abraham IAdd a VHOST-based inter-processor communication bus, which enables kernel drivers to communicate with VIRTIO-based messaging bus, running on remote processors, over shared memory using a simple messaging protocol.
--- drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 1 + drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++++++++++++++++ include/linux/rpmsg.h | 1 + 4 files changed, 1163 insertions(+) create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
drivers/rpmsg/Kconfig +10 -0
@@ -64,4 +64,14 @@ config RPMSG_VIRTIO select RPMSG select VIRTIO +config RPMSG_VHOST + tristate "Vhost RPMSG bus driver" + depends on HAS_DMA + select RPMSG + select VHOST + help + Say y here to enable support for the RPMSG VHOST driver + providing communication channels to remote processors running + RPMSG VIRTIO driver. + endmenu
drivers/rpmsg/Makefile +1 -0
@@ -7,3 +7,4 @@ obj-$(CONFIG_RPMSG_QCOM_GLINK_NATIVE) += qcom_glink_native.o obj-$(CONFIG_RPMSG_QCOM_GLINK_SMEM) += qcom_glink_smem.o obj-$(CONFIG_RPMSG_QCOM_SMD) += qcom_smd.o obj-$(CONFIG_RPMSG_VIRTIO) += virtio_rpmsg_bus.o +obj-$(CONFIG_RPMSG_VHOST) += vhost_rpmsg_bus.o
drivers/rpmsg/vhost_rpmsg_bus.c new file +1151 -0
1,151 lines not on this page · raw message
include/linux/rpmsg.h +1 -0
@@ -55,6 +55,7 @@ struct rpmsg_device {u32 dst;struct rpmsg_endpoint *ept;bool announce; +struct list_head list;const struct rpmsg_device_ops *ops; };
Signature · 1 line
2.17.1
RFC PATCH 15/22 samples/rpmsg: Setup delayed work to send message
In reply to Kishon Vijay Abraham ILet us not send any message to the remote processor before announce_create() callback has been invoked. Since announce_create() is only invoked after ->probe() is completed, setup delayed work to start sending message to the remote processor.
--- samples/rpmsg/rpmsg_client_sample.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
samples/rpmsg/rpmsg_client_sample.c +21 -7
@@ -20,6 +20,8 @@ module_param(count, int, 0644);struct instance_data {int rx_count; +struct delayed_work send_msg_work; +struct rpmsg_device *rpdev; }; static int rpmsg_sample_cb(struct rpmsg_device *rpdev, void *data, int len,
@@ -48,9 +50,21 @@ static int rpmsg_sample_cb(struct rpmsg_device *rpdev, void *data, int len,return 0 ; } -static int rpmsg_sample_probe (struct rpmsg_device *rpdev) +static void rpmsg_sample_send_msg_work (struct work_struct *work) { + struct instance_data *idata = container_of(work, struct instance_data, + send_msg_work.work); +struct rpmsg_device *rpdev = idata->rpdev;int ret; + +/* send a message to our remote processor */ + ret =rpmsg_send (rpdev->ept,MSG ,strlen (MSG )); +if (ret) +dev_err (&rpdev->dev,"rpmsg_send failed: %d\n" , ret); +} + +static int rpmsg_sample_probe (struct rpmsg_device *rpdev) +{struct instance_data *idata; dev_info(&rpdev->dev, "new channel: 0x%x -> 0x%x!\n",
@@ -62,18 +76,18 @@ static int rpmsg_sample_probe(struct rpmsg_device *rpdev)dev_set_drvdata (&rpdev->dev, idata); -/* send a message to our remote processor */ - ret =rpmsg_send (rpdev->ept,MSG ,strlen (MSG )); -if (ret) { -dev_err (&rpdev->dev,"rpmsg_send failed: %d\n" , ret); -return ret; - } + idata->rpdev = rpdev; +INIT_DELAYED_WORK (&idata->send_msg_work, rpmsg_sample_send_msg_work); +schedule_delayed_work (&idata->send_msg_work,msecs_to_jiffies (500 ));return 0 ; }static void rpmsg_sample_remove (struct rpmsg_device *rpdev) { +struct instance_data *idata =dev_get_drvdata (&rpdev->dev); + +cancel_delayed_work_sync (&idata->send_msg_work);dev_info (&rpdev->dev,"rpmsg sample client driver is removed\n" ); }
Signature · 1 line
2.17.1
RFC PATCH 16/22 samples/rpmsg: Wait for address to be bound to rpdev for sending message
In reply to Kishon Vijay Abraham Irpmsg_client_sample can use either virtio_rpmsg_bus or vhost_rpmsg_bus to send messages. In the case of vhost_rpmsg_bus, the destination address of rpdev will be assigned only after receiving address service notification from the remote virtio_rpmsg_bus. Wait for address to be bound to rpdev (rpmsg_client_sample running in vhost system) before sending messages to the remote virtio_rpmsg_bus.
--- samples/rpmsg/rpmsg_client_sample.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
samples/rpmsg/rpmsg_client_sample.c +8 -4
@@ -57,10 +57,14 @@ static void rpmsg_sample_send_msg_work(struct work_struct *work)struct rpmsg_device *rpdev = idata->rpdev;int ret; -/* send a message to our remote processor */ - ret =rpmsg_send (rpdev->ept,MSG ,strlen (MSG )); -if (ret) -dev_err (&rpdev->dev,"rpmsg_send failed: %d\n" , ret); +if (rpdev->dst !=RPMSG_ADDR_ANY ) { +/* send a message to our remote processor */ + ret =rpmsg_send (rpdev->ept,MSG ,strlen (MSG )); +if (ret) +dev_err (&rpdev->dev,"rpmsg_send failed: %d\n" , ret); + }else { +schedule_delayed_work (&idata->send_msg_work,msecs_to_jiffies (50 )); + } }static int rpmsg_sample_probe (struct rpmsg_device *rpdev)
Signature · 1 line
2.17.1
RFC PATCH 17/22 rpmsg.txt: Add Documentation to configure rpmsg using configfs
In reply to Kishon Vijay Abraham IAdd Documentation on how rpmsg device can be created using configfs required for vhost_rpmsg_bus.c
--- Documentation/rpmsg.txt | 56 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+)
Documentation/rpmsg.txt +56 -0
@@ -339,3 +339,59 @@ by the bus, and can then start sending messages to the remote service. The plan is also to add static creation of rpmsg channels via the virtio config space, but it's not implemented yet. + +Configuring rpmsg using configfs +================================ + +Usually a rpmsg_device is created when the virtproc driver (virtio_rpmsg_bus.c) +receives a name service notification from the remote core. However there could +also be cases where the user should be given the ability to create rpmsg_device +(like in the case of vhost_rpmsg_bus.c) where vhost_rpmsg_bus should be +responsible for sending name service notification. For such cases, configfs +provides an ability to the user for binding a rpmsg_client_driver with virtproc +device in order to create rpmsg_device. + +Two configfs directories are added for configuring rpmsg +:: + + # ls /sys/kernel/config/rpmsg/ + channel virtproc + +channel: Whenever a new rpmsg_driver is registered with rpmsg core, a new +sub-directory will be created for each entry provided in rpmsg_device_id +table of rpmsg_driver. + +For instance when rpmsg_sample_client is installed, it'll create the following +entry in the mounted configfs directory +:: + + # ls /sys/kernel/config/rpmsg/channel/ + rpmsg-client-sample + +virtproc: A virtproc device can choose to add an entry in this directory. +Virtproc device adds an entry if it has to allow user to control creation of +rpmsg device. (e.g vhost_rpmsg_bus.c) +:: + + # ls /sys/kernel/config/rpmsg/virtproc/ + vhost0 + + +The first step in allowing the user to create rpmsg device is to create a +sub-directory rpmsg-client-sample. For each rpmsg_device, the user would like +to create, a separate subdirectory has to be created. +:: + + # mkdir /sys/kernel/config/rpmsg/channel/rpmsg-client-sample/c1 + +The next step is to link the created sub-directory with virtproc device to +create rpmsg device. +:: + + # ln -s /sys/kernel/config/rpmsg/channel/rpmsg-client-sample/c1 \ + /sys/kernel/config/rpmsg/virtproc/vhost0 + +This will create rpmsg_device. However the driver will not register the +rpmsg device until it receives the VIRTIO_CONFIG_S_DRIVER_OK (in the case +of vhost_rpmsg_bus.c) as it can access virtio buffers only after +VIRTIO_CONFIG_S_DRIVER_OK is set.
Signature · 1 line
2.17.1
RFC PATCH 18/22 virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
In reply to Kishon Vijay Abraham IAdd VIRTIO driver to support Linux VHOST on Configurable PCIe Endpoint device in the backend.
--- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_epf.c | 670 ++++++++++++++++++++++++++++++++ include/linux/virtio.h | 3 + 4 files changed, 683 insertions(+) create mode 100644 drivers/virtio/virtio_pci_epf.c
drivers/virtio/Kconfig +9 -0
@@ -43,6 +43,15 @@ config VIRTIO_PCI_LEGACY If unsure, say Y. +config VIRTIO_PCI_EPF + bool "Support for virtio device on configurable PCIe Endpoint" + depends on VIRTIO_PCI + help + Select this configuration option to enable the VIRTIO driver + for configurable PCIe Endpoint running Linux in backend. + + If in doubt, say "N" to disable VIRTIO PCI EPF driver. + config VIRTIO_VDPA tristate "vDPA driver for virtio devices" depends on VDPA
drivers/virtio/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_VIRTIO_PCI_EPF) += virtio_pci_epf.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o
drivers/virtio/virtio_pci_epf.c new file +670 -0
670 lines not on this page · raw message
include/linux/virtio.h +3 -0
@@ -19,6 +19,7 @@* @priv: a pointer for the virtqueue implementation to use. * @index: the zero-based ordinal number for this queue. * @num_free: number of elements we expect to be able to fit. +* @mutex: mutex to protect concurrent access to the same queue * * A note on @num_free: with indirect buffers, each buffer needs one * element in the queue, otherwise a buffer will need one element per
@@ -31,6 +32,8 @@ struct virtqueue {struct virtio_device *vdev;unsigned int index;unsigned int num_free; +/* mutex to protect concurrent access to the queue while configuring */ +struct mutex lock;void *priv; };
Signature · 1 line
2.17.1
RFC PATCH 19/22 PCI: endpoint: Add EP function driver to provide VHOST interface
In reply to Kishon Vijay Abraham IAdd a new endpoint function driver to register VHOST device and provide interface for the VHOST driver to access virtqueues created by the remote host (using VIRTIO).
--- drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 +++++++++++++++++ drivers/vhost/vhost_cfs.c | 13 - include/linux/vhost.h | 14 + 5 files changed, 1170 insertions(+), 13 deletions(-) create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
drivers/pci/endpoint/functions/Kconfig +11 -0
@@ -24,3 +24,14 @@ config PCI_EPF_NTB device tree. If in doubt, say "N" to disable Endpoint NTB driver. + +config PCI_EPF_VHOST + tristate "PCI Endpoint VHOST driver" + depends on PCI_ENDPOINT + help + Select this configuration option to enable the VHOST driver + for PCI Endpoint. EPF VHOST driver implements VIRTIO backend + for EPF and uses the VHOST framework to bind any VHOST driver + to the VHOST device created by this driver. + + If in doubt, say "N" to disable Endpoint VHOST driver.
drivers/pci/endpoint/functions/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-$(CONFIG_PCI_EPF_TEST) += pci-epf-test.o obj-$(CONFIG_PCI_EPF_NTB) += pci-epf-ntb.o +obj-$(CONFIG_PCI_EPF_VHOST) += pci-epf-vhost.o
drivers/pci/endpoint/functions/pci-epf-vhost.c new file +1144 -0
1,144 lines not on this page · raw message
drivers/vhost/vhost_cfs.c +0 -13
@@ -18,12 +18,6 @@ static struct config_group *vhost_driver_group;/* VHOST device like PCIe EP, NTB etc., */ static struct config_group *vhost_device_group; -struct vhost_driver_item { -struct config_group group; -u32 vendor; -u32 device; -}; -struct vhost_driver_group {struct config_group group; };
@@ -33,13 +27,6 @@ struct vhost_device_item {struct vhost_dev *vdev; }; -static inline -struct vhost_driver_item *to_vhost_driver_item (struct config_item *item) -{ - return container_of(to_config_group(item), struct vhost_driver_item, - group); -} -static inline struct vhost_device_item *to_vhost_device_item (struct config_item *item) {
include/linux/vhost.h +14 -0
@@ -74,6 +74,7 @@ struct vhost_virtqueue {struct vhost_dev *dev;enum vhost_type type;struct vringh vringh; +int index;void (*callback)(struct vhost_virtqueue *vq);void (*notify)(struct vhost_virtqueue *vq);
@@ -148,6 +149,12 @@ struct vhost_msg_node {struct list_head node; }; +struct vhost_driver_item { +struct config_group group; +u32 vendor; +u32 device; +}; +enum vhost_notify_event {NOTIFY_SET_STATUS ,NOTIFY_FINALIZE_FEATURES ,
@@ -230,6 +237,13 @@ static inline void *vhost_get_drvdata(struct vhost_dev *vdev)return dev_get_drvdata (&vdev->dev); } +static inline +struct vhost_driver_item *to_vhost_driver_item (struct config_item *item) +{ + return container_of(to_config_group(item), struct vhost_driver_item, + group); +} +int vhost_register_driver (struct vhost_driver *driver);void vhost_unregister_driver (struct vhost_driver *driver);int vhost_register_device (struct vhost_dev *vdev);
Signature · 1 line
2.17.1
RFC PATCH 20/22 NTB: Add a new NTB client driver to implement VIRTIO functionality
In reply to Kishon Vijay Abraham IAdd a new NTB client driver to implement VIRTIO functionality. When two hosts are connected using NTB, one of the hosts should run NTB client driver that implements VIRTIO functionality and the other host should run NTB client implements VHOST functionality. This interfaces with VIRTIO layer so that any virtio client driver can exchange data with the remote vhost client driver.
Since each NTB host can only expose fewer contiguous memory range to the remote NTB host (number of memory windows supported), reserve contiguous memory range using dma_alloc_coherent() and then manage this area using gen_pool for providing buffers to the virtio client driver. The virtio client driver should only provide this buffer region to the remote vhost driver.
--- drivers/ntb/Kconfig | 9 + drivers/ntb/Makefile | 1 + drivers/ntb/ntb_virtio.c | 853 +++++++++++++++++++++++++++++++++++++++ drivers/ntb/ntb_virtio.h | 56 +++ 4 files changed, 919 insertions(+) create mode 100644 drivers/ntb/ntb_virtio.c create mode 100644 drivers/ntb/ntb_virtio.h
drivers/ntb/Kconfig +9 -0
@@ -37,4 +37,13 @@ config NTB_TRANSPORT If unsure, say N. +config NTB_VIRTIO + tristate "NTB VIRTIO" + help + The NTB virtio driver sits between the NTB HW driver and the virtio + client driver and lets the virtio client driver to exchange data with + the remote vhost driver over the NTB hardware. + + If unsure, say N. + endif # NTB
drivers/ntb/Makefile +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_NTB) += ntb.o hw/ test/ obj-$(CONFIG_NTB_TRANSPORT) += ntb_transport.o +obj-$(CONFIG_NTB_VIRTIO) += ntb_virtio.o ntb-y := core.o ntb-$(CONFIG_NTB_MSI) += msi.o
drivers/ntb/ntb_virtio.c new file +853 -0
853 lines not on this page · raw message
drivers/ntb/ntb_virtio.h new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 21/22 NTB: Add a new NTB client driver to implement VHOST functionality
In reply to Kishon Vijay Abraham IAdd a new NTB client driver to implement VHOST functionality. When two hosts are connected using NTB, one of the hosts should run NTB client driver that implements VIRTIO functionality and the other host should run NTB client implements VHOST functionality. This interfaces with VHOST layer so that any vhost client driver can exchange data with the remote virtio client driver.
--- drivers/ntb/Kconfig | 9 + drivers/ntb/Makefile | 1 + drivers/ntb/ntb_vhost.c | 776 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 786 insertions(+) create mode 100644 drivers/ntb/ntb_vhost.c
drivers/ntb/Kconfig +9 -0
@@ -46,4 +46,13 @@ config NTB_VIRTIO If unsure, say N. +config NTB_VHOST + tristate "NTB VHOST" + help + The NTB vhost driver sits between the NTB HW driver and the vhost + client driver and lets the vhost client driver to exchange data with + the remote virtio driver over the NTB hardware. + + If unsure, say N. + endif # NTB
drivers/ntb/Makefile +1 -0
@@ -2,6 +2,7 @@ obj-$(CONFIG_NTB) += ntb.o hw/ test/ obj-$(CONFIG_NTB_TRANSPORT) += ntb_transport.o obj-$(CONFIG_NTB_VIRTIO) += ntb_virtio.o +obj-$(CONFIG_NTB_VHOST) += ntb_vhost.o ntb-y := core.o ntb-$(CONFIG_NTB_MSI) += msi.o
drivers/ntb/ntb_vhost.c new file +776 -0
776 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 22/22 NTB: Describe ntb_virtio and ntb_vhost client in the documentation
In reply to Kishon Vijay Abraham IAdd a blurb in Documentation/ntb.txt to describe the ntb_virtio and ntb_vhost client
--- Documentation/driver-api/ntb.rst | 11 +++++++++++ 1 file changed, 11 insertions(+)
Documentation/driver-api/ntb.rst +11 -0
@@ -227,6 +227,17 @@ test client is interacted with through the debugfs filesystem: specified peer. That peer's interrupt's occurrence file should be incremented. +NTB Vhost Client (ntb\_vhost) and NTB Virtio Client (ntb\_virtio) +------------------------------------------------------------------ + +When two hosts are connected via NTB, one of the hosts should use NTB Vhost +Client and the other host should use NTB Virtio Client. The NTB Vhost client +interfaces with the Linux Vhost Framework and lets it to be used with any +vhost client driver. The NTB Virtio client interfaces with the Linux Virtio +Framework and lets it to be used with any virtio client driver. The Vhost +client driver and Virtio client driver creates a logic cink to exchange data +with each other. + NTB Hardware Drivers ====================
Signature · 1 line
2.17.1
Michael S. Tsirkin
maintainerQuoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ + | | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |+------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Quoted 83 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-) create mode 100644 drivers/ntb/ntb_vhost.c create mode 100644 drivers/ntb/ntb_virtio.c create mode 100644 drivers/ntb/ntb_virtio.h create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)Signature · 1 line
2.17.1
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ + | | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |+---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
Quoted 6 lines
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).
Have you considered to implement these through vDPA?
Thanks
Quoted 83 lines
Quoted 83 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
Hi Jason,
Quoted 86 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).
This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization. So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)
Quoted 2 lines
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.
The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above), all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.
Thanks
Kishon
Quoted 88 lines
Thanks
Quoted 83 lines
Quoted 83 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +-
drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +-drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {
drivers/vhost=> include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +-39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)--
2.17.1
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfc
Thanks
Quoted 4 lines
Quoted 2 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).
This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.
Quoted 7 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Quoted 7 lines
Quoted 2 lines
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.
The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),
I see.
Quoted 3 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Does this make sense?
Thanks
Quoted 89 lines
Thanks
KishonQuoted 85 lines
Thanks
Quoted 82 lines
Quoted 82 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi Jason,
Quoted 104 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 2 lines
Quoted 2 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).
Quoted 7 lines
Quoted 2 lines
This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.
I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication
#ls drivers/ntb/hw/
amd epf idt intel mscc
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.c
Quoted 11 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Quoted 14 lines
Quoted 9 lines
Quoted 4 lines
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.
The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 8 lines
Quoted 3 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at virtio_dev_probe() of virtio_bus
device_features = dev->config->get_features(dev);
Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).
Quoted 11 lines
It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue, in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 2 lines
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Thanks for your suggestions!
Regards
Kishon
Quoted 94 lines
Thanks
Quoted 89 lines
Thanks
KishonQuoted 85 lines
Thanks
Quoted 82 lines
Quoted 82 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +-
drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +-drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {
drivers/vhost=> include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +-39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)--
2.17.1
Quoted 118 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 86 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).
This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication
#ls
drivers/ntb/hw/
amd epf idt intel mscc
Quoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.c
Thanks for those backgrounds.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)
Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 26 lines
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.
The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 3 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.
Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets
VIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 6 lines
Quoted 6 lines
It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.
Quoted 7 lines
Quoted 5 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
Quoted 7 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 2 lines
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 95 lines
Regards
KishonQuoted 91 lines
Thanks
Quoted 88 lines
Thanks
KishonQuoted 85 lines
Thanks
Quoted 82 lines
Quoted 82 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi Jason,
Quoted 209 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 103 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 3 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).Quoted 6 lines
Quoted 2 lines
This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication
#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 27 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 3 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets
VIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Quoted 8 lines
Quoted 2 lines
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh? And virtio ring(2) is created by virtio pci (RC).
Quoted 3 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver) and the other is how certain inputs to virtio device such as number of buffers be determined.
Thanks again for your suggestions!
Regards
Kishon
Quoted 119 lines
Quoted 9 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 2 lines
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 95 lines
Regards
KishonQuoted 91 lines
Thanks
Quoted 88 lines
Thanks
KishonQuoted 85 lines
Thanks
Quoted 82 lines
Quoted 82 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +-
drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +-drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {
drivers/vhost=> include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +-39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)--
2.17.1
Quoted 198 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 195 lines · Jason Wang
Quoted 118 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication
#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 24 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 6 lines
Quoted 3 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 5 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets
VIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).
Quoted 8 lines
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).
Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
Quoted 2 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?
Yes.
Quoted 3 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 2 lines
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Thanks
Quoted 117 lines
Regards
KishonQuoted 113 lines
Quoted 8 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 1 line
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 94 lines
Regards
KishonQuoted 91 lines
Thanks
Quoted 88 lines
Thanks
KishonQuoted 85 lines
Thanks
Quoted 82 lines
Quoted 82 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
Hi Jason,
Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 4 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 8 lines
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 9 lines
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 6 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Quoted 8 lines
Quoted 5 lines
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Quoted 7 lines
Quoted 4 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Yes.
okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?
Quoted 23 lines
Quoted 5 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 2 lines
Thanks again for your suggestions!
Quoted 6 lines
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
I'll try to create something based on your proposed design here.
Regards
Kishon
[1] ->
https://lore.kernel.org/linux-pci/[email protected]/
Quoted 127 lines
Thanks
Quoted 122 lines
Regards
KishonQuoted 118 lines
Quoted 9 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 1 line
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 98 lines
Regards
KishonQuoted 95 lines
Thanks
Quoted 92 lines
Thanks
KishonQuoted 89 lines
Thanks
Quoted 86 lines
Quoted 86 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144
++++++++++++++++
drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151
+++++++++++++++++
drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +-drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +-drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {
drivers/vhost=> include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +-39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)--
2.17.1
Quoted 221 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 209 lines · Jason Wang
Quoted 199 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 196 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 4 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 8 lines
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 9 lines
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 2 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
Quoted 40 lines
Quoted 40 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 10 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?
Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 1 line
Thanks again for your suggestions!
Quoted 7 lines
Quoted 6 lines
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
Quoted 2 lines
I'll try to create something based on your proposed design here.
Sure, but for coding, we'd better wait for other's opinion here.
Thanks
Quoted 127 lines
Regards
Kishon[1] ->
https://lore.kernel.org/linux-pci/[email protected]/Quoted 121 lines
Thanks
Quoted 118 lines
Regards
KishonQuoted 115 lines
Quoted 8 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 1 line
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 98 lines
Regards
KishonQuoted 95 lines
Thanks
Quoted 92 lines
Thanks
KishonQuoted 89 lines
Thanks
Quoted 86 lines
Quoted 86 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe
Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the
documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:
[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]
Quoted 66 lines · Kishon Vijay Abraham I
Quoted 65 lines
Hi Jason,
Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 54 lines · Jason Wang
Quoted 53 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 50 lines · Jason Wang
Quoted 49 lines · Michael S. Tsirkin
Quoted 48 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | |
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+
First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just virtqueues + the exiting vhost interfaces?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 31 lines
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
I guess that clears up at least some of my questions from above...
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Ok.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I see.
Quoted 4 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.
This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)
Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
Quoted 40 lines
Quoted 33 lines
Quoted 15 lines
Quoted 4 lines
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at
virtio_dev_probe()ofvirtio_busQuoted 7 lines
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 7 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 7 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 3 lines
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 8 lines
Quoted 3 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).
Does that make any sense?
Quoted 62 lines
Quoted 46 lines
Quoted 46 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.Quoted 7 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
Quoted 7 lines
Quoted 5 lines
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.
Quoted 29 lines
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 24 lines
Quoted 24 lines
Quoted 4 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 1 line
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Quoted 6 lines
Quoted 2 lines
Got it. My initial design was based on my understanding of your comments [1].
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
I believe that we indeed need something bus-like to be able to support a variety of devices.
Quoted 7 lines
Quoted 4 lines
I'll try to create something based on your proposed design here.
Sure, but for coding, we'd better wait for other's opinion here.
Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.
Hi,
Quoted 83 lines · Cornelia Huck
On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]
Quoted 66 lines · Kishon Vijay Abraham I
Quoted 65 lines
Hi Jason,
Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 54 lines · Jason Wang
Quoted 53 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 50 lines · Jason Wang
Quoted 49 lines · Michael S. Tsirkin
Quoted 48 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | |Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+
Quoted 3 lines
First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just
Right, vhost_rpmsg is 'device' and virtio_rpmsg is 'driver'.
Quoted 1 line
virtqueues + the exiting vhost interfaces?
It's implemented to provide the full 'device' functionality.
Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
Quoted 3 lines
So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?
Right, the upper half should provide the functionality.
The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.
The top half should be transparent to what type of device is actually using it.
Quoted 147 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 32 lines
Quoted 14 lines
Quoted 11 lines
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
I guess that clears up at least some of my questions from above...
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Ok.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I see.
Quoted 4 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)
Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
Quoted 40 lines
Quoted 33 lines
Quoted 15 lines
Quoted 4 lines
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at
virtio_dev_probe()ofvirtio_busQuoted 7 lines
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 7 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 7 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 3 lines
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 8 lines
Quoted 3 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
Quoted 5 lines
When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).
I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication; setup_vq() in virtio_pci_legacy.c).
"Virtio Over NTB" should anyways be a new transport.
Quoted 2 lines
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
Quoted 109 lines
Quoted 63 lines
Quoted 47 lines
Quoted 47 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.Quoted 7 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
Quoted 7 lines
Quoted 5 lines
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.
Quoted 33 lines
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 26 lines
Quoted 16 lines
Quoted 4 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 8 lines
Quoted 1 line
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
Quoted 5 lines
Quoted 2 lines
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
I believe that we indeed need something bus-like to be able to support a variety of devices.
I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.
[1] -> Page 13 in https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
I'll try to create something based on your proposed design here.
Sure, but for coding, we'd better wait for other's opinion here.
Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.
I think your understanding is correct! Thanks for your inputs.
Thanks
Kishon
Quoted 278 lines · Kishon Vijay Abraham I
Hi,
Quoted 85 lines · Cornelia Huck
On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]
Quoted 64 lines · Kishon Vijay Abraham I
Quoted 63 lines
Hi Jason,
Quoted 60 lines · Jason Wang
Quoted 59 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 56 lines · Jason Wang
Quoted 55 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 52 lines · Jason Wang
Quoted 51 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 48 lines · Jason Wang
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST21) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for
rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication
between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+-----v------+ +------v-------+
| Linux | | Linux |
| Endpoint | | Root Complex |
| <-----------------> |
| | | |
| SOC1 | | SOC2 |
+------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP | |
| | CONTROLLER1 | | CONTROLLER2 | |Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| | <-----------------------------------> | |
| | | | | |
| | | | | |
| | | SoC With Multiple EP Instances | | |
| | | (Configured using NTB Function) | | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+Quoted 3 lines
First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly justRight,
vhost_rpmsgis 'device' andvirtio_rpmsgis 'driver'.Quoted 1 line
virtqueues + the exiting vhost interfaces?
It's implemented to provide the full 'device' functionality.
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Software Layering:
The high-level SW layering should look something like below. This series
adds support only for RPMSG VHOST, however something similar should be
done for net and scsi. With that any vhost device (PCI, NTB, Platform
device, user) can use any of the vhost client driver.+----------------+ +-----------+ +------------+
+----------+
| RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X |
+-------^--------+ +-----^-----+ +-----^------+
+----^-----+
| | | |
| | | |
| | | |
+-----------v-----------------v--------------v--------------v----------+| VHOST CORE |
+--------^---------------^--------------------^------------------^-----+Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| | | |
| | | |
| | | |
+--------v-------+ +----v------+ +----------v----------+
+----v-----+
| PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X |
+----------------+ +-----------+ +---------------------+
+----------+Quoted 3 lines
So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?
Right, the upper half should provide the functionality.
The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.The top half should be transparent to what type of device is actually using it.
Quoted 141 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 32 lines
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
This was initially proposed here [1]
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
[1] ->
https://lore.kernel.org/r/[email protected]I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
I guess that clears up at least some of my questions from above...
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Ok.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 4 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)
Quoted 60 lines
Quoted 52 lines
Quoted 43 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at
virtio_dev_probe()ofvirtio_busQuoted 7 lines
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 7 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 7 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 3 lines
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 9 lines
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 3 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
Quoted 7 lines
Quoted 5 lines
When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).
I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication;
setup_vq()invirtio_pci_legacy.c).
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).
Quoted 6 lines
"Virtio Over NTB" should anyways be a new transport.
Quoted 2 lines
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?
Thanks
Quoted 128 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.Quoted 7 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
Quoted 7 lines
Quoted 5 lines
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.
Quoted 30 lines
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 23 lines
Quoted 13 lines
Quoted 3 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 8 lines
Quoted 1 line
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
Quoted 5 lines
Quoted 2 lines
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
I believe that we indeed need something bus-like to be able to support a variety of devices.
I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.
[1] -> Page 13 in https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
I'll try to create something based on your proposed design here.
Sure, but for coding, we'd better wait for other's opinion here.
Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.
I think your understanding is correct! Thanks for your inputs.
Thanks
Kishon
Quoted 99 lines · On Tue, 1 Sep 2020 16:50:03 +0800 Jason Wang <[email protected]> wrote:
Quoted 98 lines · Kishon Vijay Abraham I
Hi,
Quoted 95 lines · Cornelia Huck
On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]
Quoted 86 lines · Kishon Vijay Abraham I
Quoted 85 lines
Hi Jason,
Quoted 82 lines · Jason Wang
Quoted 81 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 78 lines · Jason Wang
Quoted 77 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 74 lines · Jason Wang
Quoted 73 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 70 lines · Jason Wang
Quoted 69 lines · Michael S. Tsirkin
On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay
Abraham I wrote:
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg
communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST21) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver
(uses vhost) for
rpmsg communication between two SoCs connected to
each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg
communication
between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+-----v------+ +------v-------+
| Linux | | Linux |
| Endpoint | | Root Complex |
| <-----------------> |
| | | |
| SOC1 | | SOC2 |
+------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | |
HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP
| |
| | CONTROLLER1 | |
CONTROLLER2 | |
| | <-----------------------------------> | |
| | | | | |
| | | | | |
| | | SoC With Multiple EP Instances
| | |
| | | (Configured using NTB Function)
| | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly justRight,
vhost_rpmsgis 'device' andvirtio_rpmsgis 'driver'.Quoted 1 line
virtqueues + the exiting vhost interfaces?
It's implemented to provide the full 'device' functionality.
Ok.
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Software Layering:
The high-level SW layering should look something like
below. This series
adds support only for RPMSG VHOST, however something
similar should be
done for net and scsi. With that any vhost device (PCI,
NTB, Platform
device, user) can use any of the vhost client driver.+----------------+ +-----------+ +------------+
+----------+
| RPMSG VHOST | | NET VHOST | | SCSI VHOST
| | X |
+-------^--------+ +-----^-----+ +-----^------+
+----^-----+
| | | |
| | | |
| | | |
+-----------v-----------------v--------------v--------------v----------+| VHOST
CORE |
+--------^---------------^--------------------^------------------^-----+| | | |
| | | |
| | | |
+--------v-------+ +----v------+ +----------v----------+
+----v-----+
| PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST|
| X |
+----------------+ +-----------+ +---------------------+
+----------+
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?
Right, the upper half should provide the functionality. The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.
Ok. In that respect, the SW interface is sufficiently device-like, I guess.
Quoted 184 lines
Quoted 176 lines
The top half should be transparent to what type of device is actually using it.
Quoted 166 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 48 lines
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This was initially proposed here [1]
[1] ->
https://lore.kernel.org/r/[email protected]Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I find this very interesting. A huge patchset so will take
a bit
to review, but I certainly plan to do that. Thanks!Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 4 lines
Quoted 4 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the
work is
duplicated with vDPA (e.g the epf transport or vhost bus).Quoted 8 lines
Quoted 3 lines
This is about connecting two different HW systems both running Linux and
doesn't necessarily involve virtualization.Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the
hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between
two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe
RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and
any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccAnd similarly for the PCIe RC<->EP communication, this adds a generic endpoint
function driver and hence any SoC that supports configurable PCIe endpoint can
use virtio-vhost communicationQuoted 4 lines
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable,
one
acting as guest and one as host. So one system will provide virtioQuoted 8 lines
functionality reserving memory for virtqueues and the other provides vhost
functionality providing a way to access the virtqueues in virtio memory.
One is
source and the other is sink and there is no intermediate entity. (vhost was
probably intermediate entity in virtualization?)(Not a native English speaker) but "vhost" could introduce some confusion for
me since it was use for implementing virtio backend for userspace drivers. I
guess "vringh" could be better.Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Initially I had named this vringh but later decided to choose vhost instead of
vringh. vhost is still a virtio backend (not necessarily userspace) though it
now resides in an entirely different system. Whatever virtio is for a frontend
system, vhost can be that for a backend system. vring can be for accessing
virtqueue and can be used either in frontend or backend.I guess that clears up at least some of my questions from above...
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Ok.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg
driver
or vhost net driver is not provided.The HW connection looks something like
https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I see.
Quoted 4 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at
virtio_dev_probe()ofvirtio_busQuoted 7 lines
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 7 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 7 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 3 lines
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 8 lines
Quoted 3 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
Quoted 5 lines
When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).
Quoted 6 lines
I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication;
setup_vq()invirtio_pci_legacy.c).
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).
It seems I really need to read up on vDPA more... do you have a pointer for diving into this alternatives aspect?
Quoted 15 lines
Quoted 5 lines
"Virtio Over NTB" should anyways be a new transport.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
Quoted 2 lines
A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?
Can we maybe implement an alternative bus as well that would allow us to support different virtio device implementations (in addition to the vhost bus + userspace combination)?
Quoted 120 lines
Thanks
Quoted 116 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.Quoted 7 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
Quoted 7 lines
Quoted 5 lines
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.
Quoted 30 lines
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 23 lines
Quoted 13 lines
Quoted 3 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 8 lines
Quoted 1 line
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
Quoted 5 lines
Quoted 2 lines
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
I believe that we indeed need something bus-like to be able to support a variety of devices.
I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.
[1] -> Page 13 in
https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf
So, I guess it's a trade-off: do we expect a variety of device types, or more of a variety of devices needing different adapters?
Quoted 5 lines · Cornelia Huck
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).It seems I really need to read up on vDPA more... do you have a pointer for diving into this alternatives aspect?
See vpda_config_ops in include/linux/vdpa.h
Especially this part:
int (*set_vq_address)(struct vdpa_device *vdev, u16 idx, u64 desc_area, u64 driver_area, u64 device_area);
This means for the devices (e.g endpoint device) that is hard to implement virtio-pci layout, it can use any other register layout or vendor specific way to configure the virtqueue.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
"Virtio Over NTB" should anyways be a new transport.
Quoted 1 line
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
Quoted 5 lines
Quoted 2 lines
A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?
Can we maybe implement an alternative bus as well that would allow us to support different virtio device implementations (in addition to the vhost bus + userspace combination)?
That should be fine, but I'm not quite sure that implementing the device in kerne (kthread) is the good approach.
Thanks
Quoted 1 line
Hi Jason,
Quoted 338 lines · Jason Wang
Quoted 328 lines · Kishon Vijay Abraham I
Hi,
Quoted 95 lines · Cornelia Huck
On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]
Quoted 70 lines · Kishon Vijay Abraham I
Quoted 69 lines
Hi Jason,
Quoted 66 lines · Jason Wang
Quoted 65 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 54 lines · Jason Wang
Quoted 53 lines · Michael S. Tsirkin
On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay
Abraham I wrote:
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg
communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST21) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver
(uses vhost) for
rpmsg communication between two SoCs connected to
each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg
communication
between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+-----v------+ +------v-------+
| Linux | | Linux |
| Endpoint | | Root Complex |
| <-----------------> |
| | | |
| SOC1 | | SOC2 |
+------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | |
HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP
| |
| | CONTROLLER1 | |
CONTROLLER2 | |
| | <-----------------------------------> | |
| | | | | |Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
| | | | | |
| | | SoC With Multiple EP Instances
| | |
| | | (Configured using NTB Function)
| | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+Quoted 3 lines
First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly justRight,
vhost_rpmsgis 'device' andvirtio_rpmsgis 'driver'.Quoted 1 line
virtqueues + the exiting vhost interfaces?
It's implemented to provide the full 'device' functionality.
Quoted 38 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Software Layering:
The high-level SW layering should look something like
below. This series
adds support only for RPMSG VHOST, however something
similar should be
done for net and scsi. With that any vhost device (PCI,
NTB, Platform
device, user) can use any of the vhost client driver.+----------------+ +-----------+ +------------+
+----------+
| RPMSG VHOST | | NET VHOST | | SCSI VHOST
| | X |
+-------^--------+ +-----^-----+ +-----^------+
+----^-----+
| | | |
| | | |
| | | |
+-----------v-----------------v--------------v--------------v----------+| VHOST
CORE |
+--------^---------------^--------------------^------------------^-----+Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
| | | |
| | | |
| | | |
+--------v-------+ +----v------+ +----------v----------+
+----v-----+
| PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST|
| X |
+----------------+ +-----------+ +---------------------+
+----------+Quoted 3 lines
So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?
Right, the upper half should provide the functionality. The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.
The top half should be transparent to what type of device is actually using it.
Quoted 166 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 48 lines
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This was initially proposed here [1]
[1] ->
https://lore.kernel.org/r/[email protected]Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I find this very interesting. A huge patchset so will take
a bit
to review, but I certainly plan to do that. Thanks!Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 4 lines
Quoted 4 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the
work is
duplicated with vDPA (e.g the epf transport or vhost bus).Quoted 8 lines
Quoted 3 lines
This is about connecting two different HW systems both running Linux and
doesn't necessarily involve virtualization.Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the
hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between
two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe
RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and
any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccAnd similarly for the PCIe RC<->EP communication, this adds a generic endpoint
function driver and hence any SoC that supports configurable PCIe endpoint can
use virtio-vhost communicationQuoted 4 lines
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable,
one
acting as guest and one as host. So one system will provide virtioQuoted 8 lines
functionality reserving memory for virtqueues and the other provides vhost
functionality providing a way to access the virtqueues in virtio memory.
One is
source and the other is sink and there is no intermediate entity. (vhost was
probably intermediate entity in virtualization?)(Not a native English speaker) but "vhost" could introduce some confusion for
me since it was use for implementing virtio backend for userspace drivers. I
guess "vringh" could be better.Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Initially I had named this vringh but later decided to choose vhost instead of
vringh. vhost is still a virtio backend (not necessarily userspace) though it
now resides in an entirely different system. Whatever virtio is for a frontend
system, vhost can be that for a backend system. vring can be for accessing
virtqueue and can be used either in frontend or backend.I guess that clears up at least some of my questions from above...
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Ok.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg
driver
or vhost net driver is not provided.The HW connection looks something like
https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I see.
Quoted 4 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines
So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.
Having the same bus can also create sequencing issues.
If you look at
virtio_dev_probe()ofvirtio_busQuoted 7 lines
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 7 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 7 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Quoted 3 lines
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
Quoted 8 lines
Quoted 3 lines
okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.
Quoted 5 lines
When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).
Quoted 6 lines
I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication;
setup_vq()invirtio_pci_legacy.c).
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).
Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).
struct vdpa_config_ops {
/* Virtqueue ops */
int (*set_vq_address)(struct vdpa_device *vdev,
u16 idx, u64 desc_area, u64 driver_area,
u64 device_area);
void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num);
void (*kick_vq)(struct vdpa_device *vdev, u16 idx);
void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx,
struct vdpa_callback *cb);
void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready);
bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
const struct vdpa_vq_state *state);
int (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
struct vdpa_vq_state *state);
struct vdpa_notification_area
(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
/* vq irq is not expected to be changed once DRIVER_OK is set */
int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);
/* Device ops */
u32 (*get_vq_align)(struct vdpa_device *vdev);
u64 (*get_features)(struct vdpa_device *vdev);
int (*set_features)(struct vdpa_device *vdev, u64 features);
void (*set_config_cb)(struct vdpa_device *vdev,
struct vdpa_callback *cb);
u16 (*get_vq_num_max)(struct vdpa_device *vdev);
u32 (*get_device_id)(struct vdpa_device *vdev);
u32 (*get_vendor_id)(struct vdpa_device *vdev);
u8 (*get_status)(struct vdpa_device *vdev);
void (*set_status)(struct vdpa_device *vdev, u8 status);
void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
void *buf, unsigned int len);
void (*set_config)(struct vdpa_device *vdev, unsigned int offset,
const void *buf, unsigned int len);
u32 (*get_generation)(struct vdpa_device *vdev);
/* DMA ops */
int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb);
int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size,
u64 pa, u32 perm);
int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);
/* Free device resources */
void (*free)(struct vdpa_device *vdev);
};+struct vhost_config_ops {
+ int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs,
+ unsigned int num_bufs, struct vhost_virtqueue *vqs[], + vhost_vq_callback_t *callbacks[], + const char * const names[]);
+ void (*del_vqs)(struct vhost_dev *vdev);
+ int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+ int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+ int (*set_features)(struct vhost_dev *vdev, u64 device_features);
+ int (*set_status)(struct vhost_dev *vdev, u8 status);
+ u8 (*get_status)(struct vhost_dev *vdev);
+};
+
struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.
I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].
Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.
I think vdpa always works with client drivers in userspace and providing userspace address for vring.
Quoted 15 lines
Quoted 5 lines
"Virtio Over NTB" should anyways be a new transport.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
Quoted 2 lines
A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?
The vhost bus approach doesn't provide any restriction in where the virto backend device should be created. This series creates two types of virtio backend device (one for PCIe endpoint and the other for NTB) and both these devices are created in kernel.
Thanks
Kishon
Quoted 138 lines
Thanks
Quoted 133 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.Quoted 7 lines
It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
I think this will work however there is an addtional copy between vringh queue and virtqueue,
Quoted 7 lines
Quoted 5 lines
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 11 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Right.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.
I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.
Quoted 30 lines
Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?
Quoted 23 lines
Quoted 13 lines
Quoted 3 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 8 lines
Quoted 1 line
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
Quoted 5 lines
Quoted 2 lines
Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).
I believe that we indeed need something bus-like to be able to support a variety of devices.
I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.
[1] -> Page 13 in
https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdfQuoted 11 lines
Quoted 8 lines
Quoted 5 lines
I'll try to create something based on your proposed design here.
Sure, but for coding, we'd better wait for other's opinion here.
Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.
I think your understanding is correct! Thanks for your inputs.
Thanks
Kishon
Hi Kishon:
Quoted 89 lines · Kishon Vijay Abraham I
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).Okay, I just tried to compare the 'struct
vdpa_config_ops' and 'structvhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).struct vdpa_config_ops { /* Virtqueue ops */ int (*set_vq_address)(struct vdpa_device *vdev, u16 idx, u64 desc_area, u64 driver_area, u64 device_area); void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num); void (*kick_vq)(struct vdpa_device *vdev, u16 idx); void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx, struct vdpa_callback *cb); void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready); bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx); int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, const struct vdpa_vq_state *state); int (*get_vq_state)(struct vdpa_device *vdev, u16 idx, struct vdpa_vq_state *state); struct vdpa_notification_area (*get_vq_notification)(struct vdpa_device *vdev, u16 idx); /* vq irq is not expected to be changed once DRIVER_OK is set */ int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx); /* Device ops */ u32 (*get_vq_align)(struct vdpa_device *vdev); u64 (*get_features)(struct vdpa_device *vdev); int (*set_features)(struct vdpa_device *vdev, u64 features); void (*set_config_cb)(struct vdpa_device *vdev, struct vdpa_callback *cb); u16 (*get_vq_num_max)(struct vdpa_device *vdev); u32 (*get_device_id)(struct vdpa_device *vdev); u32 (*get_vendor_id)(struct vdpa_device *vdev); u8 (*get_status)(struct vdpa_device *vdev); void (*set_status)(struct vdpa_device *vdev, u8 status); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len); void (*set_config)(struct vdpa_device *vdev, unsigned int offset, const void *buf, unsigned int len); u32 (*get_generation)(struct vdpa_device *vdev); /* DMA ops */ int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb); int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size, u64 pa, u32 perm); int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size); /* Free device resources */ void (*free)(struct vdpa_device *vdev); };(quoted diff) +13
-0+struct vhost_config_ops { + int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs, + unsigned int num_bufs, struct vhost_virtqueue *vqs[], + vhost_vq_callback_t *callbacks[], + const char * const names[]); + void (*del_vqs)(struct vhost_dev *vdev); + int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len); + int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len); + int (*set_features)(struct vhost_dev *vdev, u64 device_features); + int (*set_status)(struct vhost_dev *vdev, u8 status); + u8 (*get_status)(struct vhost_dev *vdev); +}; +struct
virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.I think it differs in (*set_vq_address)() and (*create_vqs)(). [
create_vqs()introduced in structvhost_config_opsprovides complimentary functionality to (*find_vqs)() in structvirtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].Back to the difference between (*set_vq_address)() and (*create_vqs)(),
set_vq_address()directly provides the virtqueue address to the vdpa device butcreate_vqs()only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.
Quoted 2 lines
I think vdpa always works with client drivers in userspace and providing userspace address for vring.
Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:
My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:
+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev, + u32 command)
+{
+ struct virtio_pci_epf *pci_epf;
+ void __iomem *ioaddr;
+ ktime_t timeout;
+ bool timedout;
+ int ret = 0;
+ u8 status;
+
+ pci_epf = to_virtio_pci_epf(vp_dev);
+ ioaddr = vp_dev->ioaddr;
+
+ mutex_lock(&pci_epf->lock);
+ writeb(command, ioaddr + HOST_CMD);
+ timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+ while (1) {
+ timedout = ktime_after(ktime_get(), timeout);
+ status = readb(ioaddr + HOST_CMD_STATUS);
+
Several questions:
- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed. If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.
- You have you vendor specific layout (according to virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead
- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
"Virtio Over NTB" should anyways be a new transport.
Quoted 1 line
Does that make any sense?
yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.
Quoted 6 lines
Quoted 2 lines
A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?
The vhost bus approach doesn't provide any restriction in where the virto backend device should be created. This series creates two types of virtio backend device (one for PCIe endpoint and the other for NTB) and both these devices are created in kernel.
Ok.
Thanks
Quoted 4 lines
Thanks
Kishon
Hi Jason,
Quoted 94 lines · Jason Wang
Hi Kishon:
Quoted 91 lines · Kishon Vijay Abraham I
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).Okay, I just tried to compare the 'struct
vdpa_config_ops' and 'structvhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).struct
vdpa_config_ops{
/* Virtqueue ops */
int (*set_vq_address)(structvdpa_device*vdev,
u16 idx, u64desc_area, u64driver_area, u64device_area);
void (*set_vq_num)(structvdpa_device*vdev, u16 idx, u32 num); void (*kick_vq)(structvdpa_device*vdev, u16 idx);
void (*set_vq_cb)(structvdpa_device*vdev, u16 idx, structvdpa_callback*cb);
void (*set_vq_ready)(structvdpa_device*vdev, u16 idx, bool ready); bool (*get_vq_ready)(structvdpa_device*vdev, u16 idx); int (*set_vq_state)(structvdpa_device*vdev, u16 idx, const structvdpa_vq_state*state);
int (*get_vq_state)(structvdpa_device*vdev, u16 idx, structvdpa_vq_state*state);
structvdpa_notification_area(*get_vq_notification)(structvdpa_device*vdev, u16 idx); /* vq irq is not expected to be changed onceDRIVER_OKis set */ int (*get_vq_irq)(structvdpa_device*vdv, u16 idx);/* Device ops */
u32 (*get_vq_align)(structvdpa_device*vdev);
u64 (*get_features)(structvdpa_device*vdev);
int (*set_features)(structvdpa_device*vdev, u64 features); void (*set_config_cb)(structvdpa_device*vdev,
structvdpa_callback*cb);
u16 (*get_vq_num_max)(structvdpa_device*vdev);
u32 (*get_device_id)(structvdpa_device*vdev);
u32 (*get_vendor_id)(structvdpa_device*vdev);
u8 (*get_status)(structvdpa_device*vdev);
void (*set_status)(structvdpa_device*vdev, u8 status); void (*get_config)(structvdpa_device*vdev, unsigned int offset, void *buf, unsigned int len);
void (*set_config)(structvdpa_device*vdev, unsigned int offset, const void *buf, unsigned int len);
u32 (*get_generation)(structvdpa_device*vdev);/* DMA ops */
int (*set_map)(structvdpa_device*vdev, structvhost_iotlb*iotlb); int (*dma_map)(structvdpa_device*vdev, u64 iova, u64 size, u64 pa, u32 perm);
int (*dma_unmap)(structvdpa_device*vdev, u64 iova, u64 size);/* Free device resources */
void (*free)(structvdpa_device*vdev);
};+struct
vhost_config_ops{
+ int (*create_vqs)(structvhost_dev*vdev, unsigned int nvqs, + unsigned intnum_bufs, structvhost_virtqueue*vqs[], +vhost_vq_callback_t*callbacks[],
+ const char * const names[]);
+ void (*del_vqs)(structvhost_dev*vdev);
+ int (*write)(structvhost_dev*vdev, u64vhost_dst, void *src, int len);
+ int (*read)(structvhost_dev*vdev, void *dst, u64vhost_src, int len);
+ int (*set_features)(structvhost_dev*vdev, u64device_features); + int (*set_status)(structvhost_dev*vdev, u8 status); + u8 (*get_status)(structvhost_dev*vdev);(quoted diff) +2
-0+}; +struct
virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.I think it differs in (*set_vq_address)() and (*create_vqs)(). [
create_vqs()introduced in structvhost_config_opsprovides complimentary functionality to (*find_vqs)() in structvirtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].Back to the difference between (*set_vq_address)() and (*create_vqs)(),
set_vq_address()directly provides the virtqueue address to the vdpa device butcreate_vqs()only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.
Quoted 7 lines
Quoted 2 lines
I think vdpa always works with client drivers in userspace and providing userspace address for vring.
Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:
Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate vdpa_config_ops and invoke vdpa_register_device().
Quoted 4 lines
My question is basically for the part of
virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:
Even when we use vDPA, we have to use some sort of virtio_pci_epf_send_command() to communicate with virtio backend right?
Right, the layout is slightly different from the standard layout.
This is the layout
struct epf_vhost_reg_queue {
u8 cmd;
u8 cmd_status;
u16 status;
u16 num_buffers;
u16 msix_vector;
u64 queue_addr;
} __packed;struct epf_vhost_reg {
u64 host_features;
u64 guest_features;
u16 msix_config;
u16 num_queues;
u8 device_status;
u8 config_generation;
u32 isr;
u8 cmd;
u8 cmd_status;
struct epf_vhost_reg_queue vq[MAX_VQS];
} __packed;Quoted 23 lines
(quoted diff) +20
-0+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev, + u32 command) +{ + struct virtio_pci_epf *pci_epf; + void __iomem *ioaddr; + ktime_t timeout; + bool timedout; + int ret = 0; + u8 status; + + pci_epf = to_virtio_pci_epf(vp_dev); + ioaddr = vp_dev->ioaddr; + + mutex_lock(&pci_epf->lock); + writeb(command, ioaddr + HOST_CMD); + timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT); + while (1) { + timedout = ktime_after(ktime_get(), timeout); + status = readb(ioaddr + HOST_CMD_STATUS); +
Quoted 4 lines
Several questions:
- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of
HOST_CMD_STATUScan be changed.
The HOST_CMD (commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value in HOST_CMD. So HOST_CMD is less likely an issue.
A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in HOST_CMD_STATUS. After it expires, HOST_CMD_STATUS_NONE is written to HOST_CMD_STATUS. There could be case where EP updates HOST_CMD_STATUS after RC writes HOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.
Quoted 2 lines
If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.
Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?
Quoted 3 lines
- You have you vendor specific layout (according to
virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead
Okay, with vDPA, we are free to define our own layouts.
Quoted 3 lines
- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).
I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.
For 2, Isn't vhost-vdpa supposed to run on virtio backend?
From a high level, I think I should be able to use vDPA for virtio_pci_epf.c. Would you also suggest using vDPA for ntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).
Thanks
Kishon
Quoted 103 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 94 lines · Jason Wang
Hi Kishon:
Quoted 91 lines · Kishon Vijay Abraham I
Quoted 3 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the
queue_selis hard in the EP implementation).Okay, I just tried to compare the 'struct
vdpa_config_ops' and 'structvhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).struct
vdpa_config_ops{
/* Virtqueue ops */
int (*set_vq_address)(structvdpa_device*vdev,
u16 idx, u64desc_area, u64driver_area, u64device_area);
void (*set_vq_num)(structvdpa_device*vdev, u16 idx, u32 num); void (*kick_vq)(structvdpa_device*vdev, u16 idx);
void (*set_vq_cb)(structvdpa_device*vdev, u16 idx, structvdpa_callback*cb);
void (*set_vq_ready)(structvdpa_device*vdev, u16 idx, bool ready); bool (*get_vq_ready)(structvdpa_device*vdev, u16 idx); int (*set_vq_state)(structvdpa_device*vdev, u16 idx, const structvdpa_vq_state*state);
int (*get_vq_state)(structvdpa_device*vdev, u16 idx, structvdpa_vq_state*state);
structvdpa_notification_area(*get_vq_notification)(structvdpa_device*vdev, u16 idx); /* vq irq is not expected to be changed onceDRIVER_OKis set */ int (*get_vq_irq)(structvdpa_device*vdv, u16 idx);/* Device ops */
u32 (*get_vq_align)(structvdpa_device*vdev);
u64 (*get_features)(structvdpa_device*vdev);
int (*set_features)(structvdpa_device*vdev, u64 features); void (*set_config_cb)(structvdpa_device*vdev,
structvdpa_callback*cb);
u16 (*get_vq_num_max)(structvdpa_device*vdev);
u32 (*get_device_id)(structvdpa_device*vdev);
u32 (*get_vendor_id)(structvdpa_device*vdev);
u8 (*get_status)(structvdpa_device*vdev);
void (*set_status)(structvdpa_device*vdev, u8 status); void (*get_config)(structvdpa_device*vdev, unsigned int offset, void *buf, unsigned int len);
void (*set_config)(structvdpa_device*vdev, unsigned int offset, const void *buf, unsigned int len);
u32 (*get_generation)(structvdpa_device*vdev);/* DMA ops */
int (*set_map)(structvdpa_device*vdev, structvhost_iotlb*iotlb); int (*dma_map)(structvdpa_device*vdev, u64 iova, u64 size, u64 pa, u32 perm);
int (*dma_unmap)(structvdpa_device*vdev, u64 iova, u64 size);/* Free device resources */
void (*free)(structvdpa_device*vdev);
};+struct
vhost_config_ops{
+ int (*create_vqs)(structvhost_dev*vdev, unsigned int nvqs, + unsigned intnum_bufs, structvhost_virtqueue*vqs[], +vhost_vq_callback_t*callbacks[],
+ const char * const names[]);
+ void (*del_vqs)(structvhost_dev*vdev);
+ int (*write)(structvhost_dev*vdev, u64vhost_dst, void *src, int len);
+ int (*read)(structvhost_dev*vdev, void *dst, u64vhost_src, int len);
+ int (*set_features)(structvhost_dev*vdev, u64device_features); + int (*set_status)(structvhost_dev*vdev, u8 status); + u8 (*get_status)(structvhost_dev*vdev);(quoted diff) +2
-0+}; +struct
virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.I think it differs in (*set_vq_address)() and (*create_vqs)(). [
create_vqs()introduced in structvhost_config_opsprovides complimentary functionality to (*find_vqs)() in structvirtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].Back to the difference between (*set_vq_address)() and (*create_vqs)(),
set_vq_address()directly provides the virtqueue address to the vdpa device butcreate_vqs()only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.Quoted 6 lines
Quoted 2 lines
I think vdpa always works with client drivers in userspace and providing userspace address for vring.
Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:
Quoted 3 lines
Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate
vdpa_config_opsand invokevdpa_register_device().
Yes.
Quoted 5 lines
Quoted 3 lines
My question is basically for the part of
virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:Even when we use vDPA, we have to use some sort of
virtio_pci_epf_send_command()to communicate with virtio backend right?
Right.
Quoted 11 lines
Right, the layout is slightly different from the standard layout.
This is the layout
struct epf_vhost_reg_queue { u8 cmd; u8 cmd_status; u16 status; u16 num_buffers; u16 msix_vector; u64 queue_addr;
What's the meaning of queue_addr here?
Does not mean the device expects a contiguous memory for avail/desc/used ring?
Quoted 36 lines
}
__packed;struct epf_vhost_reg { u64 host_features; u64 guest_features; u16 msix_config; u16 num_queues; u8 device_status; u8 config_generation; u32 isr; u8 cmd; u8 cmd_status; struct epf_vhost_reg_queue vq[MAX_VQS]; } __packed;Quoted 22 lines
(quoted diff) +20
-0+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev, + u32 command) +{ + struct virtio_pci_epf *pci_epf; + void __iomem *ioaddr; + ktime_t timeout; + bool timedout; + int ret = 0; + u8 status; + + pci_epf = to_virtio_pci_epf(vp_dev); + ioaddr = vp_dev->ioaddr; + + mutex_lock(&pci_epf->lock); + writeb(command, ioaddr + HOST_CMD); + timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT); + while (1) { + timedout = ktime_after(ktime_get(), timeout); + status = readb(ioaddr + HOST_CMD_STATUS); +
Quoted 7 lines
Quoted 4 lines
Several questions:
- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of
HOST_CMD_STATUScan be changed.The
HOST_CMD(commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value inHOST_CMD. SoHOST_CMDis less likely an issue.
Here's my understanding of the protocol:
1) RC write to HOST_CMD
2) RC wait for HOST_CMD_STATUS to be HOST_CMD_STATUS_OKAY
It looks to me what EP should do is
1) EP reset HOST_CMD after reading new command
And it looks to me EP should also reset HOST_CMD_STATUS here?
(I thought there should be patch to handle stuffs like this but I didn't find it in this series)
Quoted 11 lines
A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in
HOST_CMD_STATUS. After it expires,HOST_CMD_STATUS_NONEis written toHOST_CMD_STATUS. There could be case where EP updatesHOST_CMD_STATUSafter RC writesHOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.Quoted 2 lines
If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.
Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?
Yes.
Quoted 4 lines
Quoted 3 lines
- You have you vendor specific layout (according to
virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver insteadOkay, with vDPA, we are free to define our own layouts.
Right, but vDPA have other requirements. E.g it requires the device have the ability to save/restore the state (e.g the last_avail_idx).
So it actually depends on what you want. If you don't care about userspace drivers and want to have a standard transport, you can still go virtio.
Quoted 6 lines
Quoted 3 lines
- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).
I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.
AFAIK you don't need to implement your own setup_vq and del_vq.
Quoted 2 lines
For 2, Isn't vhost-vdpa supposed to run on virtio backend?
Not currently, vDPA is a superset of virtio (e.g it support virtqueue state save/restore). This it should be possible in the future probably.
Quoted 5 lines
From a high level, I think I should be able to use vDPA for
virtio_pci_epf.c. Would you also suggest using vDPA forntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).
I think it's your call. If you want
1) a well-defined standard virtio transport
2) willing to finalize d and maintain the spec
3) doesn't care about userspace drivers
You can go with virtio, otherwise vDPA.
Thanks
Quoted 4 lines
Thanks
Kishon
Hi Jason,
Quoted 141 lines · Jason Wang
Quoted 108 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 98 lines · Jason Wang
Hi Kishon:
Quoted 95 lines · Kishon Vijay Abraham I
Quoted 4 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if
thequeue_selis hard in the EP implementation).Okay, I just tried to compare the 'struct
vdpa_config_ops' and 'structvhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).struct
vdpa_config_ops{
/* Virtqueue ops */
int (*set_vq_address)(structvdpa_device*vdev,
u16 idx, u64desc_area, u64driver_area, u64device_area);
void (*set_vq_num)(structvdpa_device*vdev, u16 idx, u32 num); void (*kick_vq)(structvdpa_device*vdev, u16 idx); void (*set_vq_cb)(structvdpa_device*vdev, u16 idx, structvdpa_callback*cb);
void (*set_vq_ready)(structvdpa_device*vdev, u16 idx, bool ready);
bool (*get_vq_ready)(structvdpa_device*vdev, u16 idx); int (*set_vq_state)(structvdpa_device*vdev, u16 idx, const structvdpa_vq_state*state);
int (*get_vq_state)(structvdpa_device*vdev, u16 idx, structvdpa_vq_state*state);
structvdpa_notification_area(*get_vq_notification)(structvdpa_device*vdev, u16 idx); /* vq irq is not expected to be changed onceDRIVER_OKis set */ int (*get_vq_irq)(structvdpa_device*vdv, u16 idx);/* Device ops */
u32 (*get_vq_align)(structvdpa_device*vdev);
u64 (*get_features)(structvdpa_device*vdev);
int (*set_features)(structvdpa_device*vdev, u64 features); void (*set_config_cb)(structvdpa_device*vdev,
structvdpa_callback*cb);
u16 (*get_vq_num_max)(structvdpa_device*vdev);
u32 (*get_device_id)(structvdpa_device*vdev);
u32 (*get_vendor_id)(structvdpa_device*vdev);
u8 (*get_status)(structvdpa_device*vdev);
void (*set_status)(structvdpa_device*vdev, u8 status); void (*get_config)(structvdpa_device*vdev, unsigned int offset, void *buf, unsigned int len);
void (*set_config)(structvdpa_device*vdev, unsigned int offset, const void *buf, unsigned int len);
u32 (*get_generation)(structvdpa_device*vdev);/* DMA ops */
int (*set_map)(structvdpa_device*vdev, structvhost_iotlb*iotlb);
int (*dma_map)(structvdpa_device*vdev, u64 iova, u64 size, u64 pa, u32 perm);
int (*dma_unmap)(structvdpa_device*vdev, u64 iova, u64 size);/* Free device resources */
void (*free)(structvdpa_device*vdev);
};+struct
vhost_config_ops{
+ int (*create_vqs)(structvhost_dev*vdev, unsigned int nvqs, + unsigned intnum_bufs, structvhost_virtqueue*vqs[], +vhost_vq_callback_t*callbacks[],
+ const char * const names[]);
+ void (*del_vqs)(structvhost_dev*vdev);
+ int (*write)(structvhost_dev*vdev, u64vhost_dst, void *src, int len);
+ int (*read)(structvhost_dev*vdev, void *dst, u64vhost_src, int len);
+ int (*set_features)(structvhost_dev*vdev, u64device_features); + int (*set_status)(structvhost_dev*vdev, u8 status); + u8 (*get_status)(structvhost_dev*vdev);(quoted diff) +2
-0+}; +struct
virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.I think it differs in (*set_vq_address)() and (*create_vqs)(). [
create_vqs()introduced in structvhost_config_opsprovides complimentary functionality to (*find_vqs)() in structvirtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].Back to the difference between (*set_vq_address)() and (*create_vqs)(),
set_vq_address()directly provides the virtqueue address to the vdpa device butcreate_vqs()only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly
provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.Quoted 7 lines
Quoted 3 lines
I think vdpa always works with client drivers in userspace and providing
userspace address for vring.Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:
Quoted 3 lines
Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate
vdpa_config_opsand invokevdpa_register_device().Yes.
Quoted 5 lines
Quoted 3 lines
My question is basically for the part of
virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:Even when we use vDPA, we have to use some sort of
virtio_pci_epf_send_command()to communicate with virtio backend right?Right.
Quoted 11 lines
Right, the layout is slightly different from the standard layout.
This is the layout
structepf_vhost_reg_queue{
u8 cmd;
u8cmd_status;
u16 status;
u16num_buffers;
u16msix_vector;
u64queue_addr;
Quoted 1 line
What's the meaning of
queue_addrhere?
Using queue_addr, the virtio front-end communicates the address of the allocated memory for virtqueue to the virtio back-end.
Quoted 3 lines
Does not mean the device expects a contiguous memory for avail/desc/used ring?
It's contiguous memory. Isn't this similar to other virtio transport (both PCI legacy and modern interface)?.
Quoted 48 lines
Quoted 39 lines
}
__packed;struct
epf_vhost_reg{
u64host_features;
u64guest_features;
u16msix_config;
u16num_queues;
u8device_status;
u8config_generation;
u32 isr;
u8 cmd;
u8cmd_status;
structepf_vhost_reg_queuevq[MAX_VQS];
}__packed;Quoted 23 lines
+static int virtio_pci_epf_send_command(struct
virtio_pci_device*vp_dev,
+ u32 command)(quoted diff) +18
-0+{ + struct virtio_pci_epf *pci_epf; + void __iomem *ioaddr; + ktime_t timeout; + bool timedout; + int ret = 0; + u8 status; + + pci_epf = to_virtio_pci_epf(vp_dev); + ioaddr = vp_dev->ioaddr; + + mutex_lock(&pci_epf->lock); + writeb(command, ioaddr + HOST_CMD); + timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT); + while (1) { + timedout = ktime_after(ktime_get(), timeout); + status = readb(ioaddr + HOST_CMD_STATUS); +Quoted 7 lines
Quoted 4 lines
Several questions:
- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of
HOST_CMD_STATUScan be changed.The
HOST_CMD(commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value inHOST_CMD. SoHOST_CMDis less likely an issue.
Quoted 4 lines
Here's my understanding of the protocol:
1) RC write to
HOST_CMD
2) RC wait forHOST_CMD_STATUSto beHOST_CMD_STATUS_OKAY
That's right!
Quoted 4 lines
It looks to me what EP should do is
1) EP reset
HOST_CMDafter reading new command
That's right! It does.
Quoted 2 lines
And it looks to me EP should also reset
HOST_CMD_STATUShere?
yeah, that would require RC to send another command to reset the status. Didn't see it required in the normal scenario but good to add this.
Quoted 3 lines
(I thought there should be patch to handle stuffs like this but I didn't find it in this series)
This is added in [RFC PATCH 19/22] PCI: endpoint: Add EP function driver to provide VHOST interface
pci_epf_vhost_cmd_handler() gets commands from RC using "reg->cmd;". On the EP side, it is local memory access (mapped to BAR memory exposed to the host) and hence accessed using structure member access.
Quoted 24 lines
Quoted 10 lines
A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in
HOST_CMD_STATUS. After it expires,HOST_CMD_STATUS_NONEis written toHOST_CMD_STATUS. There could be case where EP updatesHOST_CMD_STATUSafter RC writesHOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.Quoted 3 lines
Quoted 2 lines
If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.
Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?
Yes.
Quoted 4 lines
Quoted 3 lines
- You have you vendor specific layout (according to
virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver insteadOkay, with vDPA, we are free to define our own layouts.
Quoted 6 lines
Right, but vDPA have other requirements. E.g it requires the device have the ability to save/restore the state (e.g the
last_avail_idx).So it actually depends on what you want. If you don't care about userspace drivers and want to have a standard transport, you can still go virtio.
okay.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).
Quoted 7 lines
Quoted 3 lines
I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.
AFAIK you don't need to implement your own
setup_vqanddel_vq.
There should still be some entity that allocates memory for virtqueues and then communicate this address to the backend.
Maybe I have to look this further.
Quoted 16 lines
Quoted 3 lines
For 2, Isn't vhost-vdpa supposed to run on virtio backend?
Not currently, vDPA is a superset of virtio (e.g it support virtqueue state save/restore). This it should be possible in the future probably.
Quoted 5 lines
From a high level, I think I should be able to use vDPA for
virtio_pci_epf.c. Would you also suggest using vDPA forntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).
Quoted 5 lines
I think it's your call. If you want
1) a well-defined standard virtio transport
2) willing to finalize d and maintain the spec
3) doesn't care about userspace drivers
IIUC, we can use vDPA (virtio_vdpa.c) but still don't need userspace drivers right?
Quoted 2 lines
You can go with virtio, otherwise vDPA.
Okay, let me see. Thanks for your inputs.
Best Regards,
Kishon
Quoted 138 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 135 lines · Jason Wang
Quoted 107 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 98 lines · Jason Wang
Hi Kishon:
Quoted 95 lines · Kishon Vijay Abraham I
Quoted 4 lines
Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if
thequeue_selis hard in the EP implementation).Okay, I just tried to compare the 'struct
vdpa_config_ops' and 'structvhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).struct
vdpa_config_ops{
/* Virtqueue ops */
int (*set_vq_address)(structvdpa_device*vdev,
u16 idx, u64desc_area, u64driver_area, u64device_area);
void (*set_vq_num)(structvdpa_device*vdev, u16 idx, u32 num); void (*kick_vq)(structvdpa_device*vdev, u16 idx); void (*set_vq_cb)(structvdpa_device*vdev, u16 idx, structvdpa_callback*cb);
void (*set_vq_ready)(structvdpa_device*vdev, u16 idx, bool
ready);
bool (*get_vq_ready)(structvdpa_device*vdev, u16 idx); int (*set_vq_state)(structvdpa_device*vdev, u16 idx, const structvdpa_vq_state*state);
int (*get_vq_state)(structvdpa_device*vdev, u16 idx, structvdpa_vq_state*state);
structvdpa_notification_area(*get_vq_notification)(structvdpa_device*vdev, u16 idx); /* vq irq is not expected to be changed onceDRIVER_OKis set */ int (*get_vq_irq)(structvdpa_device*vdv, u16 idx);/* Device ops */
u32 (*get_vq_align)(structvdpa_device*vdev);
u64 (*get_features)(structvdpa_device*vdev);
int (*set_features)(structvdpa_device*vdev, u64 features); void (*set_config_cb)(structvdpa_device*vdev,
structvdpa_callback*cb);
u16 (*get_vq_num_max)(structvdpa_device*vdev);
u32 (*get_device_id)(structvdpa_device*vdev);
u32 (*get_vendor_id)(structvdpa_device*vdev);
u8 (*get_status)(structvdpa_device*vdev);
void (*set_status)(structvdpa_device*vdev, u8 status); void (*get_config)(structvdpa_device*vdev, unsigned int offset, void *buf, unsigned int len);
void (*set_config)(structvdpa_device*vdev, unsigned int offset, const void *buf, unsigned int len);
u32 (*get_generation)(structvdpa_device*vdev);/* DMA ops */
int (*set_map)(structvdpa_device*vdev, structvhost_iotlb
*iotlb);
int (*dma_map)(structvdpa_device*vdev, u64 iova, u64 size, u64 pa, u32 perm);
int (*dma_unmap)(structvdpa_device*vdev, u64 iova, u64 size);/* Free device resources */
void (*free)(structvdpa_device*vdev);
};+struct
vhost_config_ops{
+ int (*create_vqs)(structvhost_dev*vdev, unsigned int nvqs, + unsigned intnum_bufs, structvhost_virtqueue*vqs[], +vhost_vq_callback_t*callbacks[],
+ const char * const names[]);
+ void (*del_vqs)(structvhost_dev*vdev);
+ int (*write)(structvhost_dev*vdev, u64vhost_dst, void *src, int len);
+ int (*read)(structvhost_dev*vdev, void *dst, u64vhost_src, int len);
+ int (*set_features)(structvhost_dev*vdev, u64device_features); + int (*set_status)(structvhost_dev*vdev, u8 status); + u8 (*get_status)(structvhost_dev*vdev);(quoted diff) +2
-0+}; +struct
virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.I think it differs in (*set_vq_address)() and (*create_vqs)(). [
create_vqs()introduced in structvhost_config_opsprovides complimentary functionality to (*find_vqs)() in structvirtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].Back to the difference between (*set_vq_address)() and (*create_vqs)(),
set_vq_address()directly provides the virtqueue address to the vdpa device butcreate_vqs()only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly
provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.Quoted 6 lines
Quoted 3 lines
I think vdpa always works with client drivers in userspace and providing
userspace address for vring.Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:
Quoted 3 lines
Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate
vdpa_config_opsand invokevdpa_register_device().Yes.
Quoted 5 lines
Quoted 3 lines
My question is basically for the part of
virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:Even when we use vDPA, we have to use some sort of
virtio_pci_epf_send_command()to communicate with virtio backend right?Right.
Quoted 10 lines
Right, the layout is slightly different from the standard layout.
This is the layout
structepf_vhost_reg_queue{
u8 cmd;
u8cmd_status;
u16 status;
u16num_buffers;
u16msix_vector;
u64queue_addr;
Quoted 7 lines
Quoted 1 line
What's the meaning of
queue_addrhere?Using
queue_addr, the virtio front-end communicates the address of the allocated memory for virtqueue to the virtio back-end.Quoted 2 lines
Does not mean the device expects a contiguous memory for avail/desc/used ring?
It's contiguous memory. Isn't this similar to other virtio transport (both PCI legacy and modern interface)?.
That's only for legacy device, for modern device we don't have such restriction.
Quoted 57 lines
Quoted 45 lines
Quoted 37 lines
}
__packed;struct
epf_vhost_reg{
u64host_features;
u64guest_features;
u16msix_config;
u16num_queues;
u8device_status;
u8config_generation;
u32 isr;
u8 cmd;
u8cmd_status;
structepf_vhost_reg_queuevq[MAX_VQS];
}__packed;Quoted 22 lines
+static int virtio_pci_epf_send_command(struct
virtio_pci_device*vp_dev,
+ u32 command)(quoted diff) +18
-0+{ + struct virtio_pci_epf *pci_epf; + void __iomem *ioaddr; + ktime_t timeout; + bool timedout; + int ret = 0; + u8 status; + + pci_epf = to_virtio_pci_epf(vp_dev); + ioaddr = vp_dev->ioaddr; + + mutex_lock(&pci_epf->lock); + writeb(command, ioaddr + HOST_CMD); + timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT); + while (1) { + timedout = ktime_after(ktime_get(), timeout); + status = readb(ioaddr + HOST_CMD_STATUS); +Quoted 7 lines
Quoted 4 lines
Several questions:
- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of
HOST_CMD_STATUScan be changed.The
HOST_CMD(commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value inHOST_CMD. SoHOST_CMDis less likely an issue.Quoted 4 lines
Here's my understanding of the protocol:
1) RC write to
HOST_CMD
2) RC wait forHOST_CMD_STATUSto beHOST_CMD_STATUS_OKAYThat's right!
Quoted 3 lines
It looks to me what EP should do is
1) EP reset
HOST_CMDafter reading new commandThat's right! It does.
Quoted 1 line
And it looks to me EP should also reset
HOST_CMD_STATUShere?yeah, that would require RC to send another command to reset the status. Didn't see it required in the normal scenario but good to add this.
Quoted 8 lines
Quoted 2 lines
(I thought there should be patch to handle stuffs like this but I didn't find it in this series)
This is added in [RFC PATCH 19/22] PCI: endpoint: Add EP function driver to provide VHOST interface
pci_epf_vhost_cmd_handler()gets commands from RC using "reg->cmd;". On the EP side, it is local memory access (mapped to BAR memory exposed to the host) and hence accessed using structure member access.
Thanks for the pointer, will have a look at and I think this part need to be carefully designed and the key to the success of the epf transport.
Mathieu Poirier
maintainerHey Kishon,
Quoted 316 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ +| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 4 lines
all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 8 lines
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 9 lines
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of Xokay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 6 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Quoted 8 lines
Quoted 5 lines
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Quoted 7 lines
Quoted 4 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Quoted 25 lines
Quoted 5 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 2 lines
Thanks again for your suggestions!
You're welcome.
Quoted 8 lines
Quoted 4 lines
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Got it. My initial design was based on my understanding of your comments [1].
I'll try to create something based on your proposed design here.
Based on the above conversation it seems like I should wait for another revision of this set before reviewing the RPMSG part. Please confirm that my understanding is correct.
Thanks,
Mathieu
Quoted 133 lines
Regards
Kishon[1] ->
https://lore.kernel.org/linux-pci/[email protected]/Quoted 127 lines
Thanks
Quoted 122 lines
Regards
KishonQuoted 118 lines
Quoted 9 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 1 line
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 98 lines
Regards
KishonQuoted 95 lines
Thanks
Quoted 92 lines
Thanks
KishonQuoted 89 lines
Thanks
Quoted 86 lines
Quoted 86 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144
++++++++++++++++
drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151
+++++++++++++++++
drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +-drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +-drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {
drivers/vhost=> include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +-39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)--
2.17.1
Hi Mathieu,
Quoted 324 lines · Mathieu Poirier
Hey Kishon,
Quoted 321 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I
Hi Jason,
Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG + +
| | | | | | | | +------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+This was initially proposed here [1]
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.gitvhost_rpmsg_pci_ntb_rfcThanks
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.
Right, this is something similar to VOP (
Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication#ls
drivers/ntb/hw/
amd epf idt intel msccQuoted 8 lines
And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication
# ls drivers/pci/controller/dwc/*ep* drivers/pci/controller/dwc/pcie-designware-ep.c drivers/pci/controller/dwc/pcie-uniphier-ep.c drivers/pci/controller/dwc/pci-layerscape-ep.cThanks for those backgrounds.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)Quoted 8 lines
Quoted 3 lines
(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.
Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.
Ok.
Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),I see.
Quoted 7 lines
Quoted 4 lines
all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.
IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.Having the same bus can also create sequencing issues.
Quoted 8 lines
If you look at
virtio_dev_probe()ofvirtio_bus
device_features=dev->config->get_features(dev);Now if we use same bus for both front-end and back-end, both will try to
get_featureswhen there has been noset_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b
Quoted 6 lines
virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) setsVIRTIO_CONFIG_S_DRIVER_OKwhereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).Quoted 8 lines
Quoted 5 lines
epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.
Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?
Quoted 9 lines
I think it can work as:
1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of Xokay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.
Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.It works like:
virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>
The advantages is that there's no need for writing new buses and drivers.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
I think this will work however there is an addtional copy between vringh queue and virtqueue,
I think not? E.g in use case 1), if we stick to virtio bus, we will have:
virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)
IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
Yes.
Quoted 2 lines
And virtio
ring(2) is created by virtio pci (RC).Yes.
Quoted 6 lines
Quoted 2 lines
What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?
okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).
Quoted 8 lines
Quoted 5 lines
Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)Here the NTB(HOST1) will access the virtio ring(2) using vringh?
Yes, I think so it needs to use vring to access virtio ring (1) as well.
NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
Quoted 7 lines
Quoted 4 lines
Do you also think this will work seamlessly with
virtio_net.c,virtio_blk.c?Yes.
okay, I haven't looked at this but the backend of
virtio_blkshould access an actual storage device no?Quoted 29 lines
Quoted 5 lines
I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)
We can have have an array of pre-defined features indexed by virtio device id in the code.
Quoted 2 lines
and the other is how certain inputs to virtio device such as number of buffers be determined.
We can start from hard coded the value like 256, or introduce some API for user to change the value.
Quoted 2 lines
Thanks again for your suggestions!
You're welcome.
Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.
Quoted 7 lines
Quoted 3 lines
Got it. My initial design was based on my understanding of your comments [1].
I'll try to create something based on your proposed design here.
Based on the above conversation it seems like I should wait for another revision of this set before reviewing the RPMSG part. Please confirm that my understanding is correct.
Right, there are multiple parts in this series that has to be aligned. I'd still think irrespective of the approach something like Address Service Notification support might have to be supported by rpmsg.
Thanks
Kishon
Quoted 137 lines
Thanks,
MathieuQuoted 133 lines
Regards
Kishon[1] ->
https://lore.kernel.org/linux-pci/[email protected]/Quoted 127 lines
Thanks
Quoted 122 lines
Regards
KishonQuoted 118 lines
Quoted 9 lines
in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.
Quoted 1 line
Does this make sense?
Two copies in my opinion is an issue but lets get others opinions as well.
Sure.
Quoted 1 line
Thanks for your suggestions!
You're welcome.
Thanks
Quoted 98 lines
Regards
KishonQuoted 95 lines
Thanks
Quoted 92 lines
Thanks
KishonQuoted 89 lines
Thanks
Quoted 86 lines
Quoted 86 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure torpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()andvirtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe
Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the
documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-)create mode 100644
drivers/ntb/ntb_vhost.c
create mode 100644drivers/ntb/ntb_virtio.c
create mode 100644drivers/ntb/ntb_virtio.h
create mode 100644drivers/pci/endpoint/functions/pci-epf-vhost.ccreate mode 100644drivers/rpmsg/rpmsg_cfs.c
create mode 100644drivers/rpmsg/vhost_rpmsg_bus.ccreate mode 100644drivers/vhost/vhost_cfs.c
create mode 100644drivers/virtio/virtio_pci_epf.crename {drivers/vhost=>include/linux}/vhost.h(66%)Signature · 1 line
2.17.1
Hi Michael,
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ + | | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |+------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
Quoted 7 lines
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Great to hear! Thanks in advance for reviewing!
Regards
Kishon
Quoted 85 lines
Quoted 84 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-) create mode 100644 drivers/ntb/ntb_vhost.c create mode 100644 drivers/ntb/ntb_virtio.c create mode 100644 drivers/ntb/ntb_virtio.h create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)Signature · 1 line
2.17.1
Mathieu Poirier
maintainerQuoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ + | | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |+------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+
Quoted 7 lines
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Same here - it will take time. This patchset is sizable and sits behind a few others that are equally big.
Quoted 85 lines
Quoted 84 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-) create mode 100644 drivers/ntb/ntb_vhost.c create mode 100644 drivers/ntb/ntb_virtio.c create mode 100644 drivers/ntb/ntb_virtio.h create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)--
2.17.1
+Alan, Haotian
Quoted 180 lines · Mathieu Poirier
Quoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I
This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the componentsUseCase1 :
VHOST RPMSG VIRTIO RPMSG
+ + | | | | | | | | +-----v------+ +------v-------+ | Linux | | Linux | | Endpoint | | Root Complex | | <-----------------> | | | | | | SOC1 | | SOC2 | +------------+ +--------------+UseCase 2:
VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |+------v------+ +------v------+ | | | | | HOST1 | | HOST2 | | | | | +------^------+ +------^------+ | | | | +---------------------------------------------------------------------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | SoC With Multiple EP Instances | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+Software Layering:
The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.
+----------------+ +-----------+ +------------+ +----------+ | RPMSG VHOST | | NET VHOST | | SCSI VHOST | | X | +-------^--------+ +-----^-----+ +-----^------+ +----^-----+ | | | | | | | | | | | | +-----------v-----------------v--------------v--------------v----------+ | VHOST CORE | +--------^---------------^--------------------^------------------^-----+ | | | | | | | | | | | | +--------v-------+ +----v------+ +----------v----------+ +----v-----+ | PCI EPF VHOST | | NTB VHOST | |PLATFORM DEVICE VHOST| | X | +----------------+ +-----------+ +---------------------+ +----------+Quoted 7 lines
Quoted 3 lines
This was initially proposed here [1]
I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!
Same here - it will take time. This patchset is sizable and sits behind a few others that are equally big.
Quoted 85 lines
Quoted 84 lines
Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Userequest_threaded_irq()instead ofrequest_irq()
rpmsg:virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg:virtio_rpmsg_bus: Add Address Service Notification support
rpmsg:virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg:virtio_rpmsg_bus: Usevirtio_alloc_buffer()and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bussamples/rpmsg: Setup delayed work to send messagesamples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe thentb_virtioandntb_vhostclient in the documentationDocumentation/driver-api/ntb.rst | 11 + Documentation/rpmsg.txt | 56 + drivers/ntb/Kconfig | 18 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_vhost.c | 776 +++++++++++ drivers/ntb/ntb_virtio.c | 853 ++++++++++++ drivers/ntb/ntb_virtio.h | 56 + drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig | 10 + drivers/rpmsg/Makefile | 3 +- drivers/rpmsg/rpmsg_cfs.c | 394 ++++++ drivers/rpmsg/rpmsg_core.c | 7 + drivers/rpmsg/rpmsg_internal.h | 136 ++ drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++- drivers/vhost/Kconfig | 1 + drivers/vhost/Makefile | 2 +- drivers/vhost/net.c | 10 +- drivers/vhost/scsi.c | 24 +- drivers/vhost/test.c | 17 +- drivers/vhost/vdpa.c | 2 +- drivers/vhost/vhost.c | 730 ++++++++++- drivers/vhost/vhost_cfs.c | 341 +++++ drivers/vhost/vringh.c | 332 +++++ drivers/vhost/vsock.c | 20 +- drivers/virtio/Kconfig | 9 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pci_common.c | 25 +- drivers/virtio/virtio_pci_epf.c | 670 ++++++++++ include/linux/mod_devicetable.h | 6 + include/linux/rpmsg.h | 6 + {drivers/vhost => include/linux}/vhost.h | 132 +- include/linux/virtio.h | 3 + include/linux/virtio_config.h | 42 + include/linux/vringh.h | 46 + samples/rpmsg/rpmsg_client_sample.c | 32 +- tools/virtio/virtio_test.c | 2 +- 39 files changed, 7083 insertions(+), 183 deletions(-) create mode 100644 drivers/ntb/ntb_vhost.c create mode 100644 drivers/ntb/ntb_virtio.c create mode 100644 drivers/ntb/ntb_virtio.h create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)--
2.17.1