Skip to contentSkip to conversation

Enhance VHOST to enable SoC-to-SoC communication

RFC PATCH In review Kishon Vijay Abraham I 5 participants 49 messages Jul 2, 2020 – Sep 18, 2020

Subsystems (12)
lore
Share & tools
lore
#0 lore raw

cover Enhance VHOST to enable SoC-to-SoC communication

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

      +                               +
      |                               |
      |                               |
      |                               |
      |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |

   +------v------+                                   +------v------+
   |             |                                   |             |
   |    HOST1    |                                   |    HOST2    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

    +----------------+  +-----------+  +------------+  +----------+
    |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
    +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
            |                 |              |              |
            |                 |              |              |
            |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
         |               |                    |                  |
         |               |                    |                  |
         |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst              |   11 +
 Documentation/rpmsg.txt                       |   56 +
 drivers/ntb/Kconfig                           |   18 +
 drivers/ntb/Makefile                          |    2 +
 drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
 drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
 drivers/ntb/ntb_virtio.h                      |   56 +
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig                         |   10 +
 drivers/rpmsg/Makefile                        |    3 +-
 drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
 drivers/rpmsg/rpmsg_core.c                    |    7 +
 drivers/rpmsg/rpmsg_internal.h                |  136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
 drivers/vhost/Kconfig                         |    1 +
 drivers/vhost/Makefile                        |    2 +-
 drivers/vhost/net.c                           |   10 +-
 drivers/vhost/scsi.c                          |   24 +-
 drivers/vhost/test.c                          |   17 +-
 drivers/vhost/vdpa.c                          |    2 +-
 drivers/vhost/vhost.c                         |  730 ++++++++++-
 drivers/vhost/vhost_cfs.c                     |  341 +++++
 drivers/vhost/vringh.c                        |  332 +++++
 drivers/vhost/vsock.c                         |   20 +-
 drivers/virtio/Kconfig                        |    9 +
 drivers/virtio/Makefile                       |    1 +
 drivers/virtio/virtio_pci_common.c            |   25 +-
 drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
 include/linux/mod_devicetable.h               |    6 +
 include/linux/rpmsg.h                         |    6 +
 {drivers/vhost => include/linux}/vhost.h      |  132 +-
 include/linux/virtio.h                        |    3 +
 include/linux/virtio_config.h                 |   42 +
 include/linux/vringh.h                        |   46 +
 samples/rpmsg/rpmsg_client_sample.c           |   32 +-
 tools/virtio/virtio_test.c                    |    2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH 1/22 vhost: Make _feature_ bits a property of vhost device

No functional change intended. The feature bits defined in virtio specification is associated with virtio device and not virtqueue. In order to correctly reflect this in the vhost backend, remove "acked_features" from struct vhost_virtqueue and add "features" in struct vhost_dev. This will also make it look symmetrical to virtio in guest.

---
 drivers/vhost/net.c   |  7 ++++---
 drivers/vhost/scsi.c  | 22 ++++++++--------------
 drivers/vhost/test.c  | 14 +++++---------
 drivers/vhost/vhost.c | 33 +++++++++++++++++++++------------
 drivers/vhost/vhost.h |  6 +++---
 drivers/vhost/vsock.c | 18 ++++++------------
 6 files changed, 47 insertions(+), 53 deletions(-)
drivers/vhost/net.c +4 -3
index 516519dcc8ff..437126219116 100644
@@ -1137,9 +1137,9 @@ static void handle_rx(struct vhost_net *net)
 	vhost_hlen = nvq->vhost_hlen;
 	sock_hlen = nvq->sock_hlen;
 
-	vq_log = unlikely(vhost_has_feature(vq, VHOST_F_LOG_ALL)) ?
+	vq_log = unlikely(vhost_has_feature(&net->dev, VHOST_F_LOG_ALL)) ?
 		vq->log : NULL;
-	mergeable = vhost_has_feature(vq, VIRTIO_NET_F_MRG_RXBUF);
+	mergeable = vhost_has_feature(&net->dev, VIRTIO_NET_F_MRG_RXBUF);
 
 	do {
 		sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
@@ -1633,6 +1633,7 @@ static int vhost_net_set_backend_features(struct vhost_net *n, u64 features)
 static int vhost_net_set_features(struct vhost_net *n, u64 features)
 {
 	size_t vhost_hlen, sock_hlen, hdr_len;
+	struct vhost_dev *vdev = &n->dev;
 	int i;
 
 	hdr_len = (features & ((1ULL << VIRTIO_NET_F_MRG_RXBUF) |
@@ -1658,9 +1659,9 @@ static int vhost_net_set_features(struct vhost_net *n, u64 features)
 			goto out_unlock;
 	}
 
+	vdev->features = features;
 	for (i = 0; i < VHOST_NET_VQ_MAX; ++i) {
 		mutex_lock(&n->vqs[i].vq.mutex);
-		n->vqs[i].vq.acked_features = features;
 		n->vqs[i].vhost_hlen = vhost_hlen;
 		n->vqs[i].sock_hlen = sock_hlen;
 		mutex_unlock(&n->vqs[i].vq.mutex);
drivers/vhost/scsi.c +8 -14
index 8b104f76f324..f5138379659e 100644
@@ -921,7 +921,7 @@ vhost_scsi_handle_vq(struct vhost_scsi *vs, struct vhost_virtqueue *vq)
 	int ret, prot_bytes, c = 0;
 	u16 lun;
 	u8 task_attr;
-	bool t10_pi = vhost_has_feature(vq, VIRTIO_SCSI_F_T10_PI);
+	bool t10_pi = vhost_has_feature(&vs->dev, VIRTIO_SCSI_F_T10_PI);
 	void *cdb;
 
 	mutex_lock(&vq->mutex);
@@ -1573,26 +1573,20 @@ vhost_scsi_clear_endpoint(struct vhost_scsi *vs,
 
 static int vhost_scsi_set_features(struct vhost_scsi *vs, u64 features)
 {
-	struct vhost_virtqueue *vq;
-	int i;
+	struct vhost_dev *vdev = &vs->dev;
 
 	if (features & ~VHOST_SCSI_FEATURES)
 		return -EOPNOTSUPP;
 
-	mutex_lock(&vs->dev.mutex);
+	mutex_lock(&vdev->mutex);
 	if ((features & (1 << VHOST_F_LOG_ALL)) &&
-	    !vhost_log_access_ok(&vs->dev)) {
-		mutex_unlock(&vs->dev.mutex);
+	    !vhost_log_access_ok(vdev)) {
+		mutex_unlock(&vdev->mutex);
 		return -EFAULT;
 	}
 
-	for (i = 0; i < VHOST_SCSI_MAX_VQ; i++) {
-		vq = &vs->vqs[i].vq;
-		mutex_lock(&vq->mutex);
-		vq->acked_features = features;
-		mutex_unlock(&vq->mutex);
-	}
-	mutex_unlock(&vs->dev.mutex);
+	vdev->features = features;
+	mutex_unlock(&vdev->mutex);
 	return 0;
 }
 
@@ -1789,7 +1783,7 @@ vhost_scsi_do_plug(struct vhost_scsi_tpg *tpg,
 
 	vq = &vs->vqs[VHOST_SCSI_VQ_EVT].vq;
 	mutex_lock(&vq->mutex);
-	if (vhost_has_feature(vq, VIRTIO_SCSI_F_HOTPLUG))
+	if (vhost_has_feature(&vs->dev, VIRTIO_SCSI_F_HOTPLUG))
 		vhost_scsi_send_evt(vs, tpg, lun,
 				   VIRTIO_SCSI_T_TRANSPORT_RESET, reason);
 	mutex_unlock(&vq->mutex);
drivers/vhost/test.c +5 -9
index 9a3a09005e03..6518b48c0633 100644
@@ -247,19 +247,15 @@ static long vhost_test_reset_owner(struct vhost_test *n)
 
 static int vhost_test_set_features(struct vhost_test *n, u64 features)
 {
-	struct vhost_virtqueue *vq;
+	struct vhost_dev *vdev = &n->dev;
 
-	mutex_lock(&n->dev.mutex);
+	mutex_lock(&vdev->mutex);
 	if ((features & (1 << VHOST_F_LOG_ALL)) &&
-	    !vhost_log_access_ok(&n->dev)) {
-		mutex_unlock(&n->dev.mutex);
+	    !vhost_log_access_ok(vdev)) {
+		mutex_unlock(&vdev->mutex);
 		return -EFAULT;
 	}
-	vq = &n->vqs[VHOST_TEST_VQ];
-	mutex_lock(&vq->mutex);
-	vq->acked_features = features;
-	mutex_unlock(&vq->mutex);
-	mutex_unlock(&n->dev.mutex);
+	mutex_unlock(&vdev->mutex);
 	return 0;
 }
 
drivers/vhost/vhost.c +21 -12
index 21a59b598ed8..3c2633fb519d 100644
@@ -104,12 +104,14 @@ static long vhost_get_vring_endian(struct vhost_virtqueue *vq, u32 idx,
 
 static void vhost_init_is_le(struct vhost_virtqueue *vq)
 {
+	struct vhost_dev *vdev = vq->dev;
+
 	/* Note for legacy virtio: user_be is initialized at reset time
 	 * according to the host endianness. If userspace does not set an
 	 * explicit endianness, the default behavior is native endian, as
 	 * expected by legacy virtio.
 	 */
-	vq->is_le = vhost_has_feature(vq, VIRTIO_F_VERSION_1) || !vq->user_be;
+	vq->is_le = vhost_has_feature(vdev, VIRTIO_F_VERSION_1) || !vq->user_be;
 }
 #else
 static void vhost_disable_cross_endian(struct vhost_virtqueue *vq)
@@ -129,7 +131,9 @@ static long vhost_get_vring_endian(struct vhost_virtqueue *vq, u32 idx,
 
 static void vhost_init_is_le(struct vhost_virtqueue *vq)
 {
-	vq->is_le = vhost_has_feature(vq, VIRTIO_F_VERSION_1)
+	struct vhost_dev *vdev = vq->dev;
+
+	vq->is_le = vhost_has_feature(vdev, VIRTIO_F_VERSION_1)
 		|| virtio_legacy_is_little_endian();
 }
 #endif /* CONFIG_VHOST_CROSS_ENDIAN_LEGACY */
@@ -310,7 +314,6 @@ static void vhost_vq_reset(struct vhost_dev *dev,
 	vq->log_used = false;
 	vq->log_addr = -1ull;
 	vq->private_data = NULL;
-	vq->acked_features = 0;
 	vq->acked_backend_features = 0;
 	vq->log_base = NULL;
 	vq->error_ctx = NULL;
@@ -428,8 +431,9 @@ EXPORT_SYMBOL_GPL(vhost_exceeds_weight);
 static size_t vhost_get_avail_size(struct vhost_virtqueue *vq,
 				   unsigned int num)
 {
+	struct vhost_dev *vdev = vq->dev;
 	size_t event __maybe_unused =
-	       vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0;
+	       vhost_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0;
 
 	return sizeof(*vq->avail) +
 	       sizeof(*vq->avail->ring) * num + event;
@@ -438,8 +442,9 @@ static size_t vhost_get_avail_size(struct vhost_virtqueue *vq,
 static size_t vhost_get_used_size(struct vhost_virtqueue *vq,
 				  unsigned int num)
 {
+	struct vhost_dev *vdev = vq->dev;
 	size_t event __maybe_unused =
-	       vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0;
+	       vhost_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0;
 
 	return sizeof(*vq->used) +
 	       sizeof(*vq->used->ring) * num + event;
@@ -468,6 +473,7 @@ void vhost_dev_init(struct vhost_dev *dev,
 	dev->iotlb = NULL;
 	dev->mm = NULL;
 	dev->worker = NULL;
+	dev->features = 0;
 	dev->iov_limit = iov_limit;
 	dev->weight = weight;
 	dev->byte_weight = byte_weight;
@@ -738,14 +744,15 @@ static inline void __user *vhost_vq_meta_fetch(struct vhost_virtqueue *vq,
 static bool memory_access_ok(struct vhost_dev *d, struct vhost_iotlb *umem,
 			     int log_all)
 {
+	bool log;
 	int i;
 
+	log = log_all || vhost_has_feature(d, VHOST_F_LOG_ALL);
+
 	for (i = 0; i < d->nvqs; ++i) {
 		bool ok;
-		bool log;
 
 		mutex_lock(&d->vqs[i]->mutex);
-		log = log_all || vhost_has_feature(d->vqs[i], VHOST_F_LOG_ALL);
 		/* If ring is inactive, will check when it's enabled. */
 		if (d->vqs[i]->private_data)
 			ok = vq_memory_access_ok(d->vqs[i]->log_base,
@@ -1329,8 +1336,10 @@ EXPORT_SYMBOL_GPL(vhost_log_access_ok);
 static bool vq_log_access_ok(struct vhost_virtqueue *vq,
 			     void __user *log_base)
 {
+	struct vhost_dev *vdev = vq->dev;
+
 	return vq_memory_access_ok(log_base, vq->umem,
-				   vhost_has_feature(vq, VHOST_F_LOG_ALL)) &&
+				   vhost_has_feature(vdev, VHOST_F_LOG_ALL)) &&
 		(!vq->log_used || log_access_ok(log_base, vq->log_addr,
 				  vhost_get_used_size(vq, vq->num)));
 }
@@ -2376,11 +2385,11 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 	 * interrupts. */
 	smp_mb();
 
-	if (vhost_has_feature(vq, VIRTIO_F_NOTIFY_ON_EMPTY) &&
+	if (vhost_has_feature(dev, VIRTIO_F_NOTIFY_ON_EMPTY) &&
 	    unlikely(vq->avail_idx == vq->last_avail_idx))
 		return true;
 
-	if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) {
+	if (!vhost_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) {
 		__virtio16 flags;
 		if (vhost_get_avail_flags(vq, &flags)) {
 			vq_err(vq, "Failed to get flags");
@@ -2459,7 +2468,7 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 	if (!(vq->used_flags & VRING_USED_F_NO_NOTIFY))
 		return false;
 	vq->used_flags &= ~VRING_USED_F_NO_NOTIFY;
-	if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) {
+	if (!vhost_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) {
 		r = vhost_update_used_flags(vq);
 		if (r) {
 			vq_err(vq, "Failed to enable notification at %p: %d\n",
@@ -2496,7 +2505,7 @@ void vhost_disable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 	if (vq->used_flags & VRING_USED_F_NO_NOTIFY)
 		return;
 	vq->used_flags |= VRING_USED_F_NO_NOTIFY;
-	if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) {
+	if (!vhost_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) {
 		r = vhost_update_used_flags(vq);
 		if (r)
 			vq_err(vq, "Failed to enable notification at %p: %d\n",
drivers/vhost/vhost.h +3 -3
index f8403bd46b85..5d1d00363e79 100644
@@ -111,7 +111,6 @@ struct vhost_virtqueue {
 	struct vhost_iotlb *umem;
 	struct vhost_iotlb *iotlb;
 	void *private_data;
-	u64 acked_features;
 	u64 acked_backend_features;
 	/* Log write descriptors */
 	void __user *log_base;
@@ -140,6 +139,7 @@ struct vhost_dev {
 	struct mm_struct *mm;
 	struct mutex mutex;
 	struct vhost_virtqueue **vqs;
+	u64 features;
 	int nvqs;
 	struct eventfd_ctx *log_ctx;
 	struct llist_head work_list;
@@ -258,9 +258,9 @@ static inline void *vhost_vq_get_backend(struct vhost_virtqueue *vq)
 	return vq->private_data;
 }
 
-static inline bool vhost_has_feature(struct vhost_virtqueue *vq, int bit)
+static inline bool vhost_has_feature(struct vhost_dev *vdev, int bit)
 {
-	return vq->acked_features & (1ULL << bit);
+	return vdev->features & (1ULL << bit);
 }
 
 static inline bool vhost_backend_has_feature(struct vhost_virtqueue *vq, int bit)
drivers/vhost/vsock.c +6 -12
index fb4e944c4d0d..8317ad026e3d 100644
@@ -757,26 +757,20 @@ static int vhost_vsock_set_cid(struct vhost_vsock *vsock, u64 guest_cid)
 
 static int vhost_vsock_set_features(struct vhost_vsock *vsock, u64 features)
 {
-	struct vhost_virtqueue *vq;
-	int i;
+	struct vhost_dev *vdev = &vsock->dev;
 
 	if (features & ~VHOST_VSOCK_FEATURES)
 		return -EOPNOTSUPP;
 
-	mutex_lock(&vsock->dev.mutex);
+	mutex_lock(&vdev->mutex);
 	if ((features & (1 << VHOST_F_LOG_ALL)) &&
-	    !vhost_log_access_ok(&vsock->dev)) {
-		mutex_unlock(&vsock->dev.mutex);
+	    !vhost_log_access_ok(vdev)) {
+		mutex_unlock(&vdev->mutex);
 		return -EFAULT;
 	}
 
-	for (i = 0; i < ARRAY_SIZE(vsock->vqs); i++) {
-		vq = &vsock->vqs[i];
-		mutex_lock(&vq->mutex);
-		vq->acked_features = features;
-		mutex_unlock(&vq->mutex);
-	}
-	mutex_unlock(&vsock->dev.mutex);
+	vdev->features = features;
+	mutex_unlock(&vdev->mutex);
 	return 0;
 }
 
Signature · 1 line
2.17.1
#2 lore raw

RFC PATCH 2/22 vhost: Introduce standard Linux driver model in VHOST

In reply to Kishon Vijay Abraham I

Introduce standard driver model in VHOST. This will facilitate using multiple VHOST drivers (like net, scsi etc.,) over different VHOST devices using MMIO (like PCIe or NTB), using kernel pointers (like platform devices) or using userspace pointers.

---
 drivers/vhost/net.c                      |   3 +-
 drivers/vhost/scsi.c                     |   2 +-
 drivers/vhost/test.c                     |   3 +-
 drivers/vhost/vdpa.c                     |   2 +-
 drivers/vhost/vhost.c                    | 157 ++++++++++++++++++++++-
 drivers/vhost/vsock.c                    |   2 +-
 include/linux/mod_devicetable.h          |   6 +
 {drivers/vhost => include/linux}/vhost.h |  22 +++-
 tools/virtio/virtio_test.c               |   2 +-
 9 files changed, 190 insertions(+), 9 deletions(-)
 rename {drivers/vhost => include/linux}/vhost.h (93%)
drivers/vhost/net.c +2 -1
index 437126219116..3c57c345cbfd 100644
@@ -18,6 +18,7 @@
 #include <linux/slab.h>
 #include <linux/sched/clock.h>
 #include <linux/sched/signal.h>
+#include <linux/vhost.h>
 #include <linux/vmalloc.h>
 
 #include <linux/net.h>
@@ -33,7 +34,7 @@
 #include <net/sock.h>
 #include <net/xdp.h>
 
-#include "vhost.h"
+#include <uapi/linux/vhost.h>
 
 static int experimental_zcopytx = 0;
 module_param(experimental_zcopytx, int, 0444);
drivers/vhost/scsi.c +1 -1
index f5138379659e..06898b7ce7dd 100644
@@ -47,7 +47,7 @@
 #include <linux/llist.h>
 #include <linux/bitmap.h>
 
-#include "vhost.h"
+#include <uapi/linux/vhost.h>
 
 #define VHOST_SCSI_VERSION  "v0.1"
 #define VHOST_SCSI_NAMELEN 256
drivers/vhost/test.c +2 -1
index 6518b48c0633..07508526182f 100644
@@ -14,9 +14,10 @@
 #include <linux/workqueue.h>
 #include <linux/file.h>
 #include <linux/slab.h>
+#include <linux/vhost.h>
+#include <uapi/linux/vhost.h>
 
 #include "test.h"
-#include "vhost.h"
 
 /* Max number of bytes transferred before requeueing the job.
  * Using this limit prevents one virtqueue from starving others. */
drivers/vhost/vdpa.c +1 -1
index 0968361e3b77..61d90100db89 100644
@@ -22,7 +22,7 @@
 #include <linux/vhost.h>
 #include <linux/virtio_net.h>
 
-#include "vhost.h"
+#include <uapi/linux/vhost.h>
 
 enum {
 	VHOST_VDPA_FEATURES =
drivers/vhost/vhost.c +155 -2

183 lines not on this page · raw message

drivers/vhost/vsock.c +1 -1
index 8317ad026e3d..5753048b7405 100644
@@ -17,7 +17,7 @@
 #include <linux/hashtable.h>
 
 #include <net/af_vsock.h>
-#include "vhost.h"
+#include <uapi/linux/vhost.h>
 
 #define VHOST_VSOCK_DEFAULT_HOST_CID	2
 /* Max number of bytes transferred before requeueing the job.
include/linux/mod_devicetable.h +6 -0
index 8d764aab29de..c7df018989e3 100644
@@ -430,6 +430,12 @@ struct virtio_device_id {
 };
 #define VIRTIO_DEV_ANY_ID	0xffffffff
 
+/* VHOST */
+struct vhost_device_id {
+	__u32 device;
+	__u32 vendor;
+};
+
 /*
  * For Hyper-V devices we use the device guid as the id.
  */
include/linux/vhost.h renamed from drivers/vhost/vhost.h +21 -1
similarity index 93%
rename from drivers/vhost/vhost.h
rename to include/linux/vhost.h
index 5d1d00363e79..16c374a8fa12 100644
@@ -3,7 +3,6 @@
 #define _VHOST_H
 
 #include <linux/eventfd.h>
-#include <linux/vhost.h>
 #include <linux/mm.h>
 #include <linux/mutex.h>
 #include <linux/poll.h>
@@ -13,6 +12,7 @@
 #include <linux/virtio_ring.h>
 #include <linux/atomic.h>
 #include <linux/vhost_iotlb.h>
+#include <uapi/linux/vhost.h>
 
 struct vhost_work;
 typedef void (*vhost_work_fn_t)(struct vhost_work *work);
@@ -135,7 +135,20 @@ struct vhost_msg_node {
   struct list_head node;
 };
 
+struct vhost_driver {
+	struct device_driver driver;
+	struct vhost_device_id *id_table;
+	int (*probe)(struct vhost_dev *dev);
+	int (*remove)(struct vhost_dev *dev);
+};
+
+#define to_vhost_driver(drv) (container_of((drv), struct vhost_driver, driver))
+
 struct vhost_dev {
+	struct device dev;
+	struct vhost_driver *driver;
+	struct vhost_device_id id;
+	int index;
 	struct mm_struct *mm;
 	struct mutex mutex;
 	struct vhost_virtqueue **vqs;
@@ -158,6 +171,13 @@ struct vhost_dev {
 			   struct vhost_iotlb_msg *msg);
 };
 
+#define to_vhost_dev(d) container_of((d), struct vhost_dev, dev)
+
+int vhost_register_driver(struct vhost_driver *driver);
+void vhost_unregister_driver(struct vhost_driver *driver);
+int vhost_register_device(struct vhost_dev *vdev);
+void vhost_unregister_device(struct vhost_dev *vdev);
+
 bool vhost_exceeds_weight(struct vhost_virtqueue *vq, int pkts, int total_len);
 void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs,
 		    int nvqs, int iov_limit, int weight, int byte_weight,
tools/virtio/virtio_test.c +1 -1
index b427def67e7e..b13434d6c976 100644
@@ -13,9 +13,9 @@
 #include <fcntl.h>
 #include <stdbool.h>
 #include <linux/virtio_types.h>
-#include <linux/vhost.h>
 #include <linux/virtio.h>
 #include <linux/virtio_ring.h>
+#include <uapi/linux/vhost.h>
 #include "../../drivers/vhost/test.h"
 
 /* Unused */
Signature · 1 line
2.17.1
#3 lore raw

RFC PATCH 3/22 vhost: Add ops for the VHOST driver to configure VHOST device

In reply to Kishon Vijay Abraham I

Add "vhost_config_ops" in *struct vhost_driver* for the VHOST driver to configure VHOST device and add facility for VHOST device to notify VHOST driver (whenever VIRTIO sets a new status or finalize features). This is in preparation to use the same vhost_driver across different VHOST devices (like PCIe or NTB or platform device).

---
 drivers/vhost/vhost.c | 185 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/vhost.h |  57 +++++++++++++
 2 files changed, 242 insertions(+)
drivers/vhost/vhost.c +185 -0
index fa2bc6e68be2..f959abb0b1bb 100644
@@ -2558,6 +2558,190 @@ struct vhost_msg_node *vhost_dequeue_msg(struct vhost_dev *dev,
 }
 EXPORT_SYMBOL_GPL(vhost_dequeue_msg);
 
+/**
+ * vhost_create_vqs() - Invoke vhost_config_ops to create virtqueue
+ * @vdev: Vhost device that provides create_vqs() callback to create virtqueue
+ * @nvqs: Number of vhost virtqueues to be created
+ * @num_bufs: The number of buffers that should be supported by the vhost
+ *   virtqueue (number of descriptors in the vhost virtqueue)
+ * @vqs: Pointers to all the created vhost virtqueues
+ * @callback: Callback function associated with the virtqueue
+ * @names: Names associated with each virtqueue
+ *
+ * Wrapper that invokes vhost_config_ops to create virtqueue.
+ */
+int vhost_create_vqs(struct vhost_dev *vdev, unsigned int nvqs,
+		     unsigned int num_bufs, struct vhost_virtqueue *vqs[],
+		     vhost_vq_callback_t *callbacks[],
+		     const char * const names[])
+{
+	int ret;
+
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->create_vqs)
+		return -EINVAL;
+
+	mutex_lock(&vdev->mutex);
+	ret = vdev->ops->create_vqs(vdev, nvqs, num_bufs, vqs, callbacks,
+				    names);
+	mutex_unlock(&vdev->mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vhost_create_vqs);
+
+/* vhost_del_vqs - Invoke vhost_config_ops to delete the created virtqueues
+ * @vdev: Vhost device that provides del_vqs() callback to delete virtqueue
+ *
+ * Wrapper that invokes vhost_config_ops to delete all the virtqueues
+ * associated with the vhost device.
+ */
+void vhost_del_vqs(struct vhost_dev *vdev)
+{
+	if (IS_ERR_OR_NULL(vdev))
+		return;
+
+	if (!vdev->ops && !vdev->ops->del_vqs)
+		return;
+
+	mutex_lock(&vdev->mutex);
+	vdev->ops->del_vqs(vdev);
+	mutex_unlock(&vdev->mutex);
+}
+EXPORT_SYMBOL_GPL(vhost_del_vqs);
+
+/* vhost_write - Invoke vhost_config_ops to write data to buffer provided
+ *   by remote virtio driver
+ * @vdev: Vhost device that provides write() callback to write data
+ * @dst: Buffer address in the remote device provided by the remote virtio
+ *   driver
+ * @src: Buffer address in the local device provided by the vhost client driver
+ * @len: Length of the data to be copied from @src to @dst
+ *
+ * Wrapper that invokes vhost_config_ops to write data to buffer provided by
+ * remote virtio driver from buffer provided by vhost client driver.
+ */
+int vhost_write(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len)
+{
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->write)
+		return -EINVAL;
+
+	return vdev->ops->write(vdev, vhost_dst, src, len);
+}
+EXPORT_SYMBOL_GPL(vhost_write);
+
+/* vhost_read - Invoke vhost_config_ops to read data from buffers provided by
+ *   remote virtio driver
+ * @vdev: Vhost device that provides read() callback to read data
+ * @dst: Buffer address in the local device provided by the vhost client driver
+ * @src: Buffer address in the remote device provided by the remote virtio
+ *   driver
+ * @len: Length of the data to be copied from @src to @dst
+ *
+ * Wrapper that invokes vhost_config_ops to read data from buffers provided by
+ * remote virtio driver to the address provided by vhost client driver.
+ */
+int vhost_read(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len)
+{
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->read)
+		return -EINVAL;
+
+	return vdev->ops->read(vdev, dst, vhost_src, len);
+}
+EXPORT_SYMBOL_GPL(vhost_read);
+
+/* vhost_set_status - Invoke vhost_config_ops to set vhost device status
+ * @vdev: Vhost device that provides set_status() callback to set device status
+ * @status: Vhost device status configured by vhost client driver
+ *
+ * Wrapper that invokes vhost_config_ops to set vhost device status.
+ */
+int vhost_set_status(struct vhost_dev *vdev, u8 status)
+{
+	int ret;
+
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->set_status)
+		return -EINVAL;
+
+	mutex_lock(&vdev->mutex);
+	ret = vdev->ops->set_status(vdev, status);
+	mutex_unlock(&vdev->mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vhost_set_status);
+
+/* vhost_get_status - Invoke vhost_config_ops to get vhost device status
+ * @vdev: Vhost device that provides get_status() callback to get device status
+ *
+ * Wrapper that invokes vhost_config_ops to get vhost device status.
+ */
+u8 vhost_get_status(struct vhost_dev *vdev)
+{
+	u8 status;
+
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->get_status)
+		return -EINVAL;
+
+	mutex_lock(&vdev->mutex);
+	status = vdev->ops->get_status(vdev);
+	mutex_unlock(&vdev->mutex);
+
+	return status;
+}
+EXPORT_SYMBOL_GPL(vhost_get_status);
+
+/* vhost_set_features - Invoke vhost_config_ops to set vhost device features
+ * @vdev: Vhost device that provides set_features() callback to set device
+ *   features
+ *
+ * Wrapper that invokes vhost_config_ops to set device features.
+ */
+int vhost_set_features(struct vhost_dev *vdev, u64 device_features)
+{
+	int ret;
+
+	if (IS_ERR_OR_NULL(vdev))
+		return -EINVAL;
+
+	if (!vdev->ops && !vdev->ops->set_features)
+		return -EINVAL;
+
+	mutex_lock(&vdev->mutex);
+	ret = vdev->ops->set_features(vdev, device_features);
+	mutex_unlock(&vdev->mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vhost_set_features);
+
+/* vhost_register_notifier - Register notifier to receive notification from
+ *   vhost device
+ * @vdev: Vhost device from which notification has to be received.
+ * @nb: Notifier block holding the callback function
+ *
+ * Invoked by vhost client to receive notification from vhost device.
+ */
+int vhost_register_notifier(struct vhost_dev *vdev, struct notifier_block *nb)
+{
+	return blocking_notifier_chain_register(&vdev->notifier, nb);
+}
+EXPORT_SYMBOL_GPL(vhost_register_notifier);
+
 static inline int vhost_id_match(const struct vhost_dev *vdev,
 				 const struct vhost_device_id *id)
 {
@@ -2669,6 +2853,7 @@ int vhost_register_device(struct vhost_dev *vdev)
 	device_initialize(dev);
 
 	dev_set_name(dev, "vhost%u", ret);
+	BLOCKING_INIT_NOTIFIER_HEAD(&vdev->notifier);
 
 	ret = device_add(dev);
 	if (ret) {
include/linux/vhost.h +57 -0

80 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

RFC PATCH 4/22 vringh: Add helpers to access vring in MMIO

In reply to Kishon Vijay Abraham I

Add helpers to access vring in memory mapped IO. This would require using IO accessors to access vring and the addresses populated by virtio driver (in the descriptor) should be directly given to the vhost client driver. Even if the vhost runs in 32 bit system, it can access 64 bit address provided by the virtio if the vhost device supports translation.

This is in preparation for adding VHOST devices (PCIe Endpoint or Host in NTB) to access vrings created by VIRTIO devices (PCIe RC or Host in NTB) over memory mapped IO.

---
 drivers/vhost/vringh.c | 332 +++++++++++++++++++++++++++++++++++++++++
 include/linux/vringh.h |  46 ++++++
 2 files changed, 378 insertions(+)
drivers/vhost/vringh.c +332 -0
index ba8e0d6cfd97..b3f1910b99ec 100644
@@ -5,6 +5,7 @@
  * Since these may be in userspace, we use (inline) accessors.
  */
 #include <linux/compiler.h>
+#include <linux/io.h>
 #include <linux/module.h>
 #include <linux/vringh.h>
 #include <linux/virtio_ring.h>
@@ -188,6 +189,32 @@ static int move_to_indirect(const struct vringh *vrh,
 	return 0;
 }
 
+static int resize_mmiovec(struct vringh_mmiov *iov, gfp_t gfp)
+{
+	unsigned int flag, new_num = (iov->max_num & ~VRINGH_IOV_ALLOCATED) * 2;
+	struct mmiovec *new;
+
+	if (new_num < 8)
+		new_num = 8;
+
+	flag = (iov->max_num & VRINGH_IOV_ALLOCATED);
+	if (flag) {
+		new = krealloc(iov->iov, new_num * sizeof(struct iovec), gfp);
+	} else {
+		new = kmalloc_array(new_num, sizeof(struct iovec), gfp);
+		if (new) {
+			memcpy(new, iov->iov,
+			       iov->max_num * sizeof(struct iovec));
+			flag = VRINGH_IOV_ALLOCATED;
+		}
+	}
+	if (!new)
+		return -ENOMEM;
+	iov->iov = new;
+	iov->max_num = (new_num | flag);
+	return 0;
+}
+
 static int resize_iovec(struct vringh_kiov *iov, gfp_t gfp)
 {
 	struct kvec *new;
@@ -261,6 +288,142 @@ static int slow_copy(struct vringh *vrh, void *dst, const void *src,
 	return 0;
 }
 
+static inline int
+__vringh_mmiov(struct vringh *vrh, u16 i, struct vringh_mmiov *riov,
+	       struct vringh_mmiov *wiov,
+	       bool (*rcheck)(struct vringh *vrh, u64 addr, size_t *len,
+			      struct vringh_range *range,
+			      bool (*getrange)(struct vringh *, u64,
+					       struct vringh_range *)),
+	       bool (*getrange)(struct vringh *, u64, struct vringh_range *),
+	       gfp_t gfp,
+	       int (*copy)(const struct vringh *vrh,
+			   void *dst, const void *src, size_t len))
+{
+	int err, count = 0, up_next, desc_max;
+	struct vring_desc desc, *descs;
+	struct vringh_range range = { -1ULL, 0 }, slowrange;
+	bool slow = false;
+
+	/* We start traversing vring's descriptor table. */
+	descs = vrh->vring.desc;
+	desc_max = vrh->vring.num;
+	up_next = -1;
+
+	if (riov) {
+		riov->i = 0;
+		riov->used = 0;
+	} else if (wiov) {
+		wiov->i = 0;
+		wiov->used = 0;
+	} else {
+		/* You must want something! */
+		WARN_ON(1);
+	}
+
+	for (;;) {
+		u64 addr;
+		struct vringh_mmiov *iov;
+		size_t len;
+
+		if (unlikely(slow))
+			err = slow_copy(vrh, &desc, &descs[i], rcheck, getrange,
+					&slowrange, copy);
+		else
+			err = copy(vrh, &desc, &descs[i], sizeof(desc));
+		if (unlikely(err))
+			goto fail;
+
+		if (unlikely(desc.flags &
+			     cpu_to_vringh16(vrh, VRING_DESC_F_INDIRECT))) {
+			/* VRING_DESC_F_INDIRECT is not supported */
+			err = -EINVAL;
+			goto fail;
+		}
+
+		if (count++ == vrh->vring.num) {
+			vringh_bad("Descriptor loop in %p", descs);
+			err = -ELOOP;
+			goto fail;
+		}
+
+		if (desc.flags & cpu_to_vringh16(vrh, VRING_DESC_F_WRITE)) {
+			iov = wiov;
+		} else {
+			iov = riov;
+			if (unlikely(wiov && wiov->i)) {
+				vringh_bad("Readable desc %p after writable",
+					   &descs[i]);
+				err = -EINVAL;
+				goto fail;
+			}
+		}
+
+		if (!iov) {
+			vringh_bad("Unexpected %s desc",
+				   !wiov ? "writable" : "readable");
+			err = -EPROTO;
+			goto fail;
+		}
+
+again:
+		/* Make sure it's OK, and get offset. */
+		len = vringh32_to_cpu(vrh, desc.len);
+		if (!rcheck(vrh, vringh64_to_cpu(vrh, desc.addr), &len, &range,
+			    getrange)) {
+			err = -EINVAL;
+			goto fail;
+		}
+		addr = vringh64_to_cpu(vrh, desc.addr) + range.offset;
+
+		if (unlikely(iov->used == (iov->max_num & ~VRINGH_IOV_ALLOCATED))) {
+			err = resize_mmiovec(iov, gfp);
+			if (err)
+				goto fail;
+		}
+
+		iov->iov[iov->used].iov_base = addr;
+		iov->iov[iov->used].iov_len = len;
+		iov->used++;
+
+		if (unlikely(len != vringh32_to_cpu(vrh, desc.len))) {
+			desc.len =
+				cpu_to_vringh32(vrh,
+						vringh32_to_cpu(vrh, desc.len)
+						- len);
+			desc.addr =
+				cpu_to_vringh64(vrh,
+						vringh64_to_cpu(vrh, desc.addr)
+						+ len);
+			goto again;
+		}
+
+		if (desc.flags & cpu_to_vringh16(vrh, VRING_DESC_F_NEXT)) {
+			i = vringh16_to_cpu(vrh, desc.next);
+		} else {
+			/* Just in case we need to finish traversing above. */
+			if (unlikely(up_next > 0)) {
+				i = return_from_indirect(vrh, &up_next,
+							 &descs, &desc_max);
+				slow = false;
+			} else {
+				break;
+			}
+		}
+
+		if (i >= desc_max) {
+			vringh_bad("Chained index %u > %u", i, desc_max);
+			err = -EINVAL;
+			goto fail;
+		}
+	}
+
+	return 0;
+
+fail:
+	return err;
+}
+
 static inline int
 __vringh_iov(struct vringh *vrh, u16 i,
 	     struct vringh_kiov *riov,
@@ -833,6 +996,175 @@ int vringh_need_notify_user(struct vringh *vrh)
 }
 EXPORT_SYMBOL(vringh_need_notify_user);
 
+/* MMIO access helpers */
+static inline int getu16_mmio(const struct vringh *vrh,
+			      u16 *val, const __virtio16 *p)
+{
+	*val = vringh16_to_cpu(vrh, readw(p));
+	return 0;
+}
+
+static inline int putu16_mmio(const struct vringh *vrh, __virtio16 *p, u16 val)
+{
+	writew(cpu_to_vringh16(vrh, val), p);
+	return 0;
+}
+
+static inline int copydesc_mmio(const struct vringh *vrh,
+				void *dst, const void *src, size_t len)
+{
+	memcpy_fromio(dst, src, len);
+	return 0;
+}
+
+static inline int putused_mmio(const struct vringh *vrh,
+			       struct vring_used_elem *dst,
+			       const struct vring_used_elem *src,
+			       unsigned int num)
+{
+	memcpy_toio(dst, src, num * sizeof(*dst));
+	return 0;
+}
+
+/**
+ * vringh_init_mmio - initialize a vringh for a MMIO vring.
+ * @vrh: the vringh to initialize.
+ * @features: the feature bits for this ring.
+ * @num: the number of elements.
+ * @weak_barriers: true if we only need memory barriers, not I/O.
+ * @desc: the userpace descriptor pointer.
+ * @avail: the userpace avail pointer.
+ * @used: the userpace used pointer.
+ *
+ * Returns an error if num is invalid.
+ */
+int vringh_init_mmio(struct vringh *vrh, u64 features,
+		     unsigned int num, bool weak_barriers,
+		     struct vring_desc *desc,
+		     struct vring_avail *avail,
+		     struct vring_used *used)
+{
+	/* Sane power of 2 please! */
+	if (!num || num > 0xffff || (num & (num - 1))) {
+		vringh_bad("Bad ring size %u", num);
+		return -EINVAL;
+	}
+
+	vrh->little_endian = (features & (1ULL << VIRTIO_F_VERSION_1));
+	vrh->event_indices = (features & (1 << VIRTIO_RING_F_EVENT_IDX));
+	vrh->weak_barriers = weak_barriers;
+	vrh->completed = 0;
+	vrh->last_avail_idx = 0;
+	vrh->last_used_idx = 0;
+	vrh->vring.num = num;
+	vrh->vring.desc = desc;
+	vrh->vring.avail = avail;
+	vrh->vring.used = used;
+	return 0;
+}
+EXPORT_SYMBOL(vringh_init_mmio);
+
+/**
+ * vringh_getdesc_mmio - get next available descriptor from MMIO ring.
+ * @vrh: the MMIO vring.
+ * @riov: where to put the readable descriptors (or NULL)
+ * @wiov: where to put the writable descriptors (or NULL)
+ * @head: head index we received, for passing to vringh_complete_mmio().
+ * @gfp: flags for allocating larger riov/wiov.
+ *
+ * Returns 0 if there was no descriptor, 1 if there was, or -errno.
+ *
+ * Note that on error return, you can tell the difference between an
+ * invalid ring and a single invalid descriptor: in the former case,
+ * *head will be vrh->vring.num.  You may be able to ignore an invalid
+ * descriptor, but there's not much you can do with an invalid ring.
+ *
+ * Note that you may need to clean up riov and wiov, even on error!
+ */
+int vringh_getdesc_mmio(struct vringh *vrh,
+			struct vringh_mmiov *riov,
+			struct vringh_mmiov *wiov,
+			u16 *head,
+			gfp_t gfp)
+{
+	int err;
+
+	err = __vringh_get_head(vrh, getu16_mmio, &vrh->last_avail_idx);
+	if (err < 0)
+		return err;
+
+	/* Empty... */
+	if (err == vrh->vring.num)
+		return 0;
+
+	*head = err;
+	err = __vringh_mmiov(vrh, *head, riov, wiov, no_range_check, NULL,
+			     gfp, copydesc_mmio);
+	if (err)
+		return err;
+
+	return 1;
+}
+EXPORT_SYMBOL(vringh_getdesc_mmio);
+
+/**
+ * vringh_complete_mmio - we've finished with descriptor, publish it.
+ * @vrh: the vring.
+ * @head: the head as filled in by vringh_getdesc_mmio.
+ * @len: the length of data we have written.
+ *
+ * You should check vringh_need_notify_mmio() after one or more calls
+ * to this function.
+ */
+int vringh_complete_mmio(struct vringh *vrh, u16 head, u32 len)
+{
+	struct vring_used_elem used;
+
+	used.id = cpu_to_vringh32(vrh, head);
+	used.len = cpu_to_vringh32(vrh, len);
+
+	return __vringh_complete(vrh, &used, 1, putu16_mmio, putused_mmio);
+}
+EXPORT_SYMBOL(vringh_complete_mmio);
+
+/**
+ * vringh_notify_enable_mmio - we want to know if something changes.
+ * @vrh: the vring.
+ *
+ * This always enables notifications, but returns false if there are
+ * now more buffers available in the vring.
+ */
+bool vringh_notify_enable_mmio(struct vringh *vrh)
+{
+	return __vringh_notify_enable(vrh, getu16_mmio, putu16_mmio);
+}
+EXPORT_SYMBOL(vringh_notify_enable_mmio);
+
+/**
+ * vringh_notify_disable_mmio - don't tell us if something changes.
+ * @vrh: the vring.
+ *
+ * This is our normal running state: we disable and then only enable when
+ * we're going to sleep.
+ */
+void vringh_notify_disable_mmio(struct vringh *vrh)
+{
+	__vringh_notify_disable(vrh, putu16_mmio);
+}
+EXPORT_SYMBOL(vringh_notify_disable_mmio);
+
+/**
+ * vringh_need_notify_mmio - must we tell the other side about used buffers?
+ * @vrh: the vring we've called vringh_complete_mmio() on.
+ *
+ * Returns -errno or 0 if we don't need to tell the other side, 1 if we do.
+ */
+int vringh_need_notify_mmio(struct vringh *vrh)
+{
+	return __vringh_need_notify(vrh, getu16_mmio);
+}
+EXPORT_SYMBOL(vringh_need_notify_mmio);
+
 /* Kernelspace access helpers. */
 static inline int getu16_kern(const struct vringh *vrh,
 			      u16 *val, const __virtio16 *p)
include/linux/vringh.h +46 -0

58 lines not on this page · raw message

Signature · 1 line
2.17.1
#5 lore raw

RFC PATCH 5/22 vhost: Add MMIO helpers for operations on vhost virtqueue

In reply to Kishon Vijay Abraham I

Add helpers for VHOST drivers to read descriptor data from vhost_virtqueue for IN transfers or write descriptor data to vhost_virtqueue for OUT transfers respectively. Also add helpers to enable callback, disable callback and notify remote virtio for events on virtqueue.

This adds helpers only for virtqueue in MMIO (helpers for virtqueue in kernel space and user space can be added later).

---
 drivers/vhost/Kconfig |   1 +
 drivers/vhost/vhost.c | 292 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/vhost.h |  22 ++++
 3 files changed, 315 insertions(+)
drivers/vhost/Kconfig +1 -0
index c4f273793595..77e195a38469 100644
@@ -24,6 +24,7 @@ config VHOST_DPN
 
 config VHOST
 	tristate
+	select VHOST_RING
 	select VHOST_IOTLB
 	help
 	  This option is selected by any driver which needs to access
drivers/vhost/vhost.c +292 -0

298 lines not on this page · raw message

include/linux/vhost.h +22 -0
index b22a19c66109..8efb9829c1b1 100644
@@ -10,6 +10,7 @@
 #include <linux/uio.h>
 #include <linux/virtio_config.h>
 #include <linux/virtio_ring.h>
+#include <linux/vringh.h>
 #include <linux/atomic.h>
 #include <linux/vhost_iotlb.h>
 #include <uapi/linux/vhost.h>
@@ -60,9 +61,20 @@ enum vhost_uaddr_type {
 	VHOST_NUM_ADDRS = 3,
 };
 
+enum vhost_type {
+	VHOST_TYPE_UNKNOWN,
+	VHOST_TYPE_USER,
+	VHOST_TYPE_KERN,
+	VHOST_TYPE_MMIO,
+};
+
 /* The virtqueue structure describes a queue attached to a device. */
 struct vhost_virtqueue {
 	struct vhost_dev *dev;
+	enum vhost_type type;
+	struct vringh vringh;
+	void (*callback)(struct vhost_virtqueue *vq);
+	void (*notify)(struct vhost_virtqueue *vq);
 
 	/* The actual ring of buffers. */
 	struct mutex mutex;
@@ -235,6 +247,16 @@ u8 vhost_get_status(struct vhost_dev *vdev);
 
 int vhost_register_notifier(struct vhost_dev *vdev, struct notifier_block *nb);
 
+u64 vhost_virtqueue_get_outbuf(struct vhost_virtqueue *vq, u16 *head, int *len);
+u64 vhost_virtqueue_get_inbuf(struct vhost_virtqueue *vq, u16 *head, int *len);
+void vhost_virtqueue_put_buf(struct vhost_virtqueue *vq, u16 head, int len);
+
+void vhost_virtqueue_disable_cb(struct vhost_virtqueue *vq);
+bool vhost_virtqueue_enable_cb(struct vhost_virtqueue *vq);
+void vhost_virtqueue_notify(struct vhost_virtqueue *vq);
+void vhost_virtqueue_kick(struct vhost_virtqueue *vq);
+void vhost_virtqueue_callback(struct vhost_virtqueue *vq);
+
 bool vhost_exceeds_weight(struct vhost_virtqueue *vq, int pkts, int total_len);
 void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs,
 		    int nvqs, int iov_limit, int weight, int byte_weight,
Signature · 1 line
2.17.1
#6 lore raw

RFC PATCH 6/22 vhost: Introduce configfs entry for configuring VHOST

In reply to Kishon Vijay Abraham I

Create a configfs entry for each entry populated in
"struct vhost_device_id" in the VHOST driver and create a configfs entry for each VHOST device. This is used to link VHOST driver to VHOST device (by assigning deviceID and vendorID to the VHOST device) and register VHOST device, thereby letting VHOST client driver to be selected in the userspace.

---
 drivers/vhost/Makefile    |   2 +-
 drivers/vhost/vhost.c     |  63 +++++++
 drivers/vhost/vhost_cfs.c | 354 ++++++++++++++++++++++++++++++++++++++
 include/linux/vhost.h     |  11 ++
 4 files changed, 429 insertions(+), 1 deletion(-)
 create mode 100644 drivers/vhost/vhost_cfs.c
drivers/vhost/Makefile +1 -1
index f3e1897cce85..6520c820c896 100644
@@ -13,7 +13,7 @@ obj-$(CONFIG_VHOST_RING) += vringh.o
 obj-$(CONFIG_VHOST_VDPA) += vhost_vdpa.o
 vhost_vdpa-y := vdpa.o
 
-obj-$(CONFIG_VHOST)	+= vhost.o
+obj-$(CONFIG_VHOST)	+= vhost.o vhost_cfs.o
 
 obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o
 vhost_iotlb-y := iotlb.o
drivers/vhost/vhost.c +63 -0

81 lines not on this page · raw message

drivers/vhost/vhost_cfs.c new file +354 -0

354 lines not on this page · raw message

include/linux/vhost.h +11 -0
index 8efb9829c1b1..be9341ffd266 100644
@@ -2,6 +2,7 @@
 #ifndef _VHOST_H
 #define _VHOST_H
 
+#include <linux/configfs.h>
 #include <linux/eventfd.h>
 #include <linux/mm.h>
 #include <linux/mutex.h>
@@ -181,6 +182,7 @@ struct vhost_config_ops {
 struct vhost_driver {
 	struct device_driver driver;
 	struct vhost_device_id *id_table;
+	struct config_group *group;
 	int (*probe)(struct vhost_dev *dev);
 	int (*remove)(struct vhost_dev *dev);
 };
@@ -233,6 +235,15 @@ void vhost_unregister_driver(struct vhost_driver *driver);
 int vhost_register_device(struct vhost_dev *vdev);
 void vhost_unregister_device(struct vhost_dev *vdev);
 
+struct config_group *vhost_cfs_add_driver_group(const char *name);
+void vhost_cfs_remove_driver_group(struct config_group *group);
+struct config_group *
+vhost_cfs_add_driver_item(struct config_group *driver_group, u32 vendor,
+			  u32 device);
+void vhost_cfs_remove_driver_item(struct config_group *group);
+struct config_group *vhost_cfs_add_device_item(struct vhost_dev *vdev);
+void vhost_cfs_remove_device_item(struct config_group *group);
+
 int vhost_create_vqs(struct vhost_dev *vdev, unsigned int nvqs,
 		     unsigned int num_bufs, struct vhost_virtqueue *vqs[],
 		     vhost_vq_callback_t *callbacks[],
Signature · 1 line
2.17.1
#7 lore raw

RFC PATCH 7/22 virtio_pci: Use request_threaded_irq() instead of request_irq()

In reply to Kishon Vijay Abraham I

Some of the virtio drivers (like virtio_rpmsg_bus.c) use sleeping functions like mutex_*() in the virtqueue callback. Use request_threaded_irq() instead of request_irq() in order for the virtqueue callbacks to be executed in thread context instead of interrupt context.

---
 drivers/virtio/virtio_pci_common.c | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_common.c +13 -12
index 222d630c41fc..60998b4f1f30 100644
@@ -140,9 +140,9 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors,
 	v = vp_dev->msix_used_vectors;
 	snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names,
 		 "%s-config", name);
-	err = request_irq(pci_irq_vector(vp_dev->pci_dev, v),
-			  vp_config_changed, 0, vp_dev->msix_names[v],
-			  vp_dev);
+	err = request_threaded_irq(pci_irq_vector(vp_dev->pci_dev, v), 0,
+				   vp_config_changed, 0, vp_dev->msix_names[v],
+				   vp_dev);
 	if (err)
 		goto error;
 	++vp_dev->msix_used_vectors;
@@ -159,9 +159,9 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors,
 		v = vp_dev->msix_used_vectors;
 		snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names,
 			 "%s-virtqueues", name);
-		err = request_irq(pci_irq_vector(vp_dev->pci_dev, v),
-				  vp_vring_interrupt, 0, vp_dev->msix_names[v],
-				  vp_dev);
+		err = request_threaded_irq(pci_irq_vector(vp_dev->pci_dev, v),
+					   0, vp_vring_interrupt, 0,
+					   vp_dev->msix_names[v], vp_dev);
 		if (err)
 			goto error;
 		++vp_dev->msix_used_vectors;
@@ -336,10 +336,11 @@ static int vp_find_vqs_msix(struct virtio_device *vdev, unsigned nvqs,
 			 sizeof *vp_dev->msix_names,
 			 "%s-%s",
 			 dev_name(&vp_dev->vdev.dev), names[i]);
-		err = request_irq(pci_irq_vector(vp_dev->pci_dev, msix_vec),
-				  vring_interrupt, 0,
-				  vp_dev->msix_names[msix_vec],
-				  vqs[i]);
+		err = request_threaded_irq(pci_irq_vector(vp_dev->pci_dev,
+							  msix_vec),
+					   0, vring_interrupt, 0,
+					   vp_dev->msix_names[msix_vec],
+					   vqs[i]);
 		if (err)
 			goto error_find;
 	}
@@ -361,8 +362,8 @@ static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned nvqs,
 	if (!vp_dev->vqs)
 		return -ENOMEM;
 
-	err = request_irq(vp_dev->pci_dev->irq, vp_interrupt, IRQF_SHARED,
-			dev_name(&vdev->dev), vp_dev);
+	err = request_threaded_irq(vp_dev->pci_dev->irq, 0, vp_interrupt,
+				   IRQF_SHARED, dev_name(&vdev->dev), vp_dev);
 	if (err)
 		goto out_del_vqs;
 
Signature · 1 line
2.17.1
#8 lore raw

RFC PATCH 8/22 rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages

In reply to Kishon Vijay Abraham I

Since rpmsg_recv_done() reads messages in a while loop, disable callbacks until the while loop exits. This helps to get rid of the annoying "uhm, incoming signal, but no used buffer ?" message.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/rpmsg/virtio_rpmsg_bus.c +14 -0
index 376ebbf880d6..2d0d42084ac0 100644
@@ -777,6 +777,7 @@ static void rpmsg_recv_done(struct virtqueue *rvq)
 		return;
 	}
 
+	virtqueue_disable_cb(rvq);
 	while (msg) {
 		err = rpmsg_recv_single(vrp, dev, msg, len);
 		if (err)
@@ -786,6 +787,19 @@ static void rpmsg_recv_done(struct virtqueue *rvq)
 
 		msg = virtqueue_get_buf(rvq, &len);
 	}
+	virtqueue_enable_cb(rvq);
+
+	/*
+	 * Try to read message one more time in case a new message is submitted
+	 * after virtqueue_get_buf() inside the while loop but before enabling
+	 * callbacks
+	 */
+	msg = virtqueue_get_buf(rvq, &len);
+	if (msg) {
+		err = rpmsg_recv_single(vrp, dev, msg, len);
+		if (!err)
+			msgs_received++;
+	}
 
 	dev_dbg(dev, "Received %u messages\n", msgs_received);
 
Signature · 1 line
2.17.1
#9 lore raw

RFC PATCH 9/22 rpmsg: Introduce configfs entry for configuring rpmsg

In reply to Kishon Vijay Abraham I

Create a configfs entry for each "struct rpmsg_device_id" populated in rpmsg client driver and create a configfs entry for each rpmsg device. This will be used to bind a rpmsg client driver to a rpmsg device in order to create a new rpmsg channel.

This is used for creating channel for VHOST based rpmsg bus (channels are created in VIRTIO based bus during namespace announcement).

---
 drivers/rpmsg/Makefile         |   2 +-
 drivers/rpmsg/rpmsg_cfs.c      | 394 +++++++++++++++++++++++++++++++++
 drivers/rpmsg/rpmsg_core.c     |   7 +
 drivers/rpmsg/rpmsg_internal.h |  16 ++
 include/linux/rpmsg.h          |   5 +
 5 files changed, 423 insertions(+), 1 deletion(-)
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
drivers/rpmsg/Makefile +1 -1
index ae92a7fb08f6..047acfda518a 100644
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_RPMSG)		+= rpmsg_core.o
+obj-$(CONFIG_RPMSG)		+= rpmsg_core.o rpmsg_cfs.o
 obj-$(CONFIG_RPMSG_CHAR)	+= rpmsg_char.o
 obj-$(CONFIG_RPMSG_MTK_SCP)	+= mtk_rpmsg.o
 obj-$(CONFIG_RPMSG_QCOM_GLINK_RPM) += qcom_glink_rpm.o
drivers/rpmsg/rpmsg_cfs.c new file +394 -0

394 lines not on this page · raw message

drivers/rpmsg/rpmsg_core.c +7 -0
index e330ec4dfc33..68569fec03e2 100644
@@ -563,8 +563,15 @@ EXPORT_SYMBOL(rpmsg_unregister_device);
  */
 int __register_rpmsg_driver(struct rpmsg_driver *rpdrv, struct module *owner)
 {
+	const struct rpmsg_device_id *ids = rpdrv->id_table;
 	rpdrv->drv.bus = &rpmsg_bus;
 	rpdrv->drv.owner = owner;
+
+	while (ids && ids->name[0]) {
+		rpmsg_cfs_add_channel_group(ids->name);
+		ids++;
+	}
+
 	return driver_register(&rpdrv->drv);
 }
 EXPORT_SYMBOL(__register_rpmsg_driver);
drivers/rpmsg/rpmsg_internal.h +16 -0
index 3fc83cd50e98..39b3a5caf242 100644
@@ -68,6 +68,18 @@ struct rpmsg_endpoint_ops {
 			     poll_table *wait);
 };
 
+/**
+ * struct rpmsg_virtproc_ops - indirection table for rpmsg_virtproc operations
+ * @create_channel: Create a new rpdev channel
+ * @delete_channel: Delete the rpdev channel
+ * @owner: Owner of the module holding the ops
+ */
+struct rpmsg_virtproc_ops {
+	struct device *(*create_channel)(struct device *dev, const char *name);
+	void (*delete_channel)(struct device *dev);
+	struct module *owner;
+};
+
 int rpmsg_register_device(struct rpmsg_device *rpdev);
 int rpmsg_unregister_device(struct device *parent,
 			    struct rpmsg_channel_info *chinfo);
@@ -75,6 +87,10 @@ int rpmsg_unregister_device(struct device *parent,
 struct device *rpmsg_find_device(struct device *parent,
 				 struct rpmsg_channel_info *chinfo);
 
+struct config_group *
+rpmsg_cfs_add_virtproc_group(struct device *dev,
+			     const struct rpmsg_virtproc_ops *ops);
+
 /**
  * rpmsg_chrdev_register_device() - register chrdev device based on rpdev
  * @rpdev:	prepared rpdev to be used for creating endpoints
include/linux/rpmsg.h +5 -0
index 9fe156d1c018..b9d9283b46ac 100644
@@ -135,6 +135,7 @@ int rpmsg_trysend_offchannel(struct rpmsg_endpoint *ept, u32 src, u32 dst,
 __poll_t rpmsg_poll(struct rpmsg_endpoint *ept, struct file *filp,
 			poll_table *wait);
 
+struct config_group *rpmsg_cfs_add_channel_group(const char *name);
 #else
 
 static inline int register_rpmsg_device(struct rpmsg_device *dev)
@@ -242,6 +243,10 @@ static inline __poll_t rpmsg_poll(struct rpmsg_endpoint *ept,
 	return 0;
 }
 
+static inline struct config_group *rpmsg_cfs_add_channel_group(const char *name)
+{
+	return NULL;
+}
 #endif /* IS_ENABLED(CONFIG_RPMSG) */
 
 /* use a macro to avoid include chaining to get THIS_MODULE */
Signature · 1 line
2.17.1
#10 lore raw

RFC PATCH 10/22 rpmsg: virtio_rpmsg_bus: Add Address Service Notification support

In reply to Kishon Vijay Abraham I

Add support to send address service notification message to the backend rpmsg device. This informs the backend rpmsg device about the address allocated to the channel that is created in response to the name service announce message from backend rpmsg device. This is in preparation to add backend rpmsg device using VHOST framework in Linux.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 92 +++++++++++++++++++++++++++++---
 1 file changed, 85 insertions(+), 7 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +85 -7
index 2d0d42084ac0..19d930c9fc2c 100644
@@ -71,6 +71,7 @@ struct virtproc_info {
 
 /* The feature bitmap for virtio rpmsg */
 #define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_AS	1 /* RP supports address service notifications */
 
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
@@ -110,6 +111,26 @@ struct rpmsg_ns_msg {
 	u32 flags;
 } __packed;
 
+/**
+ * struct rpmsg_as_msg - dynamic address service announcement message
+ * @name: name of the created channel
+ * @dst: destination address to be used by the backend rpdev
+ * @src: source address of the backend rpdev (the one that sent name service
+ * announcement message)
+ * @flags: indicates whether service is created or destroyed
+ *
+ * This message is sent (by virtio_rpmsg_bus) when a new channel is created
+ * in response to name service announcement message by backend rpdev to create
+ * a new channel. This sends the allocated source address for the channel
+ * (destination address for the backend rpdev) to the backend rpdev.
+ */
+struct rpmsg_as_msg {
+	char name[RPMSG_NAME_SIZE];
+	u32 dst;
+	u32 src;
+	u32 flags;
+} __packed;
+
 /**
  * enum rpmsg_ns_flags - dynamic name service announcement flags
  *
@@ -119,6 +140,19 @@ struct rpmsg_ns_msg {
 enum rpmsg_ns_flags {
 	RPMSG_NS_CREATE		= 0,
 	RPMSG_NS_DESTROY	= 1,
+	RPMSG_AS_ANNOUNCE	= 2,
+};
+
+/**
+ * enum rpmsg_as_flags - dynamic address service announcement flags
+ *
+ * @RPMSG_AS_ASSIGN: address has been assigned to the newly created channel
+ * @RPMSG_AS_FREE: assigned address is freed from the channel and no longer can
+ * be used
+ */
+enum rpmsg_as_flags {
+	RPMSG_AS_ASSIGN		= 1,
+	RPMSG_AS_FREE		= 2,
 };
 
 /**
@@ -164,6 +198,9 @@ struct virtio_rpmsg_channel {
 /* Address 53 is reserved for advertising remote services */
 #define RPMSG_NS_ADDR			(53)
 
+/* Address 54 is reserved for advertising address services */
+#define RPMSG_AS_ADDR			(54)
+
 static void virtio_rpmsg_destroy_ept(struct rpmsg_endpoint *ept);
 static int virtio_rpmsg_send(struct rpmsg_endpoint *ept, void *data, int len);
 static int virtio_rpmsg_sendto(struct rpmsg_endpoint *ept, void *data, int len,
@@ -329,9 +366,11 @@ static int virtio_rpmsg_announce_create(struct rpmsg_device *rpdev)
 	struct device *dev = &rpdev->dev;
 	int err = 0;
 
+	if (!rpdev->ept || !rpdev->announce)
+		return err;
+
 	/* need to tell remote processor's name service about this channel ? */
-	if (rpdev->announce && rpdev->ept &&
-	    virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_NS)) {
+	if (virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_NS)) {
 		struct rpmsg_ns_msg nsm;
 
 		strncpy(nsm.name, rpdev->id.name, RPMSG_NAME_SIZE);
@@ -343,6 +382,23 @@ static int virtio_rpmsg_announce_create(struct rpmsg_device *rpdev)
 			dev_err(dev, "failed to announce service %d\n", err);
 	}
 
+	/*
+	 * need to tell remote processor's address service about the address allocated
+	 * to this channel
+	 */
+	if (virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_AS)) {
+		struct rpmsg_as_msg asmsg;
+
+		strncpy(asmsg.name, rpdev->id.name, RPMSG_NAME_SIZE);
+		asmsg.dst = rpdev->src;
+		asmsg.src = rpdev->dst;
+		asmsg.flags = RPMSG_AS_ASSIGN;
+
+		err = rpmsg_sendto(rpdev->ept, &asmsg, sizeof(asmsg), RPMSG_AS_ADDR);
+		if (err)
+			dev_err(dev, "failed to announce service %d\n", err);
+	}
+
 	return err;
 }
 
@@ -353,9 +409,28 @@ static int virtio_rpmsg_announce_destroy(struct rpmsg_device *rpdev)
 	struct device *dev = &rpdev->dev;
 	int err = 0;
 
+	if (!rpdev->ept || !rpdev->announce)
+		return err;
+
+	/*
+	 * need to tell remote processor's address service that we're freeing
+	 * the address allocated to this channel
+	 */
+	if (virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_AS)) {
+		struct rpmsg_as_msg asmsg;
+
+		strncpy(asmsg.name, rpdev->id.name, RPMSG_NAME_SIZE);
+		asmsg.dst = rpdev->src;
+		asmsg.src = rpdev->dst;
+		asmsg.flags = RPMSG_AS_FREE;
+
+		err = rpmsg_sendto(rpdev->ept, &asmsg, sizeof(asmsg), RPMSG_AS_ADDR);
+		if (err)
+			dev_err(dev, "failed to announce service %d\n", err);
+	}
+
 	/* tell remote processor's name service we're removing this channel */
-	if (rpdev->announce && rpdev->ept &&
-	    virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_NS)) {
+	if (virtio_has_feature(vrp->vdev, VIRTIO_RPMSG_F_NS)) {
 		struct rpmsg_ns_msg nsm;
 
 		strncpy(nsm.name, rpdev->id.name, RPMSG_NAME_SIZE);
@@ -390,7 +465,8 @@ static void virtio_rpmsg_release_device(struct device *dev)
  * channels.
  */
 static struct rpmsg_device *rpmsg_create_channel(struct virtproc_info *vrp,
-						 struct rpmsg_channel_info *chinfo)
+						 struct rpmsg_channel_info *chinfo,
+						 bool announce)
 {
 	struct virtio_rpmsg_channel *vch;
 	struct rpmsg_device *rpdev;
@@ -424,7 +500,8 @@ static struct rpmsg_device *rpmsg_create_channel(struct virtproc_info *vrp,
 	 * rpmsg server channels has predefined local address (for now),
 	 * and their existence needs to be announced remotely
 	 */
-	rpdev->announce = rpdev->src != RPMSG_ADDR_ANY;
+	if (rpdev->src != RPMSG_ADDR_ANY || announce)
+		rpdev->announce = true;
 
 	strncpy(rpdev->id.name, chinfo->name, RPMSG_NAME_SIZE);
 
@@ -873,7 +950,7 @@ static int rpmsg_ns_cb(struct rpmsg_device *rpdev, void *data, int len,
 		if (ret)
 			dev_err(dev, "rpmsg_destroy_channel failed: %d\n", ret);
 	} else {
-		newch = rpmsg_create_channel(vrp, &chinfo);
+		newch = rpmsg_create_channel(vrp, &chinfo, msg->flags & RPMSG_AS_ANNOUNCE);
 		if (!newch)
 			dev_err(dev, "rpmsg_create_channel failed\n");
 	}
@@ -1042,6 +1119,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
 	VIRTIO_RPMSG_F_NS,
+	VIRTIO_RPMSG_F_AS,
 };
 
 static struct virtio_driver virtio_ipc_driver = {
Signature · 1 line
2.17.1
#11 lore raw

RFC PATCH 11/22 rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h

In reply to Kishon Vijay Abraham I

No functional change intended. Move generic rpmsg structures like "struct rpmsg_hdr", "struct rpmsg_ns_msg" and "struct rpmsg_as_msg", its associated flags and generic macros to rpmsg_internal.h. This is in preparation to add VHOST based vhost_rpmsg_bus.c which will use the same structures and macros.

---
 drivers/rpmsg/rpmsg_internal.h   | 120 +++++++++++++++++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c | 120 -------------------------------
 2 files changed, 120 insertions(+), 120 deletions(-)
drivers/rpmsg/rpmsg_internal.h +120 -0
index 39b3a5caf242..69d9c9579b50 100644
@@ -15,6 +15,126 @@
 #include <linux/rpmsg.h>
 #include <linux/poll.h>
 
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_AS	1 /* RP supports address service notifications */
+
+/**
+ * struct rpmsg_hdr - common header for all rpmsg messages
+ * @src: source address
+ * @dst: destination address
+ * @reserved: reserved for future use
+ * @len: length of payload (in bytes)
+ * @flags: message flags
+ * @data: @len bytes of message payload data
+ *
+ * Every message sent(/received) on the rpmsg bus begins with this header.
+ */
+struct rpmsg_hdr {
+	u32 src;
+	u32 dst;
+	u32 reserved;
+	u16 len;
+	u16 flags;
+	u8 data[0];
+} __packed;
+
+/**
+ * struct rpmsg_ns_msg - dynamic name service announcement message
+ * @name: name of remote service that is published
+ * @addr: address of remote service that is published
+ * @flags: indicates whether service is created or destroyed
+ *
+ * This message is sent across to publish a new service, or announce
+ * about its removal. When we receive these messages, an appropriate
+ * rpmsg channel (i.e device) is created/destroyed. In turn, the ->probe()
+ * or ->remove() handler of the appropriate rpmsg driver will be invoked
+ * (if/as-soon-as one is registered).
+ */
+struct rpmsg_ns_msg {
+	char name[RPMSG_NAME_SIZE];
+	u32 addr;
+	u32 flags;
+} __packed;
+
+/**
+ * struct rpmsg_as_msg - dynamic address service announcement message
+ * @name: name of the created channel
+ * @dst: destination address to be used by the backend rpdev
+ * @src: source address of the backend rpdev (the one that sent name service
+ * announcement message)
+ * @flags: indicates whether service is created or destroyed
+ *
+ * This message is sent (by virtio_rpmsg_bus) when a new channel is created
+ * in response to name service announcement message by backend rpdev to create
+ * a new channel. This sends the allocated source address for the channel
+ * (destination address for the backend rpdev) to the backend rpdev.
+ */
+struct rpmsg_as_msg {
+	char name[RPMSG_NAME_SIZE];
+	u32 dst;
+	u32 src;
+	u32 flags;
+} __packed;
+
+/**
+ * enum rpmsg_ns_flags - dynamic name service announcement flags
+ *
+ * @RPMSG_NS_CREATE: a new remote service was just created
+ * @RPMSG_NS_DESTROY: a known remote service was just destroyed
+ */
+enum rpmsg_ns_flags {
+	RPMSG_NS_CREATE		= 0,
+	RPMSG_NS_DESTROY	= 1,
+	RPMSG_AS_ANNOUNCE	= 2,
+};
+
+/**
+ * enum rpmsg_as_flags - dynamic address service announcement flags
+ *
+ * @RPMSG_AS_ASSIGN: address has been assigned to the newly created channel
+ * @RPMSG_AS_FREE: assigned address is freed from the channel and no longer can
+ * be used
+ */
+enum rpmsg_as_flags {
+	RPMSG_AS_ASSIGN		= 1,
+	RPMSG_AS_FREE		= 2,
+};
+
+/*
+ * We're allocating buffers of 512 bytes each for communications. The
+ * number of buffers will be computed from the number of buffers supported
+ * by the vring, upto a maximum of 512 buffers (256 in each direction).
+ *
+ * Each buffer will have 16 bytes for the msg header and 496 bytes for
+ * the payload.
+ *
+ * This will utilize a maximum total space of 256KB for the buffers.
+ *
+ * We might also want to add support for user-provided buffers in time.
+ * This will allow bigger buffer size flexibility, and can also be used
+ * to achieve zero-copy messaging.
+ *
+ * Note that these numbers are purely a decision of this driver - we
+ * can change this without changing anything in the firmware of the remote
+ * processor.
+ */
+#define MAX_RPMSG_NUM_BUFS	(512)
+#define MAX_RPMSG_BUF_SIZE	(512)
+
+/*
+ * Local addresses are dynamically allocated on-demand.
+ * We do not dynamically assign addresses from the low 1024 range,
+ * in order to reserve that address range for predefined services.
+ */
+#define RPMSG_RESERVED_ADDRESSES	(1024)
+
+/* Address 53 is reserved for advertising remote services */
+#define RPMSG_NS_ADDR			(53)
+
+/* Address 54 is reserved for advertising address services */
+#define RPMSG_AS_ADDR			(54)
+
 #define to_rpmsg_device(d) container_of(d, struct rpmsg_device, dev)
 #define to_rpmsg_driver(d) container_of(d, struct rpmsg_driver, drv)
 
drivers/rpmsg/virtio_rpmsg_bus.c +0 -120

132 lines not on this page · raw message

Signature · 1 line
2.17.1
#12 lore raw

RFC PATCH 12/22 virtio: Add ops to allocate and free buffer

In reply to Kishon Vijay Abraham I

Add ops to allocate and free buffer in struct virtio_config_ops. Certain vhost devices can have restriction on the range of memory it can access on the virtio. The virtio drivers attached to such vhost devices reserves memory that can be accessed by vhost. This function allocates buffer for such reserved region.

For instance when virtio-vhost is used by two hosts connected to NTB, the vhost can access only memory exposed by memory windows and the size of the memory window can be limited. Here the NTB virtio driver can reserve a small region (few MBs) and provide buffer address from this pool whenever requested by virtio client driver.

---
 include/linux/virtio_config.h | 42 +++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
include/linux/virtio_config.h +42 -0
index bb4cc4910750..419f733017c2 100644
@@ -65,6 +65,9 @@ struct irq_affinity;
  *      the caller can then copy.
  * @set_vq_affinity: set the affinity for a virtqueue (optional).
  * @get_vq_affinity: get the affinity for a virtqueue (optional).
+ * @alloc_buffer: Allocate and provide buffer addresses that can be
+ *      accessed by both virtio and vhost
+ * @free_buffer: Free the allocated buffer address
  */
 typedef void vq_callback_t(struct virtqueue *);
 struct virtio_config_ops {
@@ -88,6 +91,9 @@ struct virtio_config_ops {
 			       const struct cpumask *cpu_mask);
 	const struct cpumask *(*get_vq_affinity)(struct virtio_device *vdev,
 			int index);
+	void * (*alloc_buffer)(struct virtio_device *vdev, size_t size);
+	void (*free_buffer)(struct virtio_device *vdev, void *addr,
+			    size_t size);
 };
 
 /* If driver didn't advertise the feature, it will never appear. */
@@ -232,6 +238,42 @@ const char *virtio_bus_name(struct virtio_device *vdev)
 	return vdev->config->bus_name(vdev);
 }
 
+/**
+ * virtio_alloc_buffer - Allocate buffer from the reserved memory
+ * @vdev: Virtio device which manages the reserved memory
+ * @size: Size of the buffer to be allocated
+ *
+ * Certain vhost devices can have restriction on the range of memory
+ * it can access on the virtio. The virtio drivers attached to
+ * such vhost devices reserves memory that can be accessed by
+ * vhost. This function allocates buffer for such reserved region.
+ */
+static inline void *
+virtio_alloc_buffer(struct virtio_device *vdev, size_t size)
+{
+	if (!vdev->config->alloc_buffer)
+		return NULL;
+
+	return vdev->config->alloc_buffer(vdev, size);
+}
+
+/**
+ * virtio_free_buffer - Free the allocated buffer
+ * @vdev: Virtio device which manages the reserved memory
+ * @addr: Address returned by virtio_alloc_buffer()
+ * @size: Size of the buffer that has to be freed
+ *
+ * Free the allocated buffer address given by virtio_alloc_buffer().
+ */
+static inline void
+virtio_free_buffer(struct virtio_device *vdev, void *addr, size_t size)
+{
+	if (!vdev->config->free_buffer)
+		return;
+
+	return vdev->config->free_buffer(vdev, addr, size);
+}
+
 /**
  * virtqueue_set_affinity - setting affinity for a virtqueue
  * @vq: the virtqueue
Signature · 1 line
2.17.1
#13 lore raw

RFC PATCH 13/22 rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()

In reply to Kishon Vijay Abraham I

Use virtio_alloc_buffer() and virtio_free_buffer() to allocate and free memory buffer respectively. Only if buffer allocation using virtio_alloc_buffer() try using dma_alloc_coherent(). This is required for devices like NTB to use rpmsg for communicating with other host.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 32 ++++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +22 -10
index f91143b25af7..2b25a8ae1539 100644
@@ -882,13 +882,16 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 	total_buf_space = vrp->num_bufs * vrp->buf_size;
 
-	/* allocate coherent memory for the buffers */
-	bufs_va = dma_alloc_coherent(vdev->dev.parent,
-				     total_buf_space, &vrp->bufs_dma,
-				     GFP_KERNEL);
+	bufs_va = virtio_alloc_buffer(vdev, total_buf_space);
 	if (!bufs_va) {
-		err = -ENOMEM;
-		goto vqs_del;
+		/* allocate coherent memory for the buffers */
+		bufs_va = dma_alloc_coherent(vdev->dev.parent,
+					     total_buf_space, &vrp->bufs_dma,
+					     GFP_KERNEL);
+		if (!bufs_va) {
+			err = -ENOMEM;
+			goto vqs_del;
+		}
 	}
 
 	dev_dbg(&vdev->dev, "buffers: va %pK, dma %pad\n",
@@ -951,8 +954,13 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	return 0;
 
 free_coherent:
-	dma_free_coherent(vdev->dev.parent, total_buf_space,
-			  bufs_va, vrp->bufs_dma);
+	if (!vrp->bufs_dma) {
+		virtio_free_buffer(vdev, bufs_va, total_buf_space);
+	} else {
+		dma_free_coherent(vdev->dev.parent, total_buf_space,
+				  bufs_va, vrp->bufs_dma);
+	}
+
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
 free_vrp:
@@ -986,8 +994,12 @@ static void rpmsg_remove(struct virtio_device *vdev)
 
 	vdev->config->del_vqs(vrp->vdev);
 
-	dma_free_coherent(vdev->dev.parent, total_buf_space,
-			  vrp->rbufs, vrp->bufs_dma);
+	if (!vrp->bufs_dma) {
+		virtio_free_buffer(vdev, vrp->rbufs, total_buf_space);
+	} else {
+		dma_free_coherent(vdev->dev.parent, total_buf_space,
+				  vrp->rbufs, vrp->bufs_dma);
+	}
 
 	kfree(vrp);
 }
Signature · 1 line
2.17.1
#14 lore raw

RFC PATCH 14/22 rpmsg: Add VHOST based remote processor messaging bus

In reply to Kishon Vijay Abraham I

Add a VHOST-based inter-processor communication bus, which enables kernel drivers to communicate with VIRTIO-based messaging bus, running on remote processors, over shared memory using a simple messaging protocol.

---
 drivers/rpmsg/Kconfig           |   10 +
 drivers/rpmsg/Makefile          |    1 +
 drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++++++++++++++++
 include/linux/rpmsg.h           |    1 +
 4 files changed, 1163 insertions(+)
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
drivers/rpmsg/Kconfig +10 -0
index a9108ff563dc..881712f424d3 100644
@@ -64,4 +64,14 @@ config RPMSG_VIRTIO
 	select RPMSG
 	select VIRTIO
 
+config RPMSG_VHOST
+	tristate "Vhost RPMSG bus driver"
+	depends on HAS_DMA
+	select RPMSG
+	select VHOST
+	help
+	  Say y here to enable support for the RPMSG VHOST driver
+	  providing communication channels to remote processors running
+	  RPMSG VIRTIO driver.
+
 endmenu
drivers/rpmsg/Makefile +1 -0
index 047acfda518a..44023b0abe9e 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_RPMSG_QCOM_GLINK_NATIVE) += qcom_glink_native.o
 obj-$(CONFIG_RPMSG_QCOM_GLINK_SMEM) += qcom_glink_smem.o
 obj-$(CONFIG_RPMSG_QCOM_SMD)	+= qcom_smd.o
 obj-$(CONFIG_RPMSG_VIRTIO)	+= virtio_rpmsg_bus.o
+obj-$(CONFIG_RPMSG_VHOST)	+= vhost_rpmsg_bus.o
drivers/rpmsg/vhost_rpmsg_bus.c new file +1151 -0

1,151 lines not on this page · raw message

include/linux/rpmsg.h +1 -0
index b9d9283b46ac..5d60d13efaf4 100644
@@ -55,6 +55,7 @@ struct rpmsg_device {
 	u32 dst;
 	struct rpmsg_endpoint *ept;
 	bool announce;
+	struct list_head list;
 
 	const struct rpmsg_device_ops *ops;
 };
Signature · 1 line
2.17.1
#15 lore raw

RFC PATCH 15/22 samples/rpmsg: Setup delayed work to send message

In reply to Kishon Vijay Abraham I

Let us not send any message to the remote processor before announce_create() callback has been invoked. Since announce_create() is only invoked after ->probe() is completed, setup delayed work to start sending message to the remote processor.

---
 samples/rpmsg/rpmsg_client_sample.c | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)
samples/rpmsg/rpmsg_client_sample.c +21 -7
index ae5081662283..514a51945d69 100644
@@ -20,6 +20,8 @@ module_param(count, int, 0644);
 
 struct instance_data {
 	int rx_count;
+	struct delayed_work send_msg_work;
+	struct rpmsg_device *rpdev;
 };
 
 static int rpmsg_sample_cb(struct rpmsg_device *rpdev, void *data, int len,
@@ -48,9 +50,21 @@ static int rpmsg_sample_cb(struct rpmsg_device *rpdev, void *data, int len,
 	return 0;
 }
 
-static int rpmsg_sample_probe(struct rpmsg_device *rpdev)
+static void rpmsg_sample_send_msg_work(struct work_struct *work)
 {
+	struct instance_data *idata = container_of(work, struct instance_data,
+						   send_msg_work.work);
+	struct rpmsg_device *rpdev = idata->rpdev;
 	int ret;
+
+	/* send a message to our remote processor */
+	ret = rpmsg_send(rpdev->ept, MSG, strlen(MSG));
+	if (ret)
+		dev_err(&rpdev->dev, "rpmsg_send failed: %d\n", ret);
+}
+
+static int rpmsg_sample_probe(struct rpmsg_device *rpdev)
+{
 	struct instance_data *idata;
 
 	dev_info(&rpdev->dev, "new channel: 0x%x -> 0x%x!\n",
@@ -62,18 +76,18 @@ static int rpmsg_sample_probe(struct rpmsg_device *rpdev)
 
 	dev_set_drvdata(&rpdev->dev, idata);
 
-	/* send a message to our remote processor */
-	ret = rpmsg_send(rpdev->ept, MSG, strlen(MSG));
-	if (ret) {
-		dev_err(&rpdev->dev, "rpmsg_send failed: %d\n", ret);
-		return ret;
-	}
+	idata->rpdev = rpdev;
+	INIT_DELAYED_WORK(&idata->send_msg_work, rpmsg_sample_send_msg_work);
+	schedule_delayed_work(&idata->send_msg_work, msecs_to_jiffies(500));
 
 	return 0;
 }
 
 static void rpmsg_sample_remove(struct rpmsg_device *rpdev)
 {
+	struct instance_data *idata = dev_get_drvdata(&rpdev->dev);
+
+	cancel_delayed_work_sync(&idata->send_msg_work);
 	dev_info(&rpdev->dev, "rpmsg sample client driver is removed\n");
 }
 
Signature · 1 line
2.17.1
#16 lore raw

RFC PATCH 16/22 samples/rpmsg: Wait for address to be bound to rpdev for sending message

In reply to Kishon Vijay Abraham I

rpmsg_client_sample can use either virtio_rpmsg_bus or vhost_rpmsg_bus to send messages. In the case of vhost_rpmsg_bus, the destination address of rpdev will be assigned only after receiving address service notification from the remote virtio_rpmsg_bus. Wait for address to be bound to rpdev (rpmsg_client_sample running in vhost system) before sending messages to the remote virtio_rpmsg_bus.

---
 samples/rpmsg/rpmsg_client_sample.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
samples/rpmsg/rpmsg_client_sample.c +8 -4
index 514a51945d69..07149d7fbd0c 100644
@@ -57,10 +57,14 @@ static void rpmsg_sample_send_msg_work(struct work_struct *work)
 	struct rpmsg_device *rpdev = idata->rpdev;
 	int ret;
 
-	/* send a message to our remote processor */
-	ret = rpmsg_send(rpdev->ept, MSG, strlen(MSG));
-	if (ret)
-		dev_err(&rpdev->dev, "rpmsg_send failed: %d\n", ret);
+	if (rpdev->dst != RPMSG_ADDR_ANY) {
+		/* send a message to our remote processor */
+		ret = rpmsg_send(rpdev->ept, MSG, strlen(MSG));
+		if (ret)
+			dev_err(&rpdev->dev, "rpmsg_send failed: %d\n", ret);
+	} else {
+		schedule_delayed_work(&idata->send_msg_work, msecs_to_jiffies(50));
+	}
 }
 
 static int rpmsg_sample_probe(struct rpmsg_device *rpdev)
Signature · 1 line
2.17.1
#17 lore raw

RFC PATCH 17/22 rpmsg.txt: Add Documentation to configure rpmsg using configfs

In reply to Kishon Vijay Abraham I

Add Documentation on how rpmsg device can be created using configfs required for vhost_rpmsg_bus.c

---
 Documentation/rpmsg.txt | 56 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
Documentation/rpmsg.txt +56 -0
index 24b7a9e1a5f9..0e0a32b2cb66 100644
@@ -339,3 +339,59 @@ by the bus, and can then start sending messages to the remote service.
 
 The plan is also to add static creation of rpmsg channels via the virtio
 config space, but it's not implemented yet.
+
+Configuring rpmsg using configfs
+================================
+
+Usually a rpmsg_device is created when the virtproc driver (virtio_rpmsg_bus.c)
+receives a name service notification from the remote core. However there could
+also be cases where the user should be given the ability to create rpmsg_device
+(like in the case of vhost_rpmsg_bus.c) where vhost_rpmsg_bus should be
+responsible for sending name service notification. For such cases, configfs
+provides an ability to the user for binding a rpmsg_client_driver with virtproc
+device in order to create rpmsg_device.
+
+Two configfs directories are added for configuring rpmsg
+::
+
+  # ls /sys/kernel/config/rpmsg/
+    channel   virtproc
+
+channel: Whenever a new rpmsg_driver is registered with rpmsg core, a new
+sub-directory will be created for each entry provided in rpmsg_device_id
+table of rpmsg_driver.
+
+For instance when rpmsg_sample_client is installed, it'll create the following
+entry in the mounted configfs directory
+::
+
+  # ls /sys/kernel/config/rpmsg/channel/
+    rpmsg-client-sample
+
+virtproc: A virtproc device can choose to add an entry in this directory.
+Virtproc device adds an entry if it has to allow user to control creation of
+rpmsg device. (e.g vhost_rpmsg_bus.c)
+::
+
+  # ls /sys/kernel/config/rpmsg/virtproc/
+    vhost0
+
+
+The first step in allowing the user to create rpmsg device is to create a
+sub-directory rpmsg-client-sample. For each rpmsg_device, the user would like
+to create, a separate subdirectory has to be created.
+::
+
+  # mkdir /sys/kernel/config/rpmsg/channel/rpmsg-client-sample/c1
+
+The next step is to link the created sub-directory with virtproc device to
+create rpmsg device.
+::
+
+  # ln -s /sys/kernel/config/rpmsg/channel/rpmsg-client-sample/c1 \
+        /sys/kernel/config/rpmsg/virtproc/vhost0
+
+This will create rpmsg_device. However the driver will not register the
+rpmsg device until it receives the VIRTIO_CONFIG_S_DRIVER_OK (in the case
+of vhost_rpmsg_bus.c) as it can access virtio buffers only after
+VIRTIO_CONFIG_S_DRIVER_OK is set.
Signature · 1 line
2.17.1
#18 lore raw

RFC PATCH 18/22 virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device

In reply to Kishon Vijay Abraham I

Add VIRTIO driver to support Linux VHOST on Configurable PCIe Endpoint device in the backend.

---
 drivers/virtio/Kconfig          |   9 +
 drivers/virtio/Makefile         |   1 +
 drivers/virtio/virtio_pci_epf.c | 670 ++++++++++++++++++++++++++++++++
 include/linux/virtio.h          |   3 +
 4 files changed, 683 insertions(+)
 create mode 100644 drivers/virtio/virtio_pci_epf.c
drivers/virtio/Kconfig +9 -0
index 69a32dfc318a..4b251f8482ae 100644
@@ -43,6 +43,15 @@ config VIRTIO_PCI_LEGACY
 
 	  If unsure, say Y.
 
+config VIRTIO_PCI_EPF
+	bool "Support for virtio device on configurable PCIe Endpoint"
+	depends on VIRTIO_PCI
+	help
+          Select this configuration option to enable the VIRTIO driver
+          for configurable PCIe Endpoint running Linux in backend.
+
+          If in doubt, say "N" to disable VIRTIO PCI EPF driver.
+
 config VIRTIO_VDPA
 	tristate "vDPA driver for virtio devices"
 	depends on VDPA
drivers/virtio/Makefile +1 -0
index 29a1386ecc03..08a158365d5e 100644
@@ -4,6 +4,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
 obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
 virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
 virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
+virtio_pci-$(CONFIG_VIRTIO_PCI_EPF) += virtio_pci_epf.o
 obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
 obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o
 obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o
drivers/virtio/virtio_pci_epf.c new file +670 -0

670 lines not on this page · raw message

include/linux/virtio.h +3 -0
index a493eac08393..0eb51b31545d 100644
@@ -19,6 +19,7 @@
  * @priv: a pointer for the virtqueue implementation to use.
  * @index: the zero-based ordinal number for this queue.
  * @num_free: number of elements we expect to be able to fit.
+ * @mutex: mutex to protect concurrent access to the same queue
  *
  * A note on @num_free: with indirect buffers, each buffer needs one
  * element in the queue, otherwise a buffer will need one element per
@@ -31,6 +32,8 @@ struct virtqueue {
 	struct virtio_device *vdev;
 	unsigned int index;
 	unsigned int num_free;
+	/* mutex to protect concurrent access to the queue while configuring */
+	struct mutex lock;
 	void *priv;
 };
 
Signature · 1 line
2.17.1
#19 lore raw

RFC PATCH 19/22 PCI: endpoint: Add EP function driver to provide VHOST interface

In reply to Kishon Vijay Abraham I

Add a new endpoint function driver to register VHOST device and provide interface for the VHOST driver to access virtqueues created by the remote host (using VIRTIO).

---
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 +++++++++++++++++
 drivers/vhost/vhost_cfs.c                     |   13 -
 include/linux/vhost.h                         |   14 +
 5 files changed, 1170 insertions(+), 13 deletions(-)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
drivers/pci/endpoint/functions/Kconfig +11 -0
index 55ac7bb2d469..21830576e1f4 100644
@@ -24,3 +24,14 @@ config PCI_EPF_NTB
 	   device tree.
 
 	   If in doubt, say "N" to disable Endpoint NTB driver.
+
+config PCI_EPF_VHOST
+	tristate "PCI Endpoint VHOST driver"
+	depends on PCI_ENDPOINT
+	help
+	   Select this configuration option to enable the VHOST driver
+	   for PCI Endpoint. EPF VHOST driver implements VIRTIO backend
+	   for EPF and uses the VHOST framework to bind any VHOST driver
+	   to the VHOST device created by this driver.
+
+	   If in doubt, say "N" to disable Endpoint VHOST driver.
drivers/pci/endpoint/functions/Makefile +1 -0
index 96ab932a537a..39d4f9daf63a 100644
@@ -5,3 +5,4 @@
 
 obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
+obj-$(CONFIG_PCI_EPF_VHOST)		+= pci-epf-vhost.o
drivers/pci/endpoint/functions/pci-epf-vhost.c new file +1144 -0

1,144 lines not on this page · raw message

drivers/vhost/vhost_cfs.c +0 -13
index ae46e71968f1..ab0393289200 100644
@@ -18,12 +18,6 @@ static struct config_group *vhost_driver_group;
 /* VHOST device like PCIe EP, NTB etc., */
 static struct config_group *vhost_device_group;
 
-struct vhost_driver_item {
-	struct config_group group;
-	u32 vendor;
-	u32 device;
-};
-
 struct vhost_driver_group {
 	struct config_group group;
 };
@@ -33,13 +27,6 @@ struct vhost_device_item {
 	struct vhost_dev *vdev;
 };
 
-static inline
-struct vhost_driver_item *to_vhost_driver_item(struct config_item *item)
-{
-	return container_of(to_config_group(item), struct vhost_driver_item,
-			    group);
-}
-
 static inline
 struct vhost_device_item *to_vhost_device_item(struct config_item *item)
 {
include/linux/vhost.h +14 -0
index be9341ffd266..640650311310 100644
@@ -74,6 +74,7 @@ struct vhost_virtqueue {
 	struct vhost_dev *dev;
 	enum vhost_type type;
 	struct vringh vringh;
+	int index;
 	void (*callback)(struct vhost_virtqueue *vq);
 	void (*notify)(struct vhost_virtqueue *vq);
 
@@ -148,6 +149,12 @@ struct vhost_msg_node {
   struct list_head node;
 };
 
+struct vhost_driver_item {
+	struct config_group group;
+	u32 vendor;
+	u32 device;
+};
+
 enum vhost_notify_event {
 	NOTIFY_SET_STATUS,
 	NOTIFY_FINALIZE_FEATURES,
@@ -230,6 +237,13 @@ static inline void *vhost_get_drvdata(struct vhost_dev *vdev)
 	return dev_get_drvdata(&vdev->dev);
 }
 
+static inline
+struct vhost_driver_item *to_vhost_driver_item(struct config_item *item)
+{
+	return container_of(to_config_group(item), struct vhost_driver_item,
+			    group);
+}
+
 int vhost_register_driver(struct vhost_driver *driver);
 void vhost_unregister_driver(struct vhost_driver *driver);
 int vhost_register_device(struct vhost_dev *vdev);
Signature · 1 line
2.17.1
#20 lore raw

RFC PATCH 20/22 NTB: Add a new NTB client driver to implement VIRTIO functionality

In reply to Kishon Vijay Abraham I

Add a new NTB client driver to implement VIRTIO functionality. When two hosts are connected using NTB, one of the hosts should run NTB client driver that implements VIRTIO functionality and the other host should run NTB client implements VHOST functionality. This interfaces with VIRTIO layer so that any virtio client driver can exchange data with the remote vhost client driver.

Since each NTB host can only expose fewer contiguous memory range to the remote NTB host (number of memory windows supported), reserve contiguous memory range using dma_alloc_coherent() and then manage this area using gen_pool for providing buffers to the virtio client driver. The virtio client driver should only provide this buffer region to the remote vhost driver.

---
 drivers/ntb/Kconfig      |   9 +
 drivers/ntb/Makefile     |   1 +
 drivers/ntb/ntb_virtio.c | 853 +++++++++++++++++++++++++++++++++++++++
 drivers/ntb/ntb_virtio.h |  56 +++
 4 files changed, 919 insertions(+)
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
drivers/ntb/Kconfig +9 -0
index df16c755b4da..e171b3256f68 100644
@@ -37,4 +37,13 @@ config NTB_TRANSPORT
 
 	 If unsure, say N.
 
+config NTB_VIRTIO
+	tristate "NTB VIRTIO"
+	help
+	 The NTB virtio driver sits between the NTB HW driver and the virtio
+	 client driver and lets the virtio client driver to exchange data with
+	 the remote vhost driver over the NTB hardware.
+
+	 If unsure, say N.
+
 endif # NTB
drivers/ntb/Makefile +1 -0
index 3a6fa181ff99..d37ab488bcbc 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_NTB) += ntb.o hw/ test/
 obj-$(CONFIG_NTB_TRANSPORT) += ntb_transport.o
+obj-$(CONFIG_NTB_VIRTIO) += ntb_virtio.o
 
 ntb-y			:= core.o
 ntb-$(CONFIG_NTB_MSI)	+= msi.o
drivers/ntb/ntb_virtio.c new file +853 -0

853 lines not on this page · raw message

drivers/ntb/ntb_virtio.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.17.1
#21 lore raw

RFC PATCH 21/22 NTB: Add a new NTB client driver to implement VHOST functionality

In reply to Kishon Vijay Abraham I

Add a new NTB client driver to implement VHOST functionality. When two hosts are connected using NTB, one of the hosts should run NTB client driver that implements VIRTIO functionality and the other host should run NTB client implements VHOST functionality. This interfaces with VHOST layer so that any vhost client driver can exchange data with the remote virtio client driver.

---
 drivers/ntb/Kconfig     |   9 +
 drivers/ntb/Makefile    |   1 +
 drivers/ntb/ntb_vhost.c | 776 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 786 insertions(+)
 create mode 100644 drivers/ntb/ntb_vhost.c
drivers/ntb/Kconfig +9 -0
index e171b3256f68..7d1b9bb56e71 100644
@@ -46,4 +46,13 @@ config NTB_VIRTIO
 
 	 If unsure, say N.
 
+config NTB_VHOST
+	tristate "NTB VHOST"
+	help
+	 The NTB vhost driver sits between the NTB HW driver and the vhost
+	 client driver and lets the vhost client driver to exchange data with
+	 the remote virtio driver over the NTB hardware.
+
+	 If unsure, say N.
+
 endif # NTB
drivers/ntb/Makefile +1 -0
index d37ab488bcbc..25c9937c91cf 100644
@@ -2,6 +2,7 @@
 obj-$(CONFIG_NTB) += ntb.o hw/ test/
 obj-$(CONFIG_NTB_TRANSPORT) += ntb_transport.o
 obj-$(CONFIG_NTB_VIRTIO) += ntb_virtio.o
+obj-$(CONFIG_NTB_VHOST) += ntb_vhost.o
 
 ntb-y			:= core.o
 ntb-$(CONFIG_NTB_MSI)	+= msi.o
drivers/ntb/ntb_vhost.c new file +776 -0

776 lines not on this page · raw message

Signature · 1 line
2.17.1
#22 lore raw

RFC PATCH 22/22 NTB: Describe ntb_virtio and ntb_vhost client in the documentation

In reply to Kishon Vijay Abraham I

Add a blurb in Documentation/ntb.txt to describe the ntb_virtio and ntb_vhost client

---
 Documentation/driver-api/ntb.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
Documentation/driver-api/ntb.rst +11 -0
index 87d1372da879..f84b81625397 100644
@@ -227,6 +227,17 @@ test client is interacted with through the debugfs filesystem:
 	specified peer. That peer's interrupt's occurrence file
 	should be incremented.
 
+NTB Vhost Client (ntb\_vhost) and NTB Virtio Client (ntb\_virtio)
+------------------------------------------------------------------
+
+When two hosts are connected via NTB, one of the hosts should use NTB Vhost
+Client and the other host should use NTB Virtio Client. The NTB Vhost client
+interfaces with the Linux Vhost Framework and lets it to be used with any
+vhost client driver. The NTB Virtio client interfaces with the Linux Virtio
+Framework and lets it to be used with any virtio client driver. The Vhost
+client driver and Virtio client driver creates a logic cink to exchange data
+with each other.
+
 NTB Hardware Drivers
 ====================
 
Signature · 1 line
2.17.1
#23 lore raw
In reply to Kishon Vijay Abraham I
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

      +                               +
      |                               |
      |                               |
      |                               |
      |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |

   +------v------+                                   +------v------+
   |             |                                   |             |
   |    HOST1    |                                   |    HOST2    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

    +----------------+  +-----------+  +------------+  +----------+
    |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
    +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
            |                 |              |              |
            |                 |              |              |
            |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
         |               |                    |                  |
         |               |                    |                  |
         |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Quoted 83 lines

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst              |   11 +
 Documentation/rpmsg.txt                       |   56 +
 drivers/ntb/Kconfig                           |   18 +
 drivers/ntb/Makefile                          |    2 +
 drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
 drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
 drivers/ntb/ntb_virtio.h                      |   56 +
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig                         |   10 +
 drivers/rpmsg/Makefile                        |    3 +-
 drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
 drivers/rpmsg/rpmsg_core.c                    |    7 +
 drivers/rpmsg/rpmsg_internal.h                |  136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
 drivers/vhost/Kconfig                         |    1 +
 drivers/vhost/Makefile                        |    2 +-
 drivers/vhost/net.c                           |   10 +-
 drivers/vhost/scsi.c                          |   24 +-
 drivers/vhost/test.c                          |   17 +-
 drivers/vhost/vdpa.c                          |    2 +-
 drivers/vhost/vhost.c                         |  730 ++++++++++-
 drivers/vhost/vhost_cfs.c                     |  341 +++++
 drivers/vhost/vringh.c                        |  332 +++++
 drivers/vhost/vsock.c                         |   20 +-
 drivers/virtio/Kconfig                        |    9 +
 drivers/virtio/Makefile                       |    1 +
 drivers/virtio/virtio_pci_common.c            |   25 +-
 drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
 include/linux/mod_devicetable.h               |    6 +
 include/linux/rpmsg.h                         |    6 +
 {drivers/vhost => include/linux}/vhost.h      |  132 +-
 include/linux/virtio.h                        |    3 +
 include/linux/virtio_config.h                 |   42 +
 include/linux/vringh.h                        |   46 +
 samples/rpmsg/rpmsg_client_sample.c           |   32 +-
 tools/virtio/virtio_test.c                    |    2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)
Signature · 1 line
2.17.1
#24 lore raw
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

       +                               +
       |                               |
       |                               |
       |                               |
       |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |

+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

     +----------------+  +-----------+  +------------+  +----------+
     |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
     +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
             |                 |              |              |
             |                 |              |              |
             |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
          |               |                    |                  |
          |               |                    |                  |
          |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 6 lines
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).

Have you considered to implement these through vDPA?

Thanks

Quoted 83 lines
Quoted 83 lines

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

  Documentation/driver-api/ntb.rst              |   11 +
  Documentation/rpmsg.txt                       |   56 +
  drivers/ntb/Kconfig                           |   18 +
  drivers/ntb/Makefile                          |    2 +
  drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
  drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
  drivers/ntb/ntb_virtio.h                      |   56 +
  drivers/pci/endpoint/functions/Kconfig        |   11 +
  drivers/pci/endpoint/functions/Makefile       |    1 +
  .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
  drivers/rpmsg/Kconfig                         |   10 +
  drivers/rpmsg/Makefile                        |    3 +-
  drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
  drivers/rpmsg/rpmsg_core.c                    |    7 +
  drivers/rpmsg/rpmsg_internal.h                |  136 ++
  drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
  drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
  drivers/vhost/Kconfig                         |    1 +
  drivers/vhost/Makefile                        |    2 +-
  drivers/vhost/net.c                           |   10 +-
  drivers/vhost/scsi.c                          |   24 +-
  drivers/vhost/test.c                          |   17 +-
  drivers/vhost/vdpa.c                          |    2 +-
  drivers/vhost/vhost.c                         |  730 ++++++++++-
  drivers/vhost/vhost_cfs.c                     |  341 +++++
  drivers/vhost/vringh.c                        |  332 +++++
  drivers/vhost/vsock.c                         |   20 +-
  drivers/virtio/Kconfig                        |    9 +
  drivers/virtio/Makefile                       |    1 +
  drivers/virtio/virtio_pci_common.c            |   25 +-
  drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
  include/linux/mod_devicetable.h               |    6 +
  include/linux/rpmsg.h                         |    6 +
  {drivers/vhost => include/linux}/vhost.h      |  132 +-
  include/linux/virtio.h                        |    3 +
  include/linux/virtio_config.h                 |   42 +
  include/linux/vringh.h                        |   46 +
  samples/rpmsg/rpmsg_client_sample.c           |   32 +-
  tools/virtio/virtio_test.c                    |    2 +-
  39 files changed, 7083 insertions(+), 183 deletions(-)

create mode 100644 drivers/ntb/ntb_vhost.c
create mode 100644 drivers/ntb/ntb_virtio.c
create mode 100644 drivers/ntb/ntb_virtio.h
create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1
#25 lore raw

Hi Jason,

Quoted 86 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

  VHOST RPMSG                     VIRTIO RPMSG
       +                               +

       |                               |
       |                               |
       |                               |
       |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

      VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

           |                                                 |
           |                                                 |
           |                                                 |
           |                                                 |
    +------v------+                                   +------v------+
    |             |                                   |             |
    |    HOST1    |                                   |    HOST2    |
    |             |                                   |             |
    +------^------+                                   +------^------+
           |                                                 |
           |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

     +----------------+  +-----------+  +------------+  +----------+
     |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
     +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
             |                 |              |              |
             |                 |              |              |
             |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
          |               |                    |                  |
          |               |                    |                  |
          |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization. So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 2 lines

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above), all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

Thanks
Kishon

Quoted 88 lines

Thanks

Quoted 83 lines
Quoted 83 lines

Kishon Vijay Abraham I (22):
   vhost: Make _feature_ bits a property of vhost device
   vhost: Introduce standard Linux driver model in VHOST
   vhost: Add ops for the VHOST driver to configure VHOST device
   vringh: Add helpers to access vring in MMIO
   vhost: Add MMIO helpers for operations on vhost virtqueue
   vhost: Introduce configfs entry for configuring VHOST
   virtio_pci: Use request_threaded_irq() instead of request_irq()
   rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
   rpmsg: Introduce configfs entry for configuring rpmsg
   rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
   rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
   virtio: Add ops to allocate and free buffer
   rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
   rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
   rpmsg.txt: Add Documentation to configure rpmsg using configfs
   virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
   PCI: endpoint: Add EP function driver to provide VHOST interface
   NTB: Add a new NTB client driver to implement VIRTIO functionality
   NTB: Add a new NTB client driver to implement VHOST functionality
   NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

  Documentation/driver-api/ntb.rst              |   11 + Documentation/rpmsg.txt                       |   56 + drivers/ntb/Kconfig                           |   18 + drivers/ntb/Makefile                          |    2 + drivers/ntb/ntb_vhost.c                       |  776 +++++++++++ drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++ drivers/ntb/ntb_virtio.h                      |   56 + drivers/pci/endpoint/functions/Kconfig        |   11 + drivers/pci/endpoint/functions/Makefile       |    1 + .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig                         |   10 + drivers/rpmsg/Makefile                        |    3 +- drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++ drivers/rpmsg/rpmsg_core.c                    |    7 + drivers/rpmsg/rpmsg_internal.h                |  136 ++ drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++- drivers/vhost/Kconfig                         |    1 + drivers/vhost/Makefile                        |    2 +- drivers/vhost/net.c                           |   10 +-

  drivers/vhost/scsi.c                          |   24 +-
  drivers/vhost/test.c                          |   17 +-
  drivers/vhost/vdpa.c                          |    2 +-

  drivers/vhost/vhost.c                         |  730 ++++++++++- drivers/vhost/vhost_cfs.c                     |  341 +++++ drivers/vhost/vringh.c                        |  332 +++++ drivers/vhost/vsock.c                         |   20 +- drivers/virtio/Kconfig                        |    9 + drivers/virtio/Makefile                       |    1 + drivers/virtio/virtio_pci_common.c            |   25 +- drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++ include/linux/mod_devicetable.h               |    6 + include/linux/rpmsg.h                         |    6 + {drivers/vhost => include/linux}/vhost.h      |  132 +- include/linux/virtio.h                        |    3 + include/linux/virtio_config.h                 |   42 + include/linux/vringh.h                        |   46 + samples/rpmsg/rpmsg_client_sample.c           |   32 +- tools/virtio/virtio_test.c                    |    2 +-

  39 files changed, 7083 insertions(+), 183 deletions(-)

  create mode 100644 drivers/ntb/ntb_vhost.c
  create mode 100644 drivers/ntb/ntb_virtio.c
  create mode 100644 drivers/ntb/ntb_virtio.h
  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
  create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
  create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#26 lore raw
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

  VHOST RPMSG                     VIRTIO RPMSG
       +                               +

        |                               |
        |                               |
        |                               |
        |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

      VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

            |                                                 |
            |                                                 |
            |                                                 |
            |                                                 |
     +------v------+                                   +------v------+
     |             |                                   |             |
     |    HOST1    |                                   |    HOST2    |
     |             |                                   |             |
     +------^------+                                   +------^------+
            |                                                 |
            |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

      +----------------+  +-----------+  +------------+  +----------+
      |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
      +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
              |                 |              |              |
              |                 |              |              |
              |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
           |               |                    |                  |
           |               |                    |                  |
           |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 4 lines
Quoted 2 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

Quoted 7 lines

So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Quoted 7 lines
Quoted 2 lines

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 3 lines

all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Does this make sense?

Thanks

Quoted 89 lines

Thanks
Kishon

Quoted 85 lines

Thanks

Quoted 82 lines
Quoted 82 lines

Kishon Vijay Abraham I (22):
   vhost: Make _feature_ bits a property of vhost device
   vhost: Introduce standard Linux driver model in VHOST
   vhost: Add ops for the VHOST driver to configure VHOST device
   vringh: Add helpers to access vring in MMIO
   vhost: Add MMIO helpers for operations on vhost virtqueue
   vhost: Introduce configfs entry for configuring VHOST
   virtio_pci: Use request_threaded_irq() instead of request_irq()
   rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
   rpmsg: Introduce configfs entry for configuring rpmsg
   rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
   rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
   virtio: Add ops to allocate and free buffer
   rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
   rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
   rpmsg.txt: Add Documentation to configure rpmsg using configfs
   virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
   PCI: endpoint: Add EP function driver to provide VHOST interface
   NTB: Add a new NTB client driver to implement VIRTIO functionality
   NTB: Add a new NTB client driver to implement VHOST functionality
   NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

   Documentation/driver-api/ntb.rst              |   11 +
   Documentation/rpmsg.txt                       |   56 +
   drivers/ntb/Kconfig                           |   18 +
   drivers/ntb/Makefile                          |    2 +
   drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
   drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
   drivers/ntb/ntb_virtio.h                      |   56 +
   drivers/pci/endpoint/functions/Kconfig        |   11 +
   drivers/pci/endpoint/functions/Makefile       |    1 +
   .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
   drivers/rpmsg/Kconfig                         |   10 +
   drivers/rpmsg/Makefile                        |    3 +-
   drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
   drivers/rpmsg/rpmsg_core.c                    |    7 +
   drivers/rpmsg/rpmsg_internal.h                |  136 ++
   drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
   drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
   drivers/vhost/Kconfig                         |    1 +
   drivers/vhost/Makefile                        |    2 +-
   drivers/vhost/net.c                           |   10 +-
   drivers/vhost/scsi.c                          |   24 +-
   drivers/vhost/test.c                          |   17 +-
   drivers/vhost/vdpa.c                          |    2 +-
   drivers/vhost/vhost.c                         |  730 ++++++++++-
   drivers/vhost/vhost_cfs.c                     |  341 +++++
   drivers/vhost/vringh.c                        |  332 +++++
   drivers/vhost/vsock.c                         |   20 +-
   drivers/virtio/Kconfig                        |    9 +
   drivers/virtio/Makefile                       |    1 +
   drivers/virtio/virtio_pci_common.c            |   25 +-
   drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
   include/linux/mod_devicetable.h               |    6 +
   include/linux/rpmsg.h                         |    6 +
   {drivers/vhost => include/linux}/vhost.h      |  132 +-
   include/linux/virtio.h                        |    3 +
   include/linux/virtio_config.h                 |   42 +
   include/linux/vringh.h                        |   46 +
   samples/rpmsg/rpmsg_client_sample.c           |   32 +-
   tools/virtio/virtio_test.c                    |    2 +-
   39 files changed, 7083 insertions(+), 183 deletions(-)

  create mode 100644 drivers/ntb/ntb_vhost.c
  create mode 100644 drivers/ntb/ntb_virtio.c
  create mode 100644 drivers/ntb/ntb_virtio.h
  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
  create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
  create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#27 lore raw

Hi Jason,

Quoted 104 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

   VHOST RPMSG                     VIRTIO RPMSG
        +                               +

        |                               |
        |                               |
        |                               |
        |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

       VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

            |                                                 |
            |                                                 |
            |                                                 |
            |                                                 |
     +------v------+                                   +------v------+
     |             |                                   |             |
     |    HOST1    |                                   |    HOST2    |
     |             |                                   |             |
     +------^------+                                   +------^------+
            |                                                 |
            |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

      +----------------+  +-----------+  +------------+  +----------+
      |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
      +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
              |                 |              |              |
              |                 |              |              |
              |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
           |               |                    |                  |
           |               |                    |                  |
           |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 2 lines
Quoted 2 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).

Quoted 7 lines
Quoted 2 lines

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd epf idt intel mscc

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c
Quoted 11 lines
Quoted 9 lines

  So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Quoted 14 lines
Quoted 9 lines
Quoted 4 lines

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 8 lines
Quoted 3 lines

  all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 11 lines

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue, in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 2 lines

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Thanks for your suggestions!

Regards
Kishon

Quoted 94 lines

Thanks

Quoted 89 lines

Thanks
Kishon

Quoted 85 lines

Thanks

Quoted 82 lines
Quoted 82 lines

Kishon Vijay Abraham I (22):
    vhost: Make _feature_ bits a property of vhost device
    vhost: Introduce standard Linux driver model in VHOST
    vhost: Add ops for the VHOST driver to configure VHOST device
    vringh: Add helpers to access vring in MMIO
    vhost: Add MMIO helpers for operations on vhost virtqueue
    vhost: Introduce configfs entry for configuring VHOST
    virtio_pci: Use request_threaded_irq() instead of request_irq()
    rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
    rpmsg: Introduce configfs entry for configuring rpmsg
    rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
    rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
    virtio: Add ops to allocate and free buffer
    rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
    rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
    rpmsg.txt: Add Documentation to configure rpmsg using configfs
    virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
    PCI: endpoint: Add EP function driver to provide VHOST interface
    NTB: Add a new NTB client driver to implement VIRTIO functionality
    NTB: Add a new NTB client driver to implement VHOST functionality
    NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

   Documentation/driver-api/ntb.rst              |   11 + Documentation/rpmsg.txt                       |   56 + drivers/ntb/Kconfig                           |   18 + drivers/ntb/Makefile                          |    2 + drivers/ntb/ntb_vhost.c                       |  776 +++++++++++ drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++ drivers/ntb/ntb_virtio.h                      |   56 + drivers/pci/endpoint/functions/Kconfig        |   11 + drivers/pci/endpoint/functions/Makefile       |    1 + .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig                         |   10 + drivers/rpmsg/Makefile                        |    3 +- drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++ drivers/rpmsg/rpmsg_core.c                    |    7 + drivers/rpmsg/rpmsg_internal.h                |  136 ++ drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++- drivers/vhost/Kconfig                         |    1 + drivers/vhost/Makefile                        |    2 +- drivers/vhost/net.c                           |   10 +-

   drivers/vhost/scsi.c                          |   24 +-
   drivers/vhost/test.c                          |   17 +-
   drivers/vhost/vdpa.c                          |    2 +-

   drivers/vhost/vhost.c                         |  730 ++++++++++- drivers/vhost/vhost_cfs.c                     |  341 +++++ drivers/vhost/vringh.c                        |  332 +++++ drivers/vhost/vsock.c                         |   20 +- drivers/virtio/Kconfig                        |    9 + drivers/virtio/Makefile                       |    1 + drivers/virtio/virtio_pci_common.c            |   25 +- drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++ include/linux/mod_devicetable.h               |    6 + include/linux/rpmsg.h                         |    6 + {drivers/vhost => include/linux}/vhost.h      |  132 +- include/linux/virtio.h                        |    3 + include/linux/virtio_config.h                 |   42 + include/linux/vringh.h                        |   46 + samples/rpmsg/rpmsg_client_sample.c           |   32 +- tools/virtio/virtio_test.c                    |    2 +-

   39 files changed, 7083 insertions(+), 183 deletions(-)

   create mode 100644 drivers/ntb/ntb_vhost.c
   create mode 100644 drivers/ntb/ntb_virtio.c
   create mode 100644 drivers/ntb/ntb_virtio.h
   create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
   create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
   create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#28 lore raw
Quoted 118 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 86 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

   VHOST RPMSG                     VIRTIO RPMSG
        +                               +

         |                               |
         |                               |
         |                               |
         |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

       VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

             |                                                 |
             |                                                 |
             |                                                 |
             |                                                 |
      +------v------+                                   +------v------+
      |             |                                   |             |
      |    HOST1    |                                   |    HOST2    |
      |             |                                   |             |
      +------^------+                                   +------^------+
             |                                                 |
             |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

       +----------------+  +-----------+  +------------+  +----------+
       |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
       +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
               |                 |              |              |
               |                 |              |              |
               |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
            |               |                    |                  |
            |               |                    |                  |
            |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd epf idt intel mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

  So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 26 lines
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 3 lines

  all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 6 lines
Quoted 6 lines

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

Quoted 7 lines
Quoted 5 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

Quoted 7 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 2 lines

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 95 lines

Regards
Kishon

Quoted 91 lines

Thanks

Quoted 88 lines

Thanks
Kishon

Quoted 85 lines

Thanks

Quoted 82 lines
Quoted 82 lines

Kishon Vijay Abraham I (22):
    vhost: Make _feature_ bits a property of vhost device
    vhost: Introduce standard Linux driver model in VHOST
    vhost: Add ops for the VHOST driver to configure VHOST device
    vringh: Add helpers to access vring in MMIO
    vhost: Add MMIO helpers for operations on vhost virtqueue
    vhost: Introduce configfs entry for configuring VHOST
    virtio_pci: Use request_threaded_irq() instead of request_irq()
    rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
    rpmsg: Introduce configfs entry for configuring rpmsg
    rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
    rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
    virtio: Add ops to allocate and free buffer
    rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
    rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
    rpmsg.txt: Add Documentation to configure rpmsg using configfs
    virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
    PCI: endpoint: Add EP function driver to provide VHOST interface
    NTB: Add a new NTB client driver to implement VIRTIO functionality
    NTB: Add a new NTB client driver to implement VHOST functionality
    NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

    Documentation/driver-api/ntb.rst              |   11 +
    Documentation/rpmsg.txt                       |   56 +
    drivers/ntb/Kconfig                           |   18 +
    drivers/ntb/Makefile                          |    2 +
    drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
    drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
    drivers/ntb/ntb_virtio.h                      |   56 +
    drivers/pci/endpoint/functions/Kconfig        |   11 +
    drivers/pci/endpoint/functions/Makefile       |    1 +
    .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
    drivers/rpmsg/Kconfig                         |   10 +
    drivers/rpmsg/Makefile                        |    3 +-
    drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
    drivers/rpmsg/rpmsg_core.c                    |    7 +
    drivers/rpmsg/rpmsg_internal.h                |  136 ++
    drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
    drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
    drivers/vhost/Kconfig                         |    1 +
    drivers/vhost/Makefile                        |    2 +-
    drivers/vhost/net.c                           |   10 +-
    drivers/vhost/scsi.c                          |   24 +-
    drivers/vhost/test.c                          |   17 +-
    drivers/vhost/vdpa.c                          |    2 +-
    drivers/vhost/vhost.c                         |  730 ++++++++++-
    drivers/vhost/vhost_cfs.c                     |  341 +++++
    drivers/vhost/vringh.c                        |  332 +++++
    drivers/vhost/vsock.c                         |   20 +-
    drivers/virtio/Kconfig                        |    9 +
    drivers/virtio/Makefile                       |    1 +
    drivers/virtio/virtio_pci_common.c            |   25 +-
    drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
    include/linux/mod_devicetable.h               |    6 +
    include/linux/rpmsg.h                         |    6 +
    {drivers/vhost => include/linux}/vhost.h      |  132 +-
    include/linux/virtio.h                        |    3 +
    include/linux/virtio_config.h                 |   42 +
    include/linux/vringh.h                        |   46 +
    samples/rpmsg/rpmsg_client_sample.c           |   32 +-
    tools/virtio/virtio_test.c                    |    2 +-
    39 files changed, 7083 insertions(+), 183 deletions(-)

   create mode 100644 drivers/ntb/ntb_vhost.c
   create mode 100644 drivers/ntb/ntb_virtio.c
   create mode 100644 drivers/ntb/ntb_virtio.h
   create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
   create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
   create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#29 lore raw

Hi Jason,

Quoted 209 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 103 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

    VHOST RPMSG                     VIRTIO RPMSG
         +                               +

         |                               |
         |                               |
         |                               |
         |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

        VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

             |                                                 |
             |                                                 |
             |                                                 |
             |                                                 |
      +------v------+                                   +------v------+
      |             |                                   |             |
      |    HOST1    |                                   |    HOST2    |
      |             |                                   |             |
      +------^------+                                   +------^------+
             |                                                 |
             |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

       +----------------+  +-----------+  +------------+  +----------+
       |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
       +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
               |                 |              |              |
               |                 |              |              |
               |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
            |               |                    |                  |
            |               |                    |                  |
            |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 3 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

Quoted 6 lines
Quoted 2 lines

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

   So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 27 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 3 lines

   all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Quoted 8 lines
Quoted 2 lines

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh? And virtio ring(2) is created by virtio pci (RC).

Quoted 3 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver) and the other is how certain inputs to virtio device such as number of buffers be determined.

Thanks again for your suggestions!

Regards
Kishon

Quoted 119 lines
Quoted 9 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 2 lines

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 95 lines

Regards
Kishon

Quoted 91 lines

Thanks

Quoted 88 lines

Thanks
Kishon

Quoted 85 lines

Thanks

Quoted 82 lines
Quoted 82 lines

Kishon Vijay Abraham I (22):
     vhost: Make _feature_ bits a property of vhost device
     vhost: Introduce standard Linux driver model in VHOST
     vhost: Add ops for the VHOST driver to configure VHOST device
     vringh: Add helpers to access vring in MMIO
     vhost: Add MMIO helpers for operations on vhost virtqueue
     vhost: Introduce configfs entry for configuring VHOST
     virtio_pci: Use request_threaded_irq() instead of request_irq()
     rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
     rpmsg: Introduce configfs entry for configuring rpmsg
     rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
     rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
     virtio: Add ops to allocate and free buffer
     rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
     rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
     rpmsg.txt: Add Documentation to configure rpmsg using configfs
     virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
     PCI: endpoint: Add EP function driver to provide VHOST interface
     NTB: Add a new NTB client driver to implement VIRTIO functionality
     NTB: Add a new NTB client driver to implement VHOST functionality
     NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

    Documentation/driver-api/ntb.rst              |   11 + Documentation/rpmsg.txt                       |   56 + drivers/ntb/Kconfig                           |   18 + drivers/ntb/Makefile                          |    2 + drivers/ntb/ntb_vhost.c                       |  776 +++++++++++ drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++ drivers/ntb/ntb_virtio.h                      |   56 + drivers/pci/endpoint/functions/Kconfig        |   11 + drivers/pci/endpoint/functions/Makefile       |    1 + .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++ drivers/rpmsg/Kconfig                         |   10 + drivers/rpmsg/Makefile                        |    3 +- drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++ drivers/rpmsg/rpmsg_core.c                    |    7 + drivers/rpmsg/rpmsg_internal.h                |  136 ++ drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++ drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++- drivers/vhost/Kconfig                         |    1 + drivers/vhost/Makefile                        |    2 +- drivers/vhost/net.c                           |   10 +-

    drivers/vhost/scsi.c                          |   24 +-
    drivers/vhost/test.c                          |   17 +-
    drivers/vhost/vdpa.c                          |    2 +-

    drivers/vhost/vhost.c                         |  730 ++++++++++- drivers/vhost/vhost_cfs.c                     |  341 +++++ drivers/vhost/vringh.c                        |  332 +++++ drivers/vhost/vsock.c                         |   20 +- drivers/virtio/Kconfig                        |    9 + drivers/virtio/Makefile                       |    1 + drivers/virtio/virtio_pci_common.c            |   25 +- drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++ include/linux/mod_devicetable.h               |    6 + include/linux/rpmsg.h                         |    6 + {drivers/vhost => include/linux}/vhost.h      |  132 +- include/linux/virtio.h                        |    3 + include/linux/virtio_config.h                 |   42 + include/linux/vringh.h                        |   46 + samples/rpmsg/rpmsg_client_sample.c           |   32 +- tools/virtio/virtio_test.c                    |    2 +-

    39 files changed, 7083 insertions(+), 183 deletions(-)

    create mode 100644 drivers/ntb/ntb_vhost.c
    create mode 100644 drivers/ntb/ntb_virtio.c
    create mode 100644 drivers/ntb/ntb_virtio.h
    create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
    create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
    create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#30 lore raw
Quoted 198 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 195 lines · Jason Wang
Quoted 118 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

    VHOST RPMSG                     VIRTIO RPMSG
         +                               +

          |                               |
          |                               |
          |                               |
          |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

        VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

              |                                                 |
              |                                                 |
              |                                                 |
              |                                                 |
       +------v------+                                   +------v------+
       |             |                                   |             |
       |    HOST1    |                                   |    HOST2    |
       |             |                                   |             |
       +------^------+                                   +------^------+
              |                                                 |
              |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

        +----------------+  +-----------+  +------------+  +----------+
        |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
        +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                |                 |              |              |
                |                 |              |              |
                |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
             |               |                    |                  |
             |               |                    |                  |
             |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

   So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 24 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 6 lines
Quoted 3 lines

   all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 5 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend) cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 8 lines
Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2) <- virtio ring(2) -> virtio-rpmsg (HOST2)

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

Quoted 2 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Yes.

Quoted 3 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 2 lines

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Thanks

Quoted 117 lines

Regards
Kishon

Quoted 113 lines
Quoted 8 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 1 line

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 94 lines

Regards
Kishon

Quoted 91 lines

Thanks

Quoted 88 lines

Thanks
Kishon

Quoted 85 lines

Thanks

Quoted 82 lines
Quoted 82 lines

Kishon Vijay Abraham I (22):
     vhost: Make _feature_ bits a property of vhost device
     vhost: Introduce standard Linux driver model in VHOST
     vhost: Add ops for the VHOST driver to configure VHOST device
     vringh: Add helpers to access vring in MMIO
     vhost: Add MMIO helpers for operations on vhost virtqueue
     vhost: Introduce configfs entry for configuring VHOST
     virtio_pci: Use request_threaded_irq() instead of request_irq()
     rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
     rpmsg: Introduce configfs entry for configuring rpmsg
     rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
     rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
     virtio: Add ops to allocate and free buffer
     rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
     rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
     rpmsg.txt: Add Documentation to configure rpmsg using configfs
     virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint device
     PCI: endpoint: Add EP function driver to provide VHOST interface
     NTB: Add a new NTB client driver to implement VIRTIO functionality
     NTB: Add a new NTB client driver to implement VHOST functionality
     NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

     Documentation/driver-api/ntb.rst              |   11 +
     Documentation/rpmsg.txt                       |   56 +
     drivers/ntb/Kconfig                           |   18 +
     drivers/ntb/Makefile                          |    2 +
     drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
     drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
     drivers/ntb/ntb_virtio.h                      |   56 +
     drivers/pci/endpoint/functions/Kconfig        |   11 +
     drivers/pci/endpoint/functions/Makefile       |    1 +
     .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
     drivers/rpmsg/Kconfig                         |   10 +
     drivers/rpmsg/Makefile                        |    3 +-
     drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
     drivers/rpmsg/rpmsg_core.c                    |    7 +
     drivers/rpmsg/rpmsg_internal.h                |  136 ++
     drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
     drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
     drivers/vhost/Kconfig                         |    1 +
     drivers/vhost/Makefile                        |    2 +-
     drivers/vhost/net.c                           |   10 +-
     drivers/vhost/scsi.c                          |   24 +-
     drivers/vhost/test.c                          |   17 +-
     drivers/vhost/vdpa.c                          |    2 +-
     drivers/vhost/vhost.c                         |  730 ++++++++++-
     drivers/vhost/vhost_cfs.c                     |  341 +++++
     drivers/vhost/vringh.c                        |  332 +++++
     drivers/vhost/vsock.c                         |   20 +-
     drivers/virtio/Kconfig                        |    9 +
     drivers/virtio/Makefile                       |    1 +
     drivers/virtio/virtio_pci_common.c            |   25 +-
     drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
     include/linux/mod_devicetable.h               |    6 +
     include/linux/rpmsg.h                         |    6 +
     {drivers/vhost => include/linux}/vhost.h      |  132 +-
     include/linux/virtio.h                        |    3 +
     include/linux/virtio_config.h                 |   42 +
     include/linux/vringh.h                        |   46 +
     samples/rpmsg/rpmsg_client_sample.c           |   32 +-
     tools/virtio/virtio_test.c                    |    2 +-
     39 files changed, 7083 insertions(+), 183 deletions(-)

    create mode 100644 drivers/ntb/ntb_vhost.c
    create mode 100644 drivers/ntb/ntb_virtio.c
    create mode 100644 drivers/ntb/ntb_virtio.h
    create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
    create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
    create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1
#31 lore raw

Hi Jason,

Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG
          +                               +

          |                               |
          |                               |
          |                               |
          |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

              |                                                 |
              |                                                 |
              |                                                 |
              |                                                 |
       +------v------+                                   +------v------+
       |             |                                   |             |
       |    HOST1    |                                   |    HOST2    |
       |             |                                   |             |
       +------^------+                                   +------^------+
              |                                                 |
              |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

        +----------------+  +-----------+  +------------+  +----------+
        |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
        +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                |                 |              |              |
                |                 |              |              |
                |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
             |               |                    |                  |
             |               |                    |                  |
             |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 4 lines

    all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 8 lines
Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 9 lines

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 6 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Quoted 8 lines
Quoted 5 lines

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Quoted 7 lines
Quoted 4 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Quoted 23 lines
Quoted 5 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 2 lines

Thanks again for your suggestions!

Quoted 6 lines

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

I'll try to create something based on your proposed design here.

Regards
Kishon

[1] ->
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 127 lines

Thanks

Quoted 122 lines

Regards
Kishon

Quoted 118 lines
Quoted 9 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 1 line

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 98 lines

Regards
Kishon

Quoted 95 lines

Thanks

Quoted 92 lines

Thanks
Kishon

Quoted 89 lines

Thanks

Quoted 86 lines
Quoted 86 lines

Kishon Vijay Abraham I (22):
      vhost: Make _feature_ bits a property of vhost device
      vhost: Introduce standard Linux driver model in VHOST
      vhost: Add ops for the VHOST driver to configure VHOST device
      vringh: Add helpers to access vring in MMIO
      vhost: Add MMIO helpers for operations on vhost virtqueue
      vhost: Introduce configfs entry for configuring VHOST
      virtio_pci: Use request_threaded_irq() instead of request_irq()
      rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
      rpmsg: Introduce configfs entry for configuring rpmsg
      rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
      rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
      virtio: Add ops to allocate and free buffer
      rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
      rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
      rpmsg.txt: Add Documentation to configure rpmsg using configfs
      virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
        device
      PCI: endpoint: Add EP function driver to provide VHOST interface
      NTB: Add a new NTB client driver to implement VIRTIO functionality
      NTB: Add a new NTB client driver to implement VHOST functionality
      NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

     Documentation/driver-api/ntb.rst              |   11 + Documentation/rpmsg.txt                       |   56 + drivers/ntb/Kconfig                           |   18 + drivers/ntb/Makefile                          |    2 + drivers/ntb/ntb_vhost.c                       |  776 +++++++++++ drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++ drivers/ntb/ntb_virtio.h                      |   56 + drivers/pci/endpoint/functions/Kconfig        |   11 + drivers/pci/endpoint/functions/Makefile       |    1 + .../pci/endpoint/functions/pci-epf-vhost.c    | 1144
++++++++++++++++
     drivers/rpmsg/Kconfig                         |   10 + drivers/rpmsg/Makefile                        |    3 +- drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++ drivers/rpmsg/rpmsg_core.c                    |    7 + drivers/rpmsg/rpmsg_internal.h                |  136 ++ drivers/rpmsg/vhost_rpmsg_bus.c               | 1151
+++++++++++++++++
     drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++- drivers/vhost/Kconfig                         |    1 + drivers/vhost/Makefile                        |    2 +- drivers/vhost/net.c                           |   10 +-

     drivers/vhost/scsi.c                          |   24 +-
     drivers/vhost/test.c                          |   17 +-
     drivers/vhost/vdpa.c                          |    2 +-

     drivers/vhost/vhost.c                         |  730 ++++++++++- drivers/vhost/vhost_cfs.c                     |  341 +++++ drivers/vhost/vringh.c                        |  332 +++++ drivers/vhost/vsock.c                         |   20 +- drivers/virtio/Kconfig                        |    9 + drivers/virtio/Makefile                       |    1 + drivers/virtio/virtio_pci_common.c            |   25 +- drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++ include/linux/mod_devicetable.h               |    6 + include/linux/rpmsg.h                         |    6 + {drivers/vhost => include/linux}/vhost.h      |  132 +- include/linux/virtio.h                        |    3 + include/linux/virtio_config.h                 |   42 + include/linux/vringh.h                        |   46 + samples/rpmsg/rpmsg_client_sample.c           |   32 +- tools/virtio/virtio_test.c                    |    2 +-

     39 files changed, 7083 insertions(+), 183 deletions(-)

     create mode 100644 drivers/ntb/ntb_vhost.c
     create mode 100644 drivers/ntb/ntb_virtio.c
     create mode 100644 drivers/ntb/ntb_virtio.h
     create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
     create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
     create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#32 lore raw
Quoted 221 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 209 lines · Jason Wang
Quoted 199 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 196 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG +                               +

           |                               |
           |                               |
           |                               |
           |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

               |                                                 |
               |                                                 |
               |                                                 |
               |                                                 |
        +------v------+                                   +------v------+
        |             |                                   |             |
        |    HOST1    |                                   |    HOST2    |
        |             |                                   |             |
        +------^------+                                   +------^------+
               |                                                 |
               |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

         +----------------+  +-----------+  +------------+  +----------+
         |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
         +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                 |                 |              |              |
                 |                 |              |              |
                 |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
              |               |                    |                  |
              |               |                    |                  |
              |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 4 lines

    all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 8 lines
Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 9 lines

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 2 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

Quoted 40 lines
Quoted 40 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 10 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 17 lines
Quoted 17 lines
Quoted 3 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 1 line

Thanks again for your suggestions!

Quoted 7 lines
Quoted 6 lines

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

Quoted 2 lines

I'll try to create something based on your proposed design here.

Sure, but for coding, we'd better wait for other's opinion here.

Thanks

Quoted 127 lines

Regards
Kishon

[1] ->
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 121 lines

Thanks

Quoted 118 lines

Regards
Kishon

Quoted 115 lines
Quoted 8 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 1 line

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 98 lines

Regards
Kishon

Quoted 95 lines

Thanks

Quoted 92 lines

Thanks
Kishon

Quoted 89 lines

Thanks

Quoted 86 lines
Quoted 86 lines

Kishon Vijay Abraham I (22):
      vhost: Make _feature_ bits a property of vhost device
      vhost: Introduce standard Linux driver model in VHOST
      vhost: Add ops for the VHOST driver to configure VHOST device
      vringh: Add helpers to access vring in MMIO
      vhost: Add MMIO helpers for operations on vhost virtqueue
      vhost: Introduce configfs entry for configuring VHOST
      virtio_pci: Use request_threaded_irq() instead of request_irq()
      rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
      rpmsg: Introduce configfs entry for configuring rpmsg
      rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
      rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
      virtio: Add ops to allocate and free buffer
      rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
      rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
      rpmsg.txt: Add Documentation to configure rpmsg using configfs
      virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe
Endpoint
        device
      PCI: endpoint: Add EP function driver to provide VHOST interface
      NTB: Add a new NTB client driver to implement VIRTIO functionality
      NTB: Add a new NTB client driver to implement VHOST functionality
      NTB: Describe the ntb_virtio and ntb_vhost client in the
documentation

      Documentation/driver-api/ntb.rst              |   11 +
      Documentation/rpmsg.txt                       |   56 +
      drivers/ntb/Kconfig                           |   18 +
      drivers/ntb/Makefile                          |    2 +
      drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
      drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
      drivers/ntb/ntb_virtio.h                      |   56 +
      drivers/pci/endpoint/functions/Kconfig        |   11 +
      drivers/pci/endpoint/functions/Makefile       |    1 +
      .../pci/endpoint/functions/pci-epf-vhost.c    | 1144
++++++++++++++++
      drivers/rpmsg/Kconfig                         |   10 +
      drivers/rpmsg/Makefile                        |    3 +-
      drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
      drivers/rpmsg/rpmsg_core.c                    |    7 +
      drivers/rpmsg/rpmsg_internal.h                |  136 ++
      drivers/rpmsg/vhost_rpmsg_bus.c               | 1151
+++++++++++++++++
      drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
      drivers/vhost/Kconfig                         |    1 +
      drivers/vhost/Makefile                        |    2 +-
      drivers/vhost/net.c                           |   10 +-
      drivers/vhost/scsi.c                          |   24 +-
      drivers/vhost/test.c                          |   17 +-
      drivers/vhost/vdpa.c                          |    2 +-
      drivers/vhost/vhost.c                         |  730 ++++++++++-
      drivers/vhost/vhost_cfs.c                     |  341 +++++
      drivers/vhost/vringh.c                        |  332 +++++
      drivers/vhost/vsock.c                         |   20 +-
      drivers/virtio/Kconfig                        |    9 +
      drivers/virtio/Makefile                       |    1 +
      drivers/virtio/virtio_pci_common.c            |   25 +-
      drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
      include/linux/mod_devicetable.h               |    6 +
      include/linux/rpmsg.h                         |    6 +
      {drivers/vhost => include/linux}/vhost.h      |  132 +-
      include/linux/virtio.h                        |    3 +
      include/linux/virtio_config.h                 |   42 +
      include/linux/vringh.h                        |   46 +
      samples/rpmsg/rpmsg_client_sample.c           |   32 +-
      tools/virtio/virtio_test.c                    |    2 +-
      39 files changed, 7083 insertions(+), 183 deletions(-)

     create mode 100644 drivers/ntb/ntb_vhost.c
     create mode 100644 drivers/ntb/ntb_virtio.c
     create mode 100644 drivers/ntb/ntb_virtio.h
     create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
     create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
     create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1
#33 lore raw

On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:

[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]

Quoted 66 lines · Kishon Vijay Abraham I
Quoted 65 lines

Hi Jason,

Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 54 lines · Jason Wang
Quoted 53 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 50 lines · Jason Wang
Quoted 49 lines · Michael S. Tsirkin
Quoted 48 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG +                               +

           |                               |
           |                               |
           |                               |
           |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

               |                                                 |
               |                                                 |
               |                                                 |
               |                                                 |
        +------v------+                                   +------v------+
        |             |                                   |             |
        |    HOST1    |                                   |    HOST2    |
        |             |                                   |             |
        +------^------+                                   +------^------+
               |                                                 |
               |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just virtqueues + the exiting vhost interfaces?

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
         +----------------+  +-----------+  +------------+  +----------+
         |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
         +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                 |                 |              |              |
                 |                 |              |              |
                 |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
              |               |                    |                  |
              |               |                    |                  |
              |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?

Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 31 lines
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

This was initially proposed here [1]

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c  

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

I guess that clears up at least some of my questions from above...

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Ok.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I see.

Quoted 4 lines

    all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)

Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
Quoted 40 lines
Quoted 33 lines
Quoted 15 lines
Quoted 4 lines

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

Quoted 7 lines

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 7 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 7 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 3 lines

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 8 lines
Quoted 3 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).

Does that make any sense?

Quoted 62 lines
Quoted 46 lines
Quoted 46 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

Quoted 7 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

Quoted 7 lines
Quoted 5 lines

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.

Quoted 29 lines

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 24 lines
Quoted 24 lines
Quoted 4 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 1 line

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Quoted 6 lines
Quoted 2 lines

Got it. My initial design was based on my understanding of your comments [1].

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

I believe that we indeed need something bus-like to be able to support a variety of devices.

Quoted 7 lines
Quoted 4 lines

I'll try to create something based on your proposed design here.

Sure, but for coding, we'd better wait for other's opinion here.

Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.

#34 lore raw

Hi,

Quoted 83 lines · Cornelia Huck

On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:

[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]

Quoted 66 lines · Kishon Vijay Abraham I
Quoted 65 lines

Hi Jason,

Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 54 lines · Jason Wang
Quoted 53 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 50 lines · Jason Wang
Quoted 49 lines · Michael S. Tsirkin
Quoted 48 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG +                               +

            |                               |
            |                               |
            |                               |
            |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

                |                                                 |
                |                                                 |
                |                                                 |
                |                                                 |
         +------v------+                                   +------v------+
         |             |                                   |             |
         |    HOST1    |                                   |    HOST2    |
         |             |                                   |             |
         +------^------+                                   +------^------+
                |                                                 |
                |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+
Quoted 3 lines

First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just

Right, vhost_rpmsg is 'device' and virtio_rpmsg is 'driver'.

Quoted 1 line

virtqueues + the exiting vhost interfaces?

It's implemented to provide the full 'device' functionality.

Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
          +----------------+  +-----------+  +------------+  +----------+
          |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
          +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                  |                 |              |              |
                  |                 |              |              |
                  |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
               |               |                    |                  |
               |               |                    |                  |
               |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 3 lines

So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?

Right, the upper half should provide the functionality.
The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.

The top half should be transparent to what type of device is actually using it.

Quoted 147 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 32 lines
Quoted 14 lines
Quoted 11 lines
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

This was initially proposed here [1]

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

I guess that clears up at least some of my questions from above...

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Ok.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I see.

Quoted 4 lines

    all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)

Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
Quoted 40 lines
Quoted 33 lines
Quoted 15 lines
Quoted 4 lines

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

Quoted 7 lines

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 7 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 7 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 3 lines

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 8 lines
Quoted 3 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

Quoted 5 lines

When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).

I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication; setup_vq() in virtio_pci_legacy.c).

"Virtio Over NTB" should anyways be a new transport.

Quoted 2 lines

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

Quoted 109 lines
Quoted 63 lines
Quoted 47 lines
Quoted 47 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

Quoted 7 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

Quoted 7 lines
Quoted 5 lines

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.

Quoted 33 lines

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 26 lines
Quoted 16 lines
Quoted 4 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 8 lines
Quoted 1 line

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

Quoted 5 lines
Quoted 2 lines

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

I believe that we indeed need something bus-like to be able to support a variety of devices.

I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.

[1] -> Page 13 in https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

I'll try to create something based on your proposed design here.

Sure, but for coding, we'd better wait for other's opinion here.

Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.

I think your understanding is correct! Thanks for your inputs.

Thanks
Kishon

#35 lore raw
Quoted 278 lines · Kishon Vijay Abraham I

Hi,

Quoted 85 lines · Cornelia Huck

On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:

[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]

Quoted 64 lines · Kishon Vijay Abraham I
Quoted 63 lines

Hi Jason,

Quoted 60 lines · Jason Wang
Quoted 59 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 56 lines · Jason Wang
Quoted 55 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 52 lines · Jason Wang
Quoted 51 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 48 lines · Jason Wang
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for
         rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication
         between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

       VHOST RPMSG VIRTIO RPMSG
+                               +
|                               |
|                               |
|                               |
|                               |
+-----v------+ +------v-------+
|   Linux    |                 | Linux    |
|  Endpoint  |                 | Root Complex |
| <----------------->              |
|            | |              |
|    SOC1    |                 | SOC2     |
+------------+ +--------------+

UseCase 2:

           VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
         | | |             |
         |    HOST1 |                                   | HOST2    |
         | | |             |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

| +------v------+ +------v------+  |
|  | | |             |  |
|  |     EP |                                   | EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

|  | <-----------------------------------> |  |
|  | | |             |  |
|  | | |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
| +-------------+ +-------------+  |
+---------------------------------------------------------------------+

Quoted 3 lines

First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just

Right, vhost_rpmsg is 'device' and virtio_rpmsg is 'driver'.

Quoted 1 line

virtqueues + the exiting vhost interfaces?

It's implemented to provide the full 'device' functionality.

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines

Software Layering:

The high-level SW layering should look something like below. This series
adds support only for RPMSG VHOST, however something similar should be
done for net and scsi. With that any vhost device (PCI, NTB, Platform
device, user) can use any of the vhost client driver.

          +----------------+ +-----------+  +------------+
+----------+
          |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
          +-------^--------+ +-----^-----+  +-----^------+
+----^-----+
                  | |              |              |
                  | |              |              |
                  | |              |              |
+-----------v-----------------v--------------v--------------v----------+

|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

               | |                    |                  |
               | |                    |                  |
               | |                    |                  |
+--------v-------+  +----v------+ +----------v----------+
+----v-----+
|  PCI EPF VHOST |  | NTB VHOST | |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+ +---------------------+
+----------+

Quoted 3 lines

So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?

Right, the upper half should provide the functionality.
The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.

The top half should be transparent to what type of device is actually using it.

Quoted 141 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 32 lines
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

This was initially proposed here [1]

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

      So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

I guess that clears up at least some of my questions from above...

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Ok.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 4 lines

      all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)

Quoted 60 lines
Quoted 52 lines
Quoted 43 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

Quoted 7 lines

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 7 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 7 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 3 lines

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 9 lines

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 3 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

Quoted 7 lines
Quoted 5 lines

When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).

I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication; setup_vq() in virtio_pci_legacy.c).

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

Quoted 6 lines

"Virtio Over NTB" should anyways be a new transport.

Quoted 2 lines

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?

Thanks

Quoted 128 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

Quoted 7 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

Quoted 7 lines
Quoted 5 lines

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.

Quoted 30 lines

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 23 lines
Quoted 13 lines
Quoted 3 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 8 lines
Quoted 1 line

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

Quoted 5 lines
Quoted 2 lines

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

I believe that we indeed need something bus-like to be able to support a variety of devices.

I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.

[1] -> Page 13 in https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

I'll try to create something based on your proposed design here.

Sure, but for coding, we'd better wait for other's opinion here.

Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.

I think your understanding is correct! Thanks for your inputs.

Thanks
Kishon

#36 lore raw
Quoted 99 lines · On Tue, 1 Sep 2020 16:50:03 +0800 Jason Wang <[email protected]> wrote:
Quoted 98 lines · Kishon Vijay Abraham I

Hi,

Quoted 95 lines · Cornelia Huck

On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:

[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]

Quoted 86 lines · Kishon Vijay Abraham I
Quoted 85 lines

Hi Jason,

Quoted 82 lines · Jason Wang
Quoted 81 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 78 lines · Jason Wang
Quoted 77 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 74 lines · Jason Wang
Quoted 73 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 70 lines · Jason Wang
Quoted 69 lines · Michael S. Tsirkin

On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay
Abraham I wrote:
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg
communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver
(uses vhost) for
         rpmsg communication between two SoCs connected to
each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg
communication
         between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

       VHOST RPMSG VIRTIO RPMSG
+                               +
|                               |
|                               |
|                               |
|                               |
+-----v------+ +------v-------+
|   Linux    |                 | Linux    |
|  Endpoint  |                 | Root Complex |
| <----------------->              |
|            | |              |
|    SOC1    |                 | SOC2     |
+------------+ +--------------+

UseCase 2:

           VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
         | | |             |
         |    HOST1 |                                   |
HOST2    |
         | | |             |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+

| +------v------+ +------v------+  |
|  | | |             |  |
|  |     EP |                                   | EP
|  |
|  | CONTROLLER1 |                                   |
CONTROLLER2 |  |
|  | <-----------------------------------> |  |
|  | | |             |  |
|  | | |             |  |
|  |             |  SoC With Multiple EP Instances
|             |  |
|  |             |  (Configured using NTB Function)
|             |  |
| +-------------+ +-------------+  |
+---------------------------------------------------------------------+

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just

Right, vhost_rpmsg is 'device' and virtio_rpmsg is 'driver'.

Quoted 1 line

virtqueues + the exiting vhost interfaces?

It's implemented to provide the full 'device' functionality.

Ok.

Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines

Software Layering:

The high-level SW layering should look something like
below. This series
adds support only for RPMSG VHOST, however something
similar should be
done for net and scsi. With that any vhost device (PCI,
NTB, Platform
device, user) can use any of the vhost client driver.

          +----------------+ +-----------+  +------------+
+----------+
          |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST
|  |    X     |
          +-------^--------+ +-----^-----+  +-----^------+
+----^-----+
                  | |              |              |
                  | |              |              |
                  | |              |              |
+-----------v-----------------v--------------v--------------v----------+

|                            VHOST
CORE                                |
+--------^---------------^--------------------^------------------^-----+

               | |                    |                  |
               | |                    |                  |
               | |                    |                  |
+--------v-------+  +----v------+ +----------v----------+
+----v-----+
|  PCI EPF VHOST |  | NTB VHOST | |PLATFORM DEVICE VHOST|
|    X     |
+----------------+  +-----------+ +---------------------+
+----------+

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?

Right, the upper half should provide the functionality. The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.

Ok. In that respect, the SW interface is sufficiently device-like, I guess.

Quoted 184 lines
Quoted 176 lines

The top half should be transparent to what type of device is actually using it.

Quoted 166 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 48 lines
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This was initially proposed here [1]

[1] ->
https://lore.kernel.org/r/[email protected]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I find this very interesting. A huge patchset so will take
a bit
to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 4 lines
Quoted 4 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the
work is
duplicated with vDPA (e.g the epf transport or vhost bus).

Quoted 8 lines
Quoted 3 lines

This is about connecting two different HW systems both running Linux and
doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the
hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between
two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe
RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and
any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint
function driver and hence any SoC that supports configurable PCIe endpoint can
use virtio-vhost communication

Quoted 4 lines
# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c  

Thanks for those backgrounds.

Quoted 19 lines
Quoted 19 lines
Quoted 6 lines

      So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable,
one
acting as guest and one as host. So one system will provide virtio

Quoted 8 lines

functionality reserving memory for virtqueues and the other provides vhost
functionality providing a way to access the virtqueues in virtio memory.
One is
source and the other is sink and there is no intermediate entity. (vhost was
probably intermediate entity in virtualization?)

(Not a native English speaker) but "vhost" could introduce some confusion for
me since it was use for implementing virtio backend for userspace drivers. I
guess "vringh" could be better.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Initially I had named this vringh but later decided to choose vhost instead of
vringh. vhost is still a virtio backend (not necessarily userspace) though it
now resides in an entirely different system. Whatever virtio is for a frontend
system, vhost can be that for a backend system. vring can be for accessing
virtqueue and can be used either in frontend or backend.

I guess that clears up at least some of my questions from above...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Ok.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg
driver
or vhost net driver is not provided.

The HW connection looks something like
https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I see.

Quoted 4 lines

      all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)

Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

Quoted 7 lines

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 7 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 7 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 3 lines

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 8 lines
Quoted 3 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

Quoted 5 lines

When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).

Quoted 6 lines

I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication; setup_vq() in virtio_pci_legacy.c).

Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

It seems I really need to read up on vDPA more... do you have a pointer for diving into this alternatives aspect?

Quoted 15 lines
Quoted 5 lines

"Virtio Over NTB" should anyways be a new transport.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

Quoted 2 lines

A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?

Can we maybe implement an alternative bus as well that would allow us to support different virtio device implementations (in addition to the vhost bus + userspace combination)?

Quoted 120 lines

Thanks

Quoted 116 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

Quoted 7 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

Quoted 7 lines
Quoted 5 lines

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.

Quoted 30 lines

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 23 lines
Quoted 13 lines
Quoted 3 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 8 lines
Quoted 1 line

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

Quoted 5 lines
Quoted 2 lines

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

I believe that we indeed need something bus-like to be able to support a variety of devices.

I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.

[1] -> Page 13 in
https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf

So, I guess it's a trade-off: do we expect a variety of device types, or more of a variety of devices needing different adapters?

#37 lore raw
Quoted 5 lines · Cornelia Huck
Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

It seems I really need to read up on vDPA more... do you have a pointer for diving into this alternatives aspect?

See vpda_config_ops in include/linux/vdpa.h

Especially this part:

    int (*set_vq_address)(struct vdpa_device *vdev, u16 idx, u64 desc_area, u64 driver_area, u64 device_area);

This means for the devices (e.g endpoint device) that is hard to implement virtio-pci layout, it can use any other register layout or vendor specific way to configure the virtqueue.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

"Virtio Over NTB" should anyways be a new transport.

Quoted 1 line

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

Quoted 5 lines
Quoted 2 lines

A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?

Can we maybe implement an alternative bus as well that would allow us to support different virtio device implementations (in addition to the vhost bus + userspace combination)?

That should be fine, but I'm not quite sure that implementing the device in kerne (kthread) is the good approach.

Thanks

Quoted 1 line
#38 lore raw
In reply to Jason Wang

Hi Jason,

Quoted 338 lines · Jason Wang
Quoted 328 lines · Kishon Vijay Abraham I

Hi,

Quoted 95 lines · Cornelia Huck

On Thu, 9 Jul 2020 14:26:53 +0800
Jason Wang <[email protected]> wrote:

[Let me note right at the beginning that I first noted this while listening to Kishon's talk at LPC on Wednesday. I might be very confused about the background here, so let me apologize beforehand for any confusion I might spread.]

Quoted 70 lines · Kishon Vijay Abraham I
Quoted 69 lines

Hi Jason,

Quoted 66 lines · Jason Wang
Quoted 65 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 62 lines · Jason Wang
Quoted 61 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 58 lines · Jason Wang
Quoted 57 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 54 lines · Jason Wang
Quoted 53 lines · Michael S. Tsirkin

On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay
Abraham I wrote:
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg
communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver
(uses vhost) for
         rpmsg communication between two SoCs connected to
each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg
communication
         between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

       VHOST RPMSG VIRTIO RPMSG
+                               +
|                               |
|                               |
|                               |
|                               |
+-----v------+ +------v-------+
|   Linux    |                 | Linux    |
|  Endpoint  |                 | Root Complex |
| <----------------->              |
|            | |              |
|    SOC1    |                 | SOC2     |
+------------+ +--------------+

UseCase 2:

           VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |
+------v------+ +------v------+
         | | |             |
         |    HOST1 |                                   |
HOST2    |
         | | |             |
+------^------+ +------^------+
| |
| |
+---------------------------------------------------------------------+

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

| +------v------+ +------v------+  |
|  | | |             |  |
|  |     EP |                                   | EP
|  |
|  | CONTROLLER1 |                                   |
CONTROLLER2 |  |
|  | <-----------------------------------> |  |
|  | | |             |  |

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

|  | | |             |  |
|  |             |  SoC With Multiple EP Instances
|             |  |
|  |             |  (Configured using NTB Function)
|             |  |
| +-------------+ +-------------+  |
+---------------------------------------------------------------------+

Quoted 3 lines

First of all, to clarify the terminology:
Is "vhost rpmsg" acting as what the virtio standard calls the 'device', and "virtio rpmsg" as the 'driver'? Or is the "vhost" part mostly just

Right, vhost_rpmsg is 'device' and virtio_rpmsg is 'driver'.

Quoted 1 line

virtqueues + the exiting vhost interfaces?

It's implemented to provide the full 'device' functionality.

Quoted 38 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

Software Layering:

The high-level SW layering should look something like
below. This series
adds support only for RPMSG VHOST, however something
similar should be
done for net and scsi. With that any vhost device (PCI,
NTB, Platform
device, user) can use any of the vhost client driver.

          +----------------+ +-----------+  +------------+
+----------+
          |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST
|  |    X     |
          +-------^--------+ +-----^-----+  +-----^------+
+----^-----+
                  | |              |              |
                  | |              |              |
                  | |              |              |
+-----------v-----------------v--------------v--------------v----------+

|                            VHOST
CORE                                |
+--------^---------------^--------------------^------------------^-----+

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

               | |                    |                  |
               | |                    |                  |
               | |                    |                  |
+--------v-------+  +----v------+ +----------v----------+
+----v-----+
|  PCI EPF VHOST |  | NTB VHOST | |PLATFORM DEVICE VHOST|
|    X     |
+----------------+  +-----------+ +---------------------+
+----------+

Quoted 3 lines

So, the upper half is basically various functionality types, e.g. a net device. What is the lower half, a hardware interface? Would it be equivalent to e.g. a normal PCI device?

Right, the upper half should provide the functionality. The bottom layer could be a HW interface (like PCIe device or NTB device) or it could be a SW interface (for accessing virtio ring in userspace) that could be used by Hypervisor.

The top half should be transparent to what type of device is actually using it.

Quoted 166 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 48 lines
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This was initially proposed here [1]

[1] ->
https://lore.kernel.org/r/[email protected]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I find this very interesting. A huge patchset so will take
a bit
to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 4 lines
Quoted 4 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the
work is
duplicated with vDPA (e.g the epf transport or vhost bus).

Quoted 8 lines
Quoted 3 lines

This is about connecting two different HW systems both running Linux and
doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the
hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between
two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe
RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and
any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint
function driver and hence any SoC that supports configurable PCIe endpoint can
use virtio-vhost communication

Quoted 4 lines
# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 19 lines
Quoted 19 lines
Quoted 6 lines

      So there is no guest or host as in virtualization but two entirely different systems connected via PCIe cable,
one
acting as guest and one as host. So one system will provide virtio

Quoted 8 lines

functionality reserving memory for virtqueues and the other provides vhost
functionality providing a way to access the virtqueues in virtio memory.
One is
source and the other is sink and there is no intermediate entity. (vhost was
probably intermediate entity in virtualization?)

(Not a native English speaker) but "vhost" could introduce some confusion for
me since it was use for implementing virtio backend for userspace drivers. I
guess "vringh" could be better.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Initially I had named this vringh but later decided to choose vhost instead of
vringh. vhost is still a virtio backend (not necessarily userspace) though it
now resides in an entirely different system. Whatever virtio is for a frontend
system, vhost can be that for a backend system. vring can be for accessing
virtqueue and can be used either in frontend or backend.

I guess that clears up at least some of my questions from above...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Ok.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg
driver
or vhost net driver is not provided.

The HW connection looks something like
https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I see.

Quoted 4 lines

      all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

This setup looks really interesting (sometimes, it's really hard to imagine this in the abstract.)

Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 39 lines
Quoted 32 lines
Quoted 15 lines
Quoted 4 lines

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or identical.

Having the same bus can also create sequencing issues.

If you look at virtio_dev_probe() of virtio_bus

Quoted 7 lines

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 7 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 7 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Quoted 3 lines

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost 2) Using this X for both virtio device and vhost(vringh) device 3) local virtio driver will negotiate with virtio device with feature set Y 4) remote virtio driver will negotiate with vringh device with feature set Z 5) mediate between feature Y and feature Z since both Y and Z are a subset of X

Quoted 8 lines
Quoted 3 lines

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Yes, and I think some management API is needed even in the design of your "Software Layering". In that figure, rpmsg vhost need some pre-set or hard-coded features.

Quoted 5 lines

When I saw the plumbers talk, my first idea was "this needs to be a new transport". You have some hard-coded or pre-configured features, and then features are negotiated via a transport-specific means in the usual way. There's basically an extra/extended layer for this (and status, and whatever).

Quoted 6 lines

I think for PCIe root complex to PCIe endpoint communication it's still "Virtio Over PCI Bus", though existing layout cannot be used in this context (find virtio capability will fail for modern interface and loading queue status immediately after writing queue number is not possible for root complex to endpoint communication; setup_vq() in virtio_pci_legacy.c).

Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
	/* Virtqueue ops */
	int (*set_vq_address)(struct vdpa_device *vdev,
			      u16 idx, u64 desc_area, u64 driver_area,
			      u64 device_area);
	void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num);
	void (*kick_vq)(struct vdpa_device *vdev, u16 idx);
	void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx,
			  struct vdpa_callback *cb);
	void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready);
	bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
			    const struct vdpa_vq_state *state);
	int (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
			    struct vdpa_vq_state *state);
	struct vdpa_notification_area
	(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
	/* vq irq is not expected to be changed once DRIVER_OK is set */
	int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

	/* Device ops */
	u32 (*get_vq_align)(struct vdpa_device *vdev);
	u64 (*get_features)(struct vdpa_device *vdev);
	int (*set_features)(struct vdpa_device *vdev, u64 features);
	void (*set_config_cb)(struct vdpa_device *vdev,
			      struct vdpa_callback *cb);
	u16 (*get_vq_num_max)(struct vdpa_device *vdev);
	u32 (*get_device_id)(struct vdpa_device *vdev);
	u32 (*get_vendor_id)(struct vdpa_device *vdev);
	u8 (*get_status)(struct vdpa_device *vdev);
	void (*set_status)(struct vdpa_device *vdev, u8 status);
	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
			   void *buf, unsigned int len);
	void (*set_config)(struct vdpa_device *vdev, unsigned int offset,
			   const void *buf, unsigned int len);
	u32 (*get_generation)(struct vdpa_device *vdev);

	/* DMA ops */
	int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb);
	int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size,
		       u64 pa, u32 perm);
	int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

	/* Free device resources */
	void (*free)(struct vdpa_device *vdev);
};

+struct vhost_config_ops {
+ int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs,

+			  unsigned int num_bufs, struct vhost_virtqueue *vqs[],
+			  vhost_vq_callback_t *callbacks[],
+			  const char * const names[]);

+ void (*del_vqs)(struct vhost_dev *vdev);
+ int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+ int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+ int (*set_features)(struct vhost_dev *vdev, u64 device_features);
+ int (*set_status)(struct vhost_dev *vdev, u8 status);
+ u8 (*get_status)(struct vhost_dev *vdev);
+};
+
struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

I think vdpa always works with client drivers in userspace and providing userspace address for vring.

Quoted 15 lines
Quoted 5 lines

"Virtio Over NTB" should anyways be a new transport.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

Quoted 2 lines

A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?

The vhost bus approach doesn't provide any restriction in where the virto backend device should be created. This series creates two types of virtio backend device (one for PCIe endpoint and the other for NTB) and both these devices are created in kernel.

Thanks
Kishon

Quoted 138 lines

Thanks

Quoted 133 lines
Quoted 103 lines
Quoted 60 lines
Quoted 45 lines
Quoted 45 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

Quoted 7 lines

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

I think this will work however there is an addtional copy between vringh queue and virtqueue,

Quoted 7 lines
Quoted 5 lines

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 11 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Thinking how this would look for NTB virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Right.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Good point, for non-peer device like storage. There's probably no need for it to be registered on the virtio bus and it might be better to behave as you proposed.

I might be missing something; but if you expose something as a block device, it should have something it can access with block reads/writes, shouldn't it? Of course, that can be a variety of things.

Quoted 30 lines

Just to make sure I understand the design, how is VHOST SCSI expected to work in your proposal, does it have a device for file as a backend?

Quoted 23 lines
Quoted 13 lines
Quoted 3 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 8 lines
Quoted 1 line

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

Quoted 5 lines
Quoted 2 lines

Yes, but that's just for a networking device. If we want something more generic, it may require more thought (bus etc).

I believe that we indeed need something bus-like to be able to support a variety of devices.

I think we could still have adapter layers for different types of devices ([1]) and use existing virtio bus for both front end and back end. Using bus-like will however simplify adding support for new types of devices and adding adapters for devices will be slightly more complex.

[1] -> Page 13 in
https://linuxplumbersconf.org/event/7/contributions/849/attachments/642/1175/Virtio_for_PCIe_RC_EP_NTB.pdf

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

I'll try to create something based on your proposed design here.

Sure, but for coding, we'd better wait for other's opinion here.

Please tell me if my thoughts above make any sense... I have just started looking at that, so I might be completely off.

I think your understanding is correct! Thanks for your inputs.

Thanks
Kishon

#39 lore raw

Hi Kishon:

Quoted 89 lines · Kishon Vijay Abraham I
Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
	/* Virtqueue ops */
	int (*set_vq_address)(struct vdpa_device *vdev,
			      u16 idx, u64 desc_area, u64 driver_area,
			      u64 device_area);
	void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num);
	void (*kick_vq)(struct vdpa_device *vdev, u16 idx);
	void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx,
			  struct vdpa_callback *cb);
	void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready);
	bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
			    const struct vdpa_vq_state *state);
	int (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
			    struct vdpa_vq_state *state);
	struct vdpa_notification_area
	(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
	/* vq irq is not expected to be changed once DRIVER_OK is set */
	int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

	/* Device ops */
	u32 (*get_vq_align)(struct vdpa_device *vdev);
	u64 (*get_features)(struct vdpa_device *vdev);
	int (*set_features)(struct vdpa_device *vdev, u64 features);
	void (*set_config_cb)(struct vdpa_device *vdev,
			      struct vdpa_callback *cb);
	u16 (*get_vq_num_max)(struct vdpa_device *vdev);
	u32 (*get_device_id)(struct vdpa_device *vdev);
	u32 (*get_vendor_id)(struct vdpa_device *vdev);
	u8 (*get_status)(struct vdpa_device *vdev);
	void (*set_status)(struct vdpa_device *vdev, u8 status);
	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
			   void *buf, unsigned int len);
	void (*set_config)(struct vdpa_device *vdev, unsigned int offset,
			   const void *buf, unsigned int len);
	u32 (*get_generation)(struct vdpa_device *vdev);

	/* DMA ops */
	int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb);
	int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size,
		       u64 pa, u32 perm);
	int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

	/* Free device resources */
	void (*free)(struct vdpa_device *vdev);
};
(quoted diff) +13 -0
+struct vhost_config_ops {
+	int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs,
+			  unsigned int num_bufs, struct vhost_virtqueue *vqs[],
+			  vhost_vq_callback_t *callbacks[],
+			  const char * const names[]);
+	void (*del_vqs)(struct vhost_dev *vdev);
+	int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+	int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+	int (*set_features)(struct vhost_dev *vdev, u64 device_features);
+	int (*set_status)(struct vhost_dev *vdev, u8 status);
+	u8 (*get_status)(struct vhost_dev *vdev);
+};
+

struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

Quoted 2 lines

I think vdpa always works with client drivers in userspace and providing userspace address for vring.

Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:

My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:

+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev, +                       u32 command)
+{
+    struct virtio_pci_epf *pci_epf;
+    void __iomem *ioaddr;
+    ktime_t timeout;
+    bool timedout;
+    int ret = 0;
+    u8 status;
+
+    pci_epf = to_virtio_pci_epf(vp_dev);
+    ioaddr = vp_dev->ioaddr;
+
+    mutex_lock(&pci_epf->lock);
+    writeb(command, ioaddr + HOST_CMD);
+    timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+    while (1) {
+        timedout = ktime_after(ktime_get(), timeout);
+        status = readb(ioaddr + HOST_CMD_STATUS);
+

Several questions:

- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed.  If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.
- You have you vendor specific layout (according to virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead
- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

"Virtio Over NTB" should anyways be a new transport.

Quoted 1 line

Does that make any sense?

yeah, in the approach I used the initial features are hard-coded in vhost-rpmsg (inherent to the rpmsg) but when we have to use adapter layer (vhost only for accessing virtio ring and use virtio drivers on both front end and backend), based on the functionality (e.g, rpmsg), the vhost should be configured with features (to be presented to the virtio) and that's why additional layer or APIs will be required.

Quoted 6 lines
Quoted 2 lines

A question here, if we go with vhost bus approach, does it mean the virtio device can only be implemented in EP's userspace?

The vhost bus approach doesn't provide any restriction in where the virto backend device should be created. This series creates two types of virtio backend device (one for PCIe endpoint and the other for NTB) and both these devices are created in kernel.

Ok.

Thanks

Quoted 4 lines

Thanks
Kishon

#40 lore raw

Hi Jason,

Quoted 94 lines · Jason Wang

Hi Kishon:

Quoted 91 lines · Kishon Vijay Abraham I
Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
    /* Virtqueue ops */
    int (*set_vq_address)(struct vdpa_device *vdev,
                  u16 idx, u64 desc_area, u64 driver_area, u64 device_area);
    void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num); void (*kick_vq)(struct vdpa_device *vdev, u16 idx);
    void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx, struct vdpa_callback *cb);
    void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready); bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx); int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, const struct vdpa_vq_state *state);
    int (*get_vq_state)(struct vdpa_device *vdev, u16 idx, struct vdpa_vq_state *state);
    struct vdpa_notification_area (*get_vq_notification)(struct vdpa_device *vdev, u16 idx); /* vq irq is not expected to be changed once DRIVER_OK is set */ int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

    /* Device ops */
    u32 (*get_vq_align)(struct vdpa_device *vdev);
    u64 (*get_features)(struct vdpa_device *vdev);
    int (*set_features)(struct vdpa_device *vdev, u64 features); void (*set_config_cb)(struct vdpa_device *vdev,
                  struct vdpa_callback *cb);
    u16 (*get_vq_num_max)(struct vdpa_device *vdev);
    u32 (*get_device_id)(struct vdpa_device *vdev);
    u32 (*get_vendor_id)(struct vdpa_device *vdev);
    u8 (*get_status)(struct vdpa_device *vdev);
    void (*set_status)(struct vdpa_device *vdev, u8 status); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len);
    void (*set_config)(struct vdpa_device *vdev, unsigned int offset, const void *buf, unsigned int len);
    u32 (*get_generation)(struct vdpa_device *vdev);

    /* DMA ops */
    int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb); int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size, u64 pa, u32 perm);
    int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

    /* Free device resources */
    void (*free)(struct vdpa_device *vdev);
};

+struct vhost_config_ops {
+    int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs, +              unsigned int num_bufs, struct vhost_virtqueue *vqs[], +              vhost_vq_callback_t *callbacks[],
+              const char * const names[]);
+    void (*del_vqs)(struct vhost_dev *vdev);
+    int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+    int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+    int (*set_features)(struct vhost_dev *vdev, u64 device_features); +    int (*set_status)(struct vhost_dev *vdev, u8 status); +    u8 (*get_status)(struct vhost_dev *vdev);

(quoted diff) +2 -0
+};
+

struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

Quoted 7 lines
Quoted 2 lines

I think vdpa always works with client drivers in userspace and providing userspace address for vring.

Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:

Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate vdpa_config_ops and invoke vdpa_register_device().

Quoted 4 lines

My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:

Even when we use vDPA, we have to use some sort of virtio_pci_epf_send_command() to communicate with virtio backend right?

Right, the layout is slightly different from the standard layout.

This is the layout

struct epf_vhost_reg_queue {
        u8 cmd;
        u8 cmd_status;
        u16 status;
        u16 num_buffers;
        u16 msix_vector;
        u64 queue_addr;
} __packed;
struct epf_vhost_reg {
        u64 host_features;
        u64 guest_features;
        u16 msix_config;
        u16 num_queues;
        u8 device_status;
        u8 config_generation;
        u32 isr;
        u8 cmd;
        u8 cmd_status;
        struct epf_vhost_reg_queue vq[MAX_VQS];
} __packed;
Quoted 23 lines
(quoted diff) +20 -0
+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev,
+                       u32 command)
+{
+    struct virtio_pci_epf *pci_epf;
+    void __iomem *ioaddr;
+    ktime_t timeout;
+    bool timedout;
+    int ret = 0;
+    u8 status;
+
+    pci_epf = to_virtio_pci_epf(vp_dev);
+    ioaddr = vp_dev->ioaddr;
+
+    mutex_lock(&pci_epf->lock);
+    writeb(command, ioaddr + HOST_CMD);
+    timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+    while (1) {
+        timedout = ktime_after(ktime_get(), timeout);
+        status = readb(ioaddr + HOST_CMD_STATUS);
+
Quoted 4 lines

Several questions:

- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed.

The HOST_CMD (commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value in HOST_CMD. So HOST_CMD is less likely an issue.

A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in HOST_CMD_STATUS. After it expires, HOST_CMD_STATUS_NONE is written to HOST_CMD_STATUS. There could be case where EP updates HOST_CMD_STATUS after RC writes HOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.

Quoted 2 lines

If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.

Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?

Quoted 3 lines

- You have you vendor specific layout (according to virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead

Okay, with vDPA, we are free to define our own layouts.

Quoted 3 lines

- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).

I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.

For 2, Isn't vhost-vdpa supposed to run on virtio backend?

From a high level, I think I should be able to use vDPA for virtio_pci_epf.c. Would you also suggest using vDPA for ntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).

Thanks
Kishon

#41 lore raw
Quoted 103 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 94 lines · Jason Wang

Hi Kishon:

Quoted 91 lines · Kishon Vijay Abraham I
Quoted 3 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
    /* Virtqueue ops */
    int (*set_vq_address)(struct vdpa_device *vdev,
                  u16 idx, u64 desc_area, u64 driver_area, u64 device_area);
    void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num); void (*kick_vq)(struct vdpa_device *vdev, u16 idx);
    void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx, struct vdpa_callback *cb);
    void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready); bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx); int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, const struct vdpa_vq_state *state);
    int (*get_vq_state)(struct vdpa_device *vdev, u16 idx, struct vdpa_vq_state *state);
    struct vdpa_notification_area (*get_vq_notification)(struct vdpa_device *vdev, u16 idx); /* vq irq is not expected to be changed once DRIVER_OK is set */ int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

    /* Device ops */
    u32 (*get_vq_align)(struct vdpa_device *vdev);
    u64 (*get_features)(struct vdpa_device *vdev);
    int (*set_features)(struct vdpa_device *vdev, u64 features); void (*set_config_cb)(struct vdpa_device *vdev,
                  struct vdpa_callback *cb);
    u16 (*get_vq_num_max)(struct vdpa_device *vdev);
    u32 (*get_device_id)(struct vdpa_device *vdev);
    u32 (*get_vendor_id)(struct vdpa_device *vdev);
    u8 (*get_status)(struct vdpa_device *vdev);
    void (*set_status)(struct vdpa_device *vdev, u8 status); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len);
    void (*set_config)(struct vdpa_device *vdev, unsigned int offset, const void *buf, unsigned int len);
    u32 (*get_generation)(struct vdpa_device *vdev);

    /* DMA ops */
    int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb); int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size, u64 pa, u32 perm);
    int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

    /* Free device resources */
    void (*free)(struct vdpa_device *vdev);
};

+struct vhost_config_ops {
+    int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs, +              unsigned int num_bufs, struct vhost_virtqueue *vqs[], +              vhost_vq_callback_t *callbacks[],
+              const char * const names[]);
+    void (*del_vqs)(struct vhost_dev *vdev);
+    int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+    int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+    int (*set_features)(struct vhost_dev *vdev, u64 device_features); +    int (*set_status)(struct vhost_dev *vdev, u8 status); +    u8 (*get_status)(struct vhost_dev *vdev);

(quoted diff) +2 -0
+};
+

struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

Quoted 6 lines
Quoted 2 lines

I think vdpa always works with client drivers in userspace and providing userspace address for vring.

Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:

Quoted 3 lines

Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate vdpa_config_ops and invoke vdpa_register_device().

Yes.

Quoted 5 lines
Quoted 3 lines

My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:

Even when we use vDPA, we have to use some sort of virtio_pci_epf_send_command() to communicate with virtio backend right?

Right.

Quoted 11 lines

Right, the layout is slightly different from the standard layout.

This is the layout

struct epf_vhost_reg_queue {
         u8 cmd;
         u8 cmd_status;
         u16 status;
         u16 num_buffers;
         u16 msix_vector;
         u64 queue_addr;

What's the meaning of queue_addr here?

Does not mean the device expects a contiguous memory for avail/desc/used ring?

Quoted 36 lines

} __packed;

struct epf_vhost_reg {
         u64 host_features;
         u64 guest_features;
         u16 msix_config;
         u16 num_queues;
         u8 device_status;
         u8 config_generation;
         u32 isr;
         u8 cmd;
         u8 cmd_status;
         struct epf_vhost_reg_queue vq[MAX_VQS];
} __packed;
Quoted 22 lines
(quoted diff) +20 -0
+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev,
+                       u32 command)
+{
+    struct virtio_pci_epf *pci_epf;
+    void __iomem *ioaddr;
+    ktime_t timeout;
+    bool timedout;
+    int ret = 0;
+    u8 status;
+
+    pci_epf = to_virtio_pci_epf(vp_dev);
+    ioaddr = vp_dev->ioaddr;
+
+    mutex_lock(&pci_epf->lock);
+    writeb(command, ioaddr + HOST_CMD);
+    timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+    while (1) {
+        timedout = ktime_after(ktime_get(), timeout);
+        status = readb(ioaddr + HOST_CMD_STATUS);
+
Quoted 7 lines
Quoted 4 lines

Several questions:

- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed.

The HOST_CMD (commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value in HOST_CMD. So HOST_CMD is less likely an issue.

Here's my understanding of the protocol:

1) RC write to HOST_CMD
2) RC wait for HOST_CMD_STATUS to be HOST_CMD_STATUS_OKAY

It looks to me what EP should do is

1) EP reset HOST_CMD after reading new command

And it looks to me EP should also reset HOST_CMD_STATUS here?

(I thought there should be patch to handle stuffs like this but I didn't find it in this series)

Quoted 11 lines

A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in HOST_CMD_STATUS. After it expires, HOST_CMD_STATUS_NONE is written to HOST_CMD_STATUS. There could be case where EP updates HOST_CMD_STATUS after RC writes HOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.

Quoted 2 lines

If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.

Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?

Yes.

Quoted 4 lines
Quoted 3 lines

- You have you vendor specific layout (according to virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead

Okay, with vDPA, we are free to define our own layouts.

Right, but vDPA have other requirements. E.g it requires the device have the ability to save/restore the state (e.g the last_avail_idx).

So it actually depends on what you want. If you don't care about userspace drivers and want to have a standard transport, you can still go virtio.

Quoted 6 lines
Quoted 3 lines

- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).

I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.

AFAIK you don't need to implement your own setup_vq and del_vq.

Quoted 2 lines

For 2, Isn't vhost-vdpa supposed to run on virtio backend?

Not currently, vDPA is a superset of virtio (e.g it support virtqueue state save/restore). This it should be possible in the future probably.

Quoted 5 lines

From a high level, I think I should be able to use vDPA for
virtio_pci_epf.c. Would you also suggest using vDPA for ntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).

I think it's your call. If you want

1) a well-defined standard virtio transport
2) willing to finalize d and maintain the spec
3) doesn't care about userspace drivers

You can go with virtio, otherwise vDPA.

Thanks

Quoted 4 lines

Thanks
Kishon

#42 lore raw

Hi Jason,

Quoted 141 lines · Jason Wang
Quoted 108 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 98 lines · Jason Wang

Hi Kishon:

Quoted 95 lines · Kishon Vijay Abraham I
Quoted 4 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if
the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
     /* Virtqueue ops */
     int (*set_vq_address)(struct vdpa_device *vdev,
                   u16 idx, u64 desc_area, u64 driver_area, u64 device_area);
     void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num); void (*kick_vq)(struct vdpa_device *vdev, u16 idx); void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx, struct vdpa_callback *cb);
     void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready);
     bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx); int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, const struct vdpa_vq_state *state);
     int (*get_vq_state)(struct vdpa_device *vdev, u16 idx, struct vdpa_vq_state *state);
     struct vdpa_notification_area (*get_vq_notification)(struct vdpa_device *vdev, u16 idx); /* vq irq is not expected to be changed once DRIVER_OK is set */ int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

     /* Device ops */
     u32 (*get_vq_align)(struct vdpa_device *vdev);
     u64 (*get_features)(struct vdpa_device *vdev);
     int (*set_features)(struct vdpa_device *vdev, u64 features); void (*set_config_cb)(struct vdpa_device *vdev,
                   struct vdpa_callback *cb);
     u16 (*get_vq_num_max)(struct vdpa_device *vdev);
     u32 (*get_device_id)(struct vdpa_device *vdev);
     u32 (*get_vendor_id)(struct vdpa_device *vdev);
     u8 (*get_status)(struct vdpa_device *vdev);
     void (*set_status)(struct vdpa_device *vdev, u8 status); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len);
     void (*set_config)(struct vdpa_device *vdev, unsigned int offset, const void *buf, unsigned int len);
     u32 (*get_generation)(struct vdpa_device *vdev);

     /* DMA ops */
     int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb *iotlb);
     int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size, u64 pa, u32 perm);
     int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

     /* Free device resources */
     void (*free)(struct vdpa_device *vdev);
};

+struct vhost_config_ops {
+    int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs, +              unsigned int num_bufs, struct vhost_virtqueue *vqs[], +              vhost_vq_callback_t *callbacks[],
+              const char * const names[]);
+    void (*del_vqs)(struct vhost_dev *vdev);
+    int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+    int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+    int (*set_features)(struct vhost_dev *vdev, u64 device_features); +    int (*set_status)(struct vhost_dev *vdev, u8 status); +    u8 (*get_status)(struct vhost_dev *vdev);

(quoted diff) +2 -0
+};
+

struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly
provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

Quoted 7 lines
Quoted 3 lines

I think vdpa always works with client drivers in userspace and providing
userspace address for vring.

Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:

Quoted 3 lines

Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate vdpa_config_ops and invoke vdpa_register_device().

Yes.

Quoted 5 lines
Quoted 3 lines

My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:

Even when we use vDPA, we have to use some sort of virtio_pci_epf_send_command() to communicate with virtio backend right?

Right.

Quoted 11 lines

Right, the layout is slightly different from the standard layout.

This is the layout
struct epf_vhost_reg_queue {
         u8 cmd;
         u8 cmd_status;
         u16 status;
         u16 num_buffers;
         u16 msix_vector;
         u64 queue_addr;

Quoted 1 line

What's the meaning of queue_addr here?

Using queue_addr, the virtio front-end communicates the address of the allocated memory for virtqueue to the virtio back-end.

Quoted 3 lines

Does not mean the device expects a contiguous memory for avail/desc/used ring?

It's contiguous memory. Isn't this similar to other virtio transport (both PCI legacy and modern interface)?.

Quoted 48 lines
Quoted 39 lines

} __packed;

struct epf_vhost_reg {
         u64 host_features;
         u64 guest_features;
         u16 msix_config;
         u16 num_queues;
         u8 device_status;
         u8 config_generation;
         u32 isr;
         u8 cmd;
         u8 cmd_status;
         struct epf_vhost_reg_queue vq[MAX_VQS];
} __packed;

Quoted 23 lines

+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev,
+                       u32 command)

(quoted diff) +18 -0
+{
+    struct virtio_pci_epf *pci_epf;
+    void __iomem *ioaddr;
+    ktime_t timeout;
+    bool timedout;
+    int ret = 0;
+    u8 status;
+
+    pci_epf = to_virtio_pci_epf(vp_dev);
+    ioaddr = vp_dev->ioaddr;
+
+    mutex_lock(&pci_epf->lock);
+    writeb(command, ioaddr + HOST_CMD);
+    timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+    while (1) {
+        timedout = ktime_after(ktime_get(), timeout);
+        status = readb(ioaddr + HOST_CMD_STATUS);
+
Quoted 7 lines
Quoted 4 lines

Several questions:

- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed.

The HOST_CMD (commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value in HOST_CMD. So HOST_CMD is less likely an issue.

Quoted 4 lines

Here's my understanding of the protocol:

1) RC write to HOST_CMD
2) RC wait for HOST_CMD_STATUS to be HOST_CMD_STATUS_OKAY

That's right!

Quoted 4 lines

It looks to me what EP should do is

1) EP reset HOST_CMD after reading new command

That's right! It does.

Quoted 2 lines

And it looks to me EP should also reset HOST_CMD_STATUS here?

yeah, that would require RC to send another command to reset the status. Didn't see it required in the normal scenario but good to add this.

Quoted 3 lines

(I thought there should be patch to handle stuffs like this but I didn't find it in this series)

This is added in [RFC PATCH 19/22] PCI: endpoint: Add EP function driver to provide VHOST interface

pci_epf_vhost_cmd_handler() gets commands from RC using "reg->cmd;". On the EP side, it is local memory access (mapped to BAR memory exposed to the host) and hence accessed using structure member access.

Quoted 24 lines
Quoted 10 lines

A sufficiently large time is given for the EP to complete it's operation (1 Sec) where the EP provides the status in HOST_CMD_STATUS. After it expires, HOST_CMD_STATUS_NONE is written to HOST_CMD_STATUS. There could be case where EP updates HOST_CMD_STATUS after RC writes HOST_CMD_STATUS_NONE, but by then HOST has already detected this as failure and error-ed out.

Quoted 3 lines
Quoted 2 lines

If you still want to introduce a new transport, a virtio spec patch would be helpful for us to understand the device API.

Okay, that should be on https://github.com/oasis-tcs/virtio-spec.git?

Yes.

Quoted 4 lines
Quoted 3 lines

- You have you vendor specific layout (according to virtio_pci_epb_table()), so I guess you it's better to have a vendor specific vDPA driver instead

Okay, with vDPA, we are free to define our own layouts.

Quoted 6 lines

Right, but vDPA have other requirements. E.g it requires the device have the ability to save/restore the state (e.g the last_avail_idx).

So it actually depends on what you want. If you don't care about userspace drivers and want to have a standard transport, you can still go virtio.

okay.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

- The advantage of vendor specific vDPA driver is that it can 1) have less codes 2) support userspace drivers through vhost-vDPA (instead of inventing new APIs since we can't use vfio-pci here).

Quoted 7 lines
Quoted 3 lines

I see there's an additional level of indirection from virtio to vDPA and probably no need for spec update but don't exactly see how it'll reduce code.

AFAIK you don't need to implement your own setup_vq and del_vq.

There should still be some entity that allocates memory for virtqueues and then communicate this address to the backend.

Maybe I have to look this further.

Quoted 16 lines
Quoted 3 lines

For 2, Isn't vhost-vdpa supposed to run on virtio backend?

Not currently, vDPA is a superset of virtio (e.g it support virtqueue state save/restore). This it should be possible in the future probably.

Quoted 5 lines

 From a high level, I think I should be able to use vDPA for virtio_pci_epf.c. Would you also suggest using vDPA for ntb_virtio.c? ([RFC PATCH 20/22] NTB: Add a new NTB client driver to implement VIRTIO functionality).

Quoted 5 lines

I think it's your call. If you want

1) a well-defined standard virtio transport
2) willing to finalize d and maintain the spec
3) doesn't care about userspace drivers

IIUC, we can use vDPA (virtio_vdpa.c) but still don't need userspace drivers right?

Quoted 2 lines

You can go with virtio, otherwise vDPA.

Okay, let me see. Thanks for your inputs.

Best Regards,
Kishon

#43 lore raw
Quoted 138 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 135 lines · Jason Wang
Quoted 107 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 98 lines · Jason Wang

Hi Kishon:

Quoted 95 lines · Kishon Vijay Abraham I
Quoted 4 lines

Then you need something that is functional equivalent to virtio PCI which is actually the concept of vDPA (e.g vDPA provides alternatives if
the queue_sel is hard in the EP implementation).

Okay, I just tried to compare the 'struct vdpa_config_ops' and 'struct vhost_config_ops' ( introduced in [RFC PATCH 03/22] vhost: Add ops for the VHOST driver to configure VHOST device).

struct vdpa_config_ops {
     /* Virtqueue ops */
     int (*set_vq_address)(struct vdpa_device *vdev,
                   u16 idx, u64 desc_area, u64 driver_area, u64 device_area);
     void (*set_vq_num)(struct vdpa_device *vdev, u16 idx, u32 num); void (*kick_vq)(struct vdpa_device *vdev, u16 idx); void (*set_vq_cb)(struct vdpa_device *vdev, u16 idx, struct vdpa_callback *cb);
     void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool
ready);
     bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx); int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, const struct vdpa_vq_state *state);
     int (*get_vq_state)(struct vdpa_device *vdev, u16 idx, struct vdpa_vq_state *state);
     struct vdpa_notification_area (*get_vq_notification)(struct vdpa_device *vdev, u16 idx); /* vq irq is not expected to be changed once DRIVER_OK is set */ int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

     /* Device ops */
     u32 (*get_vq_align)(struct vdpa_device *vdev);
     u64 (*get_features)(struct vdpa_device *vdev);
     int (*set_features)(struct vdpa_device *vdev, u64 features); void (*set_config_cb)(struct vdpa_device *vdev,
                   struct vdpa_callback *cb);
     u16 (*get_vq_num_max)(struct vdpa_device *vdev);
     u32 (*get_device_id)(struct vdpa_device *vdev);
     u32 (*get_vendor_id)(struct vdpa_device *vdev);
     u8 (*get_status)(struct vdpa_device *vdev);
     void (*set_status)(struct vdpa_device *vdev, u8 status); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len);
     void (*set_config)(struct vdpa_device *vdev, unsigned int offset, const void *buf, unsigned int len);
     u32 (*get_generation)(struct vdpa_device *vdev);

     /* DMA ops */
     int (*set_map)(struct vdpa_device *vdev, struct vhost_iotlb
*iotlb);
     int (*dma_map)(struct vdpa_device *vdev, u64 iova, u64 size, u64 pa, u32 perm);
     int (*dma_unmap)(struct vdpa_device *vdev, u64 iova, u64 size);

     /* Free device resources */
     void (*free)(struct vdpa_device *vdev);
};

+struct vhost_config_ops {
+    int (*create_vqs)(struct vhost_dev *vdev, unsigned int nvqs, +              unsigned int num_bufs, struct vhost_virtqueue *vqs[], +              vhost_vq_callback_t *callbacks[],
+              const char * const names[]);
+    void (*del_vqs)(struct vhost_dev *vdev);
+    int (*write)(struct vhost_dev *vdev, u64 vhost_dst, void *src, int len);
+    int (*read)(struct vhost_dev *vdev, void *dst, u64 vhost_src, int len);
+    int (*set_features)(struct vhost_dev *vdev, u64 device_features); +    int (*set_status)(struct vhost_dev *vdev, u8 status); +    u8 (*get_status)(struct vhost_dev *vdev);

(quoted diff) +2 -0
+};
+

struct virtio_config_ops
I think there's some overlap here and some of the ops tries to do the same thing.

I think it differs in (*set_vq_address)() and (*create_vqs)(). [create_vqs() introduced in struct vhost_config_ops provides complimentary functionality to (*find_vqs)() in struct virtio_config_ops. It seemingly encapsulates the functionality of (*set_vq_address)(), (*set_vq_num)(), (*set_vq_cb)(),..].

Back to the difference between (*set_vq_address)() and (*create_vqs)(), set_vq_address() directly provides the virtqueue address to the vdpa device but create_vqs() only provides the parameters of the virtqueue (like the number of virtqueues, number of buffers) but does not directly
provide the address. IMO the backend client drivers (like net or vhost) shouldn't/cannot by itself know how to access the vring created on virtio front-end. The vdpa device/vhost device should have logic for that. That will help the client drivers to work with different types of vdpa device/vhost device and can access the vring created by virtio irrespective of whether the vring can be accessed via mmio or kernel space or user space.

Quoted 6 lines
Quoted 3 lines

I think vdpa always works with client drivers in userspace and providing
userspace address for vring.

Sorry for being unclear. What I meant is not replacing vDPA with the vhost(bus) you proposed but the possibility of replacing virtio-pci-epf with vDPA in:

Quoted 3 lines

Okay, so the virtio back-end still use vhost and front end should use vDPA. I see. So the host side PCI driver for EPF should populate vdpa_config_ops and invoke vdpa_register_device().

Yes.

Quoted 5 lines
Quoted 3 lines

My question is basically for the part of virtio_pci_epf_send_command(), so it looks to me you have a vendor specific API to replace the virtio-pci layout of the BAR:

Even when we use vDPA, we have to use some sort of virtio_pci_epf_send_command() to communicate with virtio backend right?

Right.

Quoted 10 lines

Right, the layout is slightly different from the standard layout.

This is the layout
struct epf_vhost_reg_queue {
         u8 cmd;
         u8 cmd_status;
         u16 status;
         u16 num_buffers;
         u16 msix_vector;
         u64 queue_addr;

Quoted 7 lines
Quoted 1 line

What's the meaning of queue_addr here?

Using queue_addr, the virtio front-end communicates the address of the allocated memory for virtqueue to the virtio back-end.

Quoted 2 lines

Does not mean the device expects a contiguous memory for avail/desc/used ring?

It's contiguous memory. Isn't this similar to other virtio transport (both PCI legacy and modern interface)?.

That's only for legacy device, for modern device we don't have such restriction.

Quoted 57 lines
Quoted 45 lines
Quoted 37 lines

} __packed;

struct epf_vhost_reg {
         u64 host_features;
         u64 guest_features;
         u16 msix_config;
         u16 num_queues;
         u8 device_status;
         u8 config_generation;
         u32 isr;
         u8 cmd;
         u8 cmd_status;
         struct epf_vhost_reg_queue vq[MAX_VQS];
} __packed;

Quoted 22 lines

+static int virtio_pci_epf_send_command(struct virtio_pci_device *vp_dev,
+                       u32 command)

(quoted diff) +18 -0
+{
+    struct virtio_pci_epf *pci_epf;
+    void __iomem *ioaddr;
+    ktime_t timeout;
+    bool timedout;
+    int ret = 0;
+    u8 status;
+
+    pci_epf = to_virtio_pci_epf(vp_dev);
+    ioaddr = vp_dev->ioaddr;
+
+    mutex_lock(&pci_epf->lock);
+    writeb(command, ioaddr + HOST_CMD);
+    timeout = ktime_add_ms(ktime_get(), COMMAND_TIMEOUT);
+    while (1) {
+        timedout = ktime_after(ktime_get(), timeout);
+        status = readb(ioaddr + HOST_CMD_STATUS);
+
Quoted 7 lines
Quoted 4 lines

Several questions:

- It's not clear to me how the synchronization is done between the RC and EP. E.g how and when the value of HOST_CMD_STATUS can be changed.

The HOST_CMD (commands sent to the EP) is serialized by using mutex. Once the EP reads the command, it resets the value in HOST_CMD. So HOST_CMD is less likely an issue.

Quoted 4 lines

Here's my understanding of the protocol:

1) RC write to HOST_CMD
2) RC wait for HOST_CMD_STATUS to be HOST_CMD_STATUS_OKAY

That's right!

Quoted 3 lines

It looks to me what EP should do is

1) EP reset HOST_CMD after reading new command

That's right! It does.

Quoted 1 line

And it looks to me EP should also reset HOST_CMD_STATUS here?

yeah, that would require RC to send another command to reset the status. Didn't see it required in the normal scenario but good to add this.

Quoted 8 lines
Quoted 2 lines

(I thought there should be patch to handle stuffs like this but I didn't find it in this series)

This is added in [RFC PATCH 19/22] PCI: endpoint: Add EP function driver to provide VHOST interface

pci_epf_vhost_cmd_handler() gets commands from RC using "reg->cmd;". On the EP side, it is local memory access (mapped to BAR memory exposed to the host) and hence accessed using structure member access.

Thanks for the pointer, will have a look at and I think this part need to be carefully designed and the key to the success of the epf transport.

maintainer
#44 lore raw
In reply to Kishon Vijay Abraham I

Hey Kishon,

Quoted 316 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG
          +                               +

          |                               |
          |                               |
          |                               |
          |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

              |                                                 |
              |                                                 |
              |                                                 |
              |                                                 |
       +------v------+                                   +------v------+
       |             |                                   |             |
       |    HOST1    |                                   |    HOST2    |
       |             |                                   |             |
       +------^------+                                   +------^------+
              |                                                 |
              |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

        +----------------+  +-----------+  +------------+  +----------+
        |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
        +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                |                 |              |              |
                |                 |              |              |
                |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
             |               |                    |                  |
             |               |                    |                  |
             |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 4 lines

    all the boards run Linux. The middle board provides NTB functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 8 lines
Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 9 lines

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 6 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Quoted 8 lines
Quoted 5 lines

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Quoted 7 lines
Quoted 4 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Quoted 25 lines
Quoted 5 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 2 lines

Thanks again for your suggestions!

You're welcome.

Quoted 8 lines
Quoted 4 lines

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Got it. My initial design was based on my understanding of your comments [1].

I'll try to create something based on your proposed design here.

Based on the above conversation it seems like I should wait for another revision of this set before reviewing the RPMSG part. Please confirm that my understanding is correct.

Thanks,
Mathieu

Quoted 133 lines

Regards
Kishon

[1] ->
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 127 lines

Thanks

Quoted 122 lines

Regards
Kishon

Quoted 118 lines
Quoted 9 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 1 line

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 98 lines

Regards
Kishon

Quoted 95 lines

Thanks

Quoted 92 lines

Thanks
Kishon

Quoted 89 lines

Thanks

Quoted 86 lines
Quoted 86 lines

Kishon Vijay Abraham I (22):
      vhost: Make _feature_ bits a property of vhost device
      vhost: Introduce standard Linux driver model in VHOST
      vhost: Add ops for the VHOST driver to configure VHOST device
      vringh: Add helpers to access vring in MMIO
      vhost: Add MMIO helpers for operations on vhost virtqueue
      vhost: Introduce configfs entry for configuring VHOST
      virtio_pci: Use request_threaded_irq() instead of request_irq()
      rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
      rpmsg: Introduce configfs entry for configuring rpmsg
      rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
      rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
      virtio: Add ops to allocate and free buffer
      rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
      rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
      rpmsg.txt: Add Documentation to configure rpmsg using configfs
      virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
        device
      PCI: endpoint: Add EP function driver to provide VHOST interface
      NTB: Add a new NTB client driver to implement VIRTIO functionality
      NTB: Add a new NTB client driver to implement VHOST functionality
      NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

     Documentation/driver-api/ntb.rst              |   11 + Documentation/rpmsg.txt                       |   56 + drivers/ntb/Kconfig                           |   18 + drivers/ntb/Makefile                          |    2 + drivers/ntb/ntb_vhost.c                       |  776 +++++++++++ drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++ drivers/ntb/ntb_virtio.h                      |   56 + drivers/pci/endpoint/functions/Kconfig        |   11 + drivers/pci/endpoint/functions/Makefile       |    1 + .../pci/endpoint/functions/pci-epf-vhost.c    | 1144
++++++++++++++++
     drivers/rpmsg/Kconfig                         |   10 + drivers/rpmsg/Makefile                        |    3 +- drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++ drivers/rpmsg/rpmsg_core.c                    |    7 + drivers/rpmsg/rpmsg_internal.h                |  136 ++ drivers/rpmsg/vhost_rpmsg_bus.c               | 1151
+++++++++++++++++
     drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++- drivers/vhost/Kconfig                         |    1 + drivers/vhost/Makefile                        |    2 +- drivers/vhost/net.c                           |   10 +-

     drivers/vhost/scsi.c                          |   24 +-
     drivers/vhost/test.c                          |   17 +-
     drivers/vhost/vdpa.c                          |    2 +-

     drivers/vhost/vhost.c                         |  730 ++++++++++- drivers/vhost/vhost_cfs.c                     |  341 +++++ drivers/vhost/vringh.c                        |  332 +++++ drivers/vhost/vsock.c                         |   20 +- drivers/virtio/Kconfig                        |    9 + drivers/virtio/Makefile                       |    1 + drivers/virtio/virtio_pci_common.c            |   25 +- drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++ include/linux/mod_devicetable.h               |    6 + include/linux/rpmsg.h                         |    6 + {drivers/vhost => include/linux}/vhost.h      |  132 +- include/linux/virtio.h                        |    3 + include/linux/virtio_config.h                 |   42 + include/linux/vringh.h                        |   46 + samples/rpmsg/rpmsg_client_sample.c           |   32 +- tools/virtio/virtio_test.c                    |    2 +-

     39 files changed, 7083 insertions(+), 183 deletions(-)

     create mode 100644 drivers/ntb/ntb_vhost.c
     create mode 100644 drivers/ntb/ntb_virtio.c
     create mode 100644 drivers/ntb/ntb_virtio.h
     create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
     create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
     create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#45 lore raw

Hi Mathieu,

Quoted 324 lines · Mathieu Poirier

Hey Kishon,

Quoted 321 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 215 lines · Jason Wang
Quoted 203 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 200 lines · Jason Wang
Quoted 120 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 99 lines · Jason Wang
Quoted 88 lines · Kishon Vijay Abraham I

Hi Jason,

Quoted 85 lines · Jason Wang
Quoted 84 lines · Michael S. Tsirkin
Quoted 83 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

     VHOST RPMSG                     VIRTIO RPMSG +                               +

           |                               |
           |                               |
           |                               |
           |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

         VHOST RPMSG                                      VIRTIO RPMSG +                                                 +

               |                                                 |
               |                                                 |
               |                                                 |
               |                                                 |
        +------v------+                                   +------v------+
        |             |                                   |             |
        |    HOST1    |                                   |    HOST2    |
        |             |                                   |             |
        +------^------+                                   +------^------+
               |                                                 |
               |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

         +----------------+  +-----------+  +------------+  +----------+
         |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
         +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
                 |                 |              |              |
                 |                 |              |              |
                 |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
              |               |                    |                  |
              |               |                    |                  |
              |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+

This was initially proposed here [1]

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

[1] ->
https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.

I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).

This is about connecting two different HW systems both running Linux and doesn't necessarily involve virtualization.

Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.

I'd also like to point out, this series tries to have communication between two
SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and any
of the HW in NTB below should be able to use a virtio-vhost communication

#ls drivers/ntb/hw/
amd  epf  idt  intel  mscc

Quoted 8 lines

And similarly for the PCIe RC<->EP communication, this adds a generic endpoint function driver and hence any SoC that supports configurable PCIe endpoint can use virtio-vhost communication

# ls drivers/pci/controller/dwc/*ep*
drivers/pci/controller/dwc/pcie-designware-ep.c
drivers/pci/controller/dwc/pcie-uniphier-ep.c
drivers/pci/controller/dwc/pci-layerscape-ep.c

Thanks for those backgrounds.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

    So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio functionality reserving memory for virtqueues and the other provides vhost functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was probably intermediate entity in virtualization?)

Quoted 8 lines
Quoted 3 lines

(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.

Initially I had named this vringh but later decided to choose vhost instead of vringh. vhost is still a virtio backend (not necessarily userspace) though it now resides in an entirely different system. Whatever virtio is for a frontend system, vhost can be that for a backend system. vring can be for accessing virtqueue and can be used either in frontend or backend.

Ok.

Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
Quoted 1 line

Have you considered to implement these through vDPA?

IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),

I see.

Quoted 7 lines
Quoted 4 lines

    all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.

So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

IMHO we'll need two buses one for frontend and other for backend because the two components can then co-operate/interact with each other to provide a functionality. Though both will seemingly provide similar callbacks, they are both provide symmetrical or complimentary funcitonality and need not be same or
identical.

Having the same bus can also create sequencing issues.

Quoted 8 lines

If you look at virtio_dev_probe() of virtio_bus

device_features = dev->config->get_features(dev);

Now if we use same bus for both front-end and back-end, both will try to get_features when there has been no set_features. Ideally vhost device should be initialized first with the set of features it supports. Vhost and virtio should use "status" and "features" complimentarily and not identically.

Yes, but there's no need for doing status/features passthrough in epf vhost drivers.b

Quoted 6 lines

virtio device (or frontend) cannot be initialized before vhost device (or backend) gets initialized with data such as features. Similarly vhost (backend)
cannot access virqueues or buffers before virtio (frontend) sets VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as the physical memory for virtqueues are created by virtio (frontend).

Quoted 8 lines
Quoted 5 lines

epf vhost drivers need to implement two devices: vhost(vringh) device and virtio device (which is a mediated device). The vhost(vringh) device is doing feature negotiation with the virtio device via RC/EP or NTB. The virtio device is doing feature negotiation with local virtio drivers. If there're feature mismatch, epf vhost drivers and do mediation between them.

Here epf vhost should be initialized with a set of features for it to negotiate either as vhost device or virtio device no? Where should the initial feature set for epf vhost come from?

Quoted 9 lines

I think it can work as:

1) Having an initial features (hard coded in the code) set X in epf vhost
2) Using this X for both virtio device and vhost(vringh) device
3) local virtio driver will negotiate with virtio device with feature set Y
4) remote virtio driver will negotiate with vringh device with feature set Z
5) mediate between feature Y and feature Z since both Y and Z are a subset of X

okay. I'm also thinking if we could have configfs for configuring this. Anyways we could find different approaches of configuring this.

Quoted 40 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

It will have virtqueues but only used for the communication between itself and
uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and
vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I think this will work however there is an addtional copy between vringh queue and virtqueue,

I think not? E.g in use case 1), if we stick to virtio bus, we will have:

virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2) -> virtio pci (RC) <-> virtio rpmsg (RC)

IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?

Yes.

Quoted 2 lines

And virtio
ring(2) is created by virtio pci (RC).

Yes.

Quoted 6 lines
Quoted 2 lines

What epf vhost driver did is to read from virtio ring(1) about the buffer len and addr and them DMA to Linux(RC)?

okay, I made some optimization here where vhost-rpmsg using a helper writes a buffer from rpmsg's upper layer directly to remote Linux (RC) as against here were it has to be first written to virtio ring (1).

Quoted 8 lines
Quoted 5 lines

Thinking how this would look for NTB
virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)  <- virtio ring(2) -> virtio-rpmsg (HOST2)

Here the NTB(HOST1) will access the virtio ring(2) using vringh?

Yes, I think so it needs to use vring to access virtio ring (1) as well.

NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.

Quoted 7 lines
Quoted 4 lines

Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?

Yes.

okay, I haven't looked at this but the backend of virtio_blk should access an actual storage device no?

Quoted 29 lines
Quoted 5 lines

I'd like to get clarity on two things in the approach you suggested, one is features (since epf vhost should ideally be transparent to any virtio driver)

We can have have an array of pre-defined features indexed by virtio device id in the code.

Quoted 2 lines

and the other is how certain inputs to virtio device such as number of buffers be determined.

We can start from hard coded the value like 256, or introduce some API for user to change the value.

Quoted 2 lines

Thanks again for your suggestions!

You're welcome.

Note that I just want to check whether or not we can reuse the virtio bus/driver. It's something similar to what you proposed in Software Layering but we just replace "vhost core" with "virtio bus" and move the vhost core below epf/ntb/platform transport.

Quoted 7 lines
Quoted 3 lines

Got it. My initial design was based on my understanding of your comments [1].

I'll try to create something based on your proposed design here.

Based on the above conversation it seems like I should wait for another revision of this set before reviewing the RPMSG part. Please confirm that my understanding is correct.

Right, there are multiple parts in this series that has to be aligned. I'd still think irrespective of the approach something like Address Service Notification support might have to be supported by rpmsg.

Thanks
Kishon

Quoted 137 lines

Thanks,
Mathieu

Quoted 133 lines

Regards
Kishon

[1] ->
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 127 lines

Thanks

Quoted 122 lines

Regards
Kishon

Quoted 118 lines
Quoted 9 lines

in some cases adds latency because of forwarding interrupts between vhost and virtio driver, vhost drivers providing features (which means it has to be aware of which virtio driver will be connected). virtio drivers (front end) generally access the buffers from it's local memory but when in backend it can access over MMIO (like PCI EPF or NTB) or userspace.

Quoted 1 line

Does this make sense?

Two copies in my opinion is an issue but lets get others opinions as well.

Sure.

Quoted 1 line

Thanks for your suggestions!

You're welcome.

Thanks

Quoted 98 lines

Regards
Kishon

Quoted 95 lines

Thanks

Quoted 92 lines

Thanks
Kishon

Quoted 89 lines

Thanks

Quoted 86 lines
Quoted 86 lines

Kishon Vijay Abraham I (22):
      vhost: Make _feature_ bits a property of vhost device
      vhost: Introduce standard Linux driver model in VHOST
      vhost: Add ops for the VHOST driver to configure VHOST device
      vringh: Add helpers to access vring in MMIO
      vhost: Add MMIO helpers for operations on vhost virtqueue
      vhost: Introduce configfs entry for configuring VHOST
      virtio_pci: Use request_threaded_irq() instead of request_irq()
      rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when reading messages
      rpmsg: Introduce configfs entry for configuring rpmsg
      rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
      rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to rpmsg_internal.h
      virtio: Add ops to allocate and free buffer
      rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and virtio_free_buffer()
      rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending message
      rpmsg.txt: Add Documentation to configure rpmsg using configfs
      virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe
Endpoint
        device
      PCI: endpoint: Add EP function driver to provide VHOST interface
      NTB: Add a new NTB client driver to implement VIRTIO functionality
      NTB: Add a new NTB client driver to implement VHOST functionality
      NTB: Describe the ntb_virtio and ntb_vhost client in the
documentation

      Documentation/driver-api/ntb.rst              |   11 +
      Documentation/rpmsg.txt                       |   56 +
      drivers/ntb/Kconfig                           |   18 +
      drivers/ntb/Makefile                          |    2 +
      drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
      drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
      drivers/ntb/ntb_virtio.h                      |   56 +
      drivers/pci/endpoint/functions/Kconfig        |   11 +
      drivers/pci/endpoint/functions/Makefile       |    1 +
      .../pci/endpoint/functions/pci-epf-vhost.c    | 1144
++++++++++++++++
      drivers/rpmsg/Kconfig                         |   10 +
      drivers/rpmsg/Makefile                        |    3 +-
      drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
      drivers/rpmsg/rpmsg_core.c                    |    7 +
      drivers/rpmsg/rpmsg_internal.h                |  136 ++
      drivers/rpmsg/vhost_rpmsg_bus.c               | 1151
+++++++++++++++++
      drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
      drivers/vhost/Kconfig                         |    1 +
      drivers/vhost/Makefile                        |    2 +-
      drivers/vhost/net.c                           |   10 +-
      drivers/vhost/scsi.c                          |   24 +-
      drivers/vhost/test.c                          |   17 +-
      drivers/vhost/vdpa.c                          |    2 +-
      drivers/vhost/vhost.c                         |  730 ++++++++++-
      drivers/vhost/vhost_cfs.c                     |  341 +++++
      drivers/vhost/vringh.c                        |  332 +++++
      drivers/vhost/vsock.c                         |   20 +-
      drivers/virtio/Kconfig                        |    9 +
      drivers/virtio/Makefile                       |    1 +
      drivers/virtio/virtio_pci_common.c            |   25 +-
      drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
      include/linux/mod_devicetable.h               |    6 +
      include/linux/rpmsg.h                         |    6 +
      {drivers/vhost => include/linux}/vhost.h      |  132 +-
      include/linux/virtio.h                        |    3 +
      include/linux/virtio_config.h                 |   42 +
      include/linux/vringh.h                        |   46 +
      samples/rpmsg/rpmsg_client_sample.c           |   32 +-
      tools/virtio/virtio_test.c                    |    2 +-
      39 files changed, 7083 insertions(+), 183 deletions(-)

     create mode 100644 drivers/ntb/ntb_vhost.c
     create mode 100644 drivers/ntb/ntb_virtio.c
     create mode 100644 drivers/ntb/ntb_virtio.h
     create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c create mode 100644 drivers/rpmsg/rpmsg_cfs.c
     create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c create mode 100644 drivers/vhost/vhost_cfs.c
     create mode 100644 drivers/virtio/virtio_pci_epf.c rename {drivers/vhost => include/linux}/vhost.h (66%)

Signature · 1 line
2.17.1
#46 lore raw
In reply to Michael S. Tsirkin

Hi Michael,

Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

      +                               +
      |                               |
      |                               |
      |                               |
      |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |

   +------v------+                                   +------v------+
   |             |                                   |             |
   |    HOST1    |                                   |    HOST2    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

    +----------------+  +-----------+  +------------+  +----------+
    |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
    +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
            |                 |              |              |
            |                 |              |              |
            |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
         |               |                    |                  |
         |               |                    |                  |
         |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 7 lines
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Great to hear! Thanks in advance for reviewing!

Regards
Kishon

Quoted 85 lines
Quoted 84 lines

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst              |   11 +
 Documentation/rpmsg.txt                       |   56 +
 drivers/ntb/Kconfig                           |   18 +
 drivers/ntb/Makefile                          |    2 +
 drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
 drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
 drivers/ntb/ntb_virtio.h                      |   56 +
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig                         |   10 +
 drivers/rpmsg/Makefile                        |    3 +-
 drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
 drivers/rpmsg/rpmsg_core.c                    |    7 +
 drivers/rpmsg/rpmsg_internal.h                |  136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
 drivers/vhost/Kconfig                         |    1 +
 drivers/vhost/Makefile                        |    2 +-
 drivers/vhost/net.c                           |   10 +-
 drivers/vhost/scsi.c                          |   24 +-
 drivers/vhost/test.c                          |   17 +-
 drivers/vhost/vdpa.c                          |    2 +-
 drivers/vhost/vhost.c                         |  730 ++++++++++-
 drivers/vhost/vhost_cfs.c                     |  341 +++++
 drivers/vhost/vringh.c                        |  332 +++++
 drivers/vhost/vsock.c                         |   20 +-
 drivers/virtio/Kconfig                        |    9 +
 drivers/virtio/Makefile                       |    1 +
 drivers/virtio/virtio_pci_common.c            |   25 +-
 drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
 include/linux/mod_devicetable.h               |    6 +
 include/linux/rpmsg.h                         |    6 +
 {drivers/vhost => include/linux}/vhost.h      |  132 +-
 include/linux/virtio.h                        |    3 +
 include/linux/virtio_config.h                 |   42 +
 include/linux/vringh.h                        |   46 +
 samples/rpmsg/rpmsg_client_sample.c           |   32 +-
 tools/virtio/virtio_test.c                    |    2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)
Signature · 1 line
2.17.1
maintainer
#47 lore raw
In reply to Michael S. Tsirkin
Quoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

      +                               +
      |                               |
      |                               |
      |                               |
      |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |

   +------v------+                                   +------v------+
   |             |                                   |             |
   |    HOST1    |                                   |    HOST2    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

    +----------------+  +-----------+  +------------+  +----------+
    |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
    +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
            |                 |              |              |
            |                 |              |              |
            |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
         |               |                    |                  |
         |               |                    |                  |
         |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 7 lines
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Same here - it will take time. This patchset is sizable and sits behind a few others that are equally big.

Quoted 85 lines
Quoted 84 lines

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst              |   11 +
 Documentation/rpmsg.txt                       |   56 +
 drivers/ntb/Kconfig                           |   18 +
 drivers/ntb/Makefile                          |    2 +
 drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
 drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
 drivers/ntb/ntb_virtio.h                      |   56 +
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig                         |   10 +
 drivers/rpmsg/Makefile                        |    3 +-
 drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
 drivers/rpmsg/rpmsg_core.c                    |    7 +
 drivers/rpmsg/rpmsg_internal.h                |  136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
 drivers/vhost/Kconfig                         |    1 +
 drivers/vhost/Makefile                        |    2 +-
 drivers/vhost/net.c                           |   10 +-
 drivers/vhost/scsi.c                          |   24 +-
 drivers/vhost/test.c                          |   17 +-
 drivers/vhost/vdpa.c                          |    2 +-
 drivers/vhost/vhost.c                         |  730 ++++++++++-
 drivers/vhost/vhost_cfs.c                     |  341 +++++
 drivers/vhost/vringh.c                        |  332 +++++
 drivers/vhost/vsock.c                         |   20 +-
 drivers/virtio/Kconfig                        |    9 +
 drivers/virtio/Makefile                       |    1 +
 drivers/virtio/virtio_pci_common.c            |   25 +-
 drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
 include/linux/mod_devicetable.h               |    6 +
 include/linux/rpmsg.h                         |    6 +
 {drivers/vhost => include/linux}/vhost.h      |  132 +-
 include/linux/virtio.h                        |    3 +
 include/linux/virtio_config.h                 |   42 +
 include/linux/vringh.h                        |   46 +
 samples/rpmsg/rpmsg_client_sample.c           |   32 +-
 tools/virtio/virtio_test.c                    |    2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1

#48 lore raw

+Alan, Haotian

Quoted 180 lines · Mathieu Poirier
Quoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Kishon Vijay Abraham I

This series enhances Linux Vhost support to enable SoC-to-SoC communication over MMIO. This series enables rpmsg communication between two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

VHOST RPMSG VIRTIO RPMSG

      +                               +
      |                               |
      |                               |
      |                               |
      |                               |
+-----v------+                 +------v-------+
|   Linux    |                 |     Linux    |
|  Endpoint  |                 | Root Complex |
|            <----------------->              |
|            |                 |              |
|    SOC1    |                 |     SOC2     |
+------------+                 +--------------+

UseCase 2:

VHOST RPMSG VIRTIO RPMSG
+ +
| |
| |
| |
| |

   +------v------+                                   +------v------+
   |             |                                   |             |
   |    HOST1    |                                   |    HOST2    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------------------------------------------------------------------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |  SoC With Multiple EP Instances   |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series adds support only for RPMSG VHOST, however something similar should be done for net and scsi. With that any vhost device (PCI, NTB, Platform device, user) can use any of the vhost client driver.

    +----------------+  +-----------+  +------------+  +----------+
    |  RPMSG VHOST   |  | NET VHOST |  | SCSI VHOST |  |    X     |
    +-------^--------+  +-----^-----+  +-----^------+  +----^-----+
            |                 |              |              |
            |                 |              |              |
            |                 |              |              |
+-----------v-----------------v--------------v--------------v----------+
|                            VHOST CORE                                |
+--------^---------------^--------------------^------------------^-----+
         |               |                    |                  |
         |               |                    |                  |
         |               |                    |                  |
+--------v-------+  +----v------+  +----------v----------+  +----v-----+
|  PCI EPF VHOST |  | NTB VHOST |  |PLATFORM DEVICE VHOST|  |    X     |
+----------------+  +-----------+  +---------------------+  +----------+
Quoted 7 lines
Quoted 3 lines

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/[email protected]

I find this very interesting. A huge patchset so will take a bit to review, but I certainly plan to do that. Thanks!

Same here - it will take time. This patchset is sizable and sits behind a few others that are equally big.

Quoted 85 lines
Quoted 84 lines

Kishon Vijay Abraham I (22):
vhost: Make _feature_ bits a property of vhost device
vhost: Introduce standard Linux driver model in VHOST
vhost: Add ops for the VHOST driver to configure VHOST device
vringh: Add helpers to access vring in MMIO
vhost: Add MMIO helpers for operations on vhost virtqueue
vhost: Introduce configfs entry for configuring VHOST
virtio_pci: Use request_threaded_irq() instead of request_irq()
rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
reading messages
rpmsg: Introduce configfs entry for configuring rpmsg
rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
rpmsg_internal.h
virtio: Add ops to allocate and free buffer
rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
virtio_free_buffer()
rpmsg: Add VHOST based remote processor messaging bus samples/rpmsg: Setup delayed work to send message samples/rpmsg: Wait for address to be bound to rpdev for sending
message
rpmsg.txt: Add Documentation to configure rpmsg using configfs
virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
device
PCI: endpoint: Add EP function driver to provide VHOST interface
NTB: Add a new NTB client driver to implement VIRTIO functionality
NTB: Add a new NTB client driver to implement VHOST functionality
NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst              |   11 +
 Documentation/rpmsg.txt                       |   56 +
 drivers/ntb/Kconfig                           |   18 +
 drivers/ntb/Makefile                          |    2 +
 drivers/ntb/ntb_vhost.c                       |  776 +++++++++++
 drivers/ntb/ntb_virtio.c                      |  853 ++++++++++++
 drivers/ntb/ntb_virtio.h                      |   56 +
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 .../pci/endpoint/functions/pci-epf-vhost.c    | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig                         |   10 +
 drivers/rpmsg/Makefile                        |    3 +-
 drivers/rpmsg/rpmsg_cfs.c                     |  394 ++++++
 drivers/rpmsg/rpmsg_core.c                    |    7 +
 drivers/rpmsg/rpmsg_internal.h                |  136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c               | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c              |  184 ++-
 drivers/vhost/Kconfig                         |    1 +
 drivers/vhost/Makefile                        |    2 +-
 drivers/vhost/net.c                           |   10 +-
 drivers/vhost/scsi.c                          |   24 +-
 drivers/vhost/test.c                          |   17 +-
 drivers/vhost/vdpa.c                          |    2 +-
 drivers/vhost/vhost.c                         |  730 ++++++++++-
 drivers/vhost/vhost_cfs.c                     |  341 +++++
 drivers/vhost/vringh.c                        |  332 +++++
 drivers/vhost/vsock.c                         |   20 +-
 drivers/virtio/Kconfig                        |    9 +
 drivers/virtio/Makefile                       |    1 +
 drivers/virtio/virtio_pci_common.c            |   25 +-
 drivers/virtio/virtio_pci_epf.c               |  670 ++++++++++
 include/linux/mod_devicetable.h               |    6 +
 include/linux/rpmsg.h                         |    6 +
 {drivers/vhost => include/linux}/vhost.h      |  132 +-
 include/linux/virtio.h                        |    3 +
 include/linux/virtio_config.h                 |   42 +
 include/linux/vringh.h                        |   46 +
 samples/rpmsg/rpmsg_client_sample.c           |   32 +-
 tools/virtio/virtio_test.c                    |    2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1