Skip to contentSkip to conversation

Introduce emudev library and iavf emudev driver

PATCH Superseded Chenbo Xia 7 participants 46 messages Dec 18, 2020 – Feb 12, 2024

Subsystems (2)
lore
Share & tools
lore

Chenbo Xia

author
#0 lore raw

cover Introduce emudev library and iavf emudev driver

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Design overview
---------------

                    +---------------------------------------+
                    |   +---------------+    +-----------+  |
                    |   |  iavf_emudev  |<-->| data path |  |
                    |   |    driver     |    |   driver  |  |
                    |   +---------------+    +-----------+  |
                    |           |                           |
                    | --------------------------- VDEV BUS  |
                    |           |                           |
                    |   +---------------+                   |
+--------------+    |   | vdev:         |                   |
| +----------+ |    |   | /path/to/vfio |                   |
| | Generic  | |    |   +---------------+                   |
| | vfio-dev | |    |           |                           |
| +----------+ |    |           |                           |
| +----------+ |    |      +----------+                     |
| | vfio-user| |    |      | vfio-user|                     |
| | client   | |<---|----->| server   |                     |
| +----------+ |    |      +----------+                     |
| QEMU/DPDK    |    | DPDK                                  |
+--------------+    +---------------------------------------+

- Generic vfio-dev/vfio-user client/vfio-user server
Above concepts are all introduced in librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

- vdev:/path/to/vfio.
It binds to vdev bus driver. The vdev device is defined by DPDK applications through command line as '--vdev=emu_iavf, path=/path/to/socket' in iavf_emudev case. Parameters in command line include device name (emu_iavf) which is used to identify corresponding driver (in this case, iavf_emudev driver), path=/path/to/socket which is used to open the transport interface to vfio-user client in QEMU/DPDK.

- data path driver.
The data path handling is splited to another standalone driver for modular design.

Chenbo Xia (8):
lib: introduce emudev library
doc: add emudev library guide
emu: introduce emulated iavf driver
emu/iavf: add vfio-user device register and unregister emu/iavf: add resource management and internal logic of iavf emu/iavf: add emudev operations to fit in emudev framework test/emudev: introduce functional test
doc: update release notes for iavf emudev driver

 MAINTAINERS                            |   12 +
 app/test/meson.build                   |    5 +-
 app/test/test_emudev.c                 |   29 +
 doc/guides/prog_guide/emudev.rst       |  122 +++
 doc/guides/prog_guide/index.rst        |    1 +
 doc/guides/rel_notes/release_21_02.rst |   16 +
 drivers/emu/iavf/iavf_emu.c            |  250 ++++++
 drivers/emu/iavf/iavf_emu_internal.h   |   69 ++
 drivers/emu/iavf/iavf_emu_test.c       |  174 ++++
 drivers/emu/iavf/iavf_emudev.c         |  237 ++++++
 drivers/emu/iavf/iavf_vfio_user.c      | 1053 ++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h      |   57 ++
 drivers/emu/iavf/meson.build           |   17 +
 drivers/emu/iavf/rte_iavf_emu.h        |  119 +++
 drivers/emu/iavf/version.map           |    3 +
 drivers/emu/meson.build                |    6 +
 drivers/meson.build                    |    1 +
 lib/librte_emudev/meson.build          |    5 +
 lib/librte_emudev/rte_emudev.c         |  486 +++++++++++
 lib/librte_emudev/rte_emudev.h         |  410 +++++++++
 lib/librte_emudev/rte_emudev_vdev.h    |   53 ++
 lib/librte_emudev/version.map          |   27 +
 lib/meson.build                        |    2 +-
 23 files changed, 3152 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 doc/guides/prog_guide/emudev.rst
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/8 lib: introduce emudev library

This patch introduces the emudev library. Emudev library is used to abstract an emulated device, whose type could be general (e.g., network, crypto and etc.). Several device-level APIs are implemented to use or manipulate the device. It can be attached to another data path driver (e.g., ethdev driver) to plug in its high performance data path.

---
 MAINTAINERS                         |   5 +
 lib/librte_emudev/meson.build       |   5 +
 lib/librte_emudev/rte_emudev.c      | 486 ++++++++++++++++++++++++++++
 lib/librte_emudev/rte_emudev.h      | 410 +++++++++++++++++++++++
 lib/librte_emudev/rte_emudev_vdev.h |  53 +++
 lib/librte_emudev/version.map       |  27 ++
 lib/meson.build                     |   2 +-
 7 files changed, 987 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
MAINTAINERS +5 -0
index 5fb4880758..1b395e181d 100644
@@ -1545,6 +1545,11 @@ M: Chenbo Xia <[email protected]>
 M: Xiuchun Lu <[email protected]>
 F: lib/librte_vfio_user/
 
+Emudev - EXPERIMENTAL
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: lib/librte_emudev/
+
 Test Applications
 -----------------
 
lib/librte_emudev/meson.build new file +5 -0
new file mode 100644
index 0000000000..4e16cecbaf
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('rte_emudev.c')
+headers = files('rte_emudev.h', 'rte_emudev_vdev.h')
lib/librte_emudev/rte_emudev.c new file +486 -0

486 lines not on this page · raw message

lib/librte_emudev/rte_emudev.h new file +410 -0

410 lines not on this page · raw message

lib/librte_emudev/rte_emudev_vdev.h new file +53 -0

53 lines not on this page · raw message

lib/librte_emudev/version.map new file +27 -0
new file mode 100644
index 0000000000..f800b4c21c
@@ -0,0 +1,27 @@
+EXPERIMENTAL {
+	global:
+
+	rte_emudev_allocate;
+	rte_emudev_release;
+	rte_emudev_allocated;
+	rte_emudev_start;
+	rte_emudev_stop;
+	rte_emudev_configure;
+	rte_emudev_close;
+	rte_emudev_subscribe_event;
+	rte_emudev_unsubscribe_event;
+	rte_emudev_count;
+	rte_emudev_get_dev_id;
+	rte_emudev_get_dev_info;
+	rte_emudev_get_mem_table;
+	rte_emudev_get_queue_info;
+	rte_emudev_get_irq_info;
+	rte_emudev_get_db_info;
+	rte_emudev_set_attr;
+	rte_emudev_get_attr;
+	rte_emudev_region_map;
+	rte_emudev_selftest;
+	rte_emudev_is_valid_id;
+
+	local: *;
+};
lib/meson.build +1 -1
index b7fbfcc95b..6dd07fb73e 100644
@@ -28,7 +28,7 @@ libraries = [
 	'rib', 'reorder', 'sched', 'security', 'stack', 'vhost',
 	# ipsec lib depends on net, crypto and security
 	'ipsec',
-	'vfio_user',
+	'vfio_user', 'emudev',
 	#fib lib depends on rib
 	'fib',
 	# add pkt framework libs which use other libs from above
Signature · 1 line
2.17.1
author
#2 lore raw

PATCH 2/8 doc: add emudev library guide

In reply to Chenbo Xia

Add emudev library guide and update release notes.

---
 doc/guides/prog_guide/emudev.rst       | 122 +++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/rel_notes/release_21_02.rst |  12 +++
 3 files changed, 135 insertions(+)
 create mode 100644 doc/guides/prog_guide/emudev.rst
doc/guides/prog_guide/emudev.rst new file +122 -0
new file mode 100644
index 0000000000..91ad520de7
@@ -0,0 +1,122 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Emulated Device Library
+=================
+
+Introduction
+------------
+
+The DPDK Emudev library is an abstraction for emulated device. This library
+provides a generic set of APIs for device provider, data path provider and
+applications to use.
+
+A device provider could be implemented as a driver on vdev bus. It should
+expose itself as an emudev for applications to use. It is responsible for the
+device resource management and the device's internal logic. All specifics of a
+device, except data path handling, should be implemented in the device
+provider. The device provider uses emudev APIs mainly for create/destroy an
+emudev instance. The device provider should also use a tranport to communicate
+with device consumer (e.g., virtual machine monitor or container). A potential
+choice could be vfio-user library, which implements the vfio-user protocol for
+emulating devices outside of a virtual machine monitor.
+
+A data path provider could be implemented as any type of driver on vdev bus.
+If the device you want to emulate is a network device, you could implement
+it as an ethdev driver. It is responsible for all data path handling. The data
+path provider uses emudev APIs mainly for getting device-related information
+from the device provider.
+
+Applications uses emudev APIs for device lifecycle management and configuration.
+
+Design
+------------
+
+Some key objects are designed in emudev.
+
+  ``Regions`` are the device layout exposed to the data path provider.
+
+  ``Queues`` are the data path queues that the data path provider needs. Queue
+  information includes queue base address, queue size, queue-related doorbell
+  and interrupt information.
+
+  ``Memory Table`` is the DMA mapping table. The data path provider could use
+  it to perform DMA read/write on device consumer's memory.
+
+Information of above key objects could be acquired through emudev APIs. The
+following will introduce the emudev APIs which are used by data path provider
+and applications. The APIs for device provider to use are allocate/release APIs
+and will not be listed because it's similar to other device abstraction.
+
+There are five categories of APIs:
+
+1. Lifecycle management
+
+* ``rte_emu_dev_start(dev_id)``
+* ``rte_emu_dev_stop(dev_id)``
+* ``rte_emu_dev_configure(dev_id)``
+* ``rte_emu_dev_close(dev_id)``
+
+  Above APIs are respectively for device start/stop/configure/close and mainly
+  for applications to use.
+
+  ``dev_id`` is the emudev device ID.
+
+2. Notification
+
+* ``rte_emu_subscribe_event(dev_id, ev_chnl)``
+* ``rte_emu_unsubscribe_event(dev_id, ev_chnl)``
+
+  Above APIs are for data path provider and applications to register events.
+  The mechanism of event notification could be different in different device
+  providers. A possbile implementation could be event callbacks.
+
+  ``ev_chnl`` is the event channel pointer. The definition varies between
+  different devices.
+
+3. Region-related
+
+* ``rte_emu_region_map(dev_id, index, region_size, base_addr)``
+* ``rte_emu_get_attr(dev_id, attr_name, attr)``
+* ``rte_emu_set_attr(dev_id, attr_name, attr)``
+
+  Above APIs are for data path provider and applications to read/write regions.
+  ``rte_emu_region_map`` is for directly mapping the region and use the mapped
+  address to read/write it. ``rte_emu_get_attr`` and ``rte_emu_set_attr`` are
+  respectively for getting/setting certain attributes in all regions.
+
+  Applications will set attributes or write regions for device configuration.
+
+  In ``rte_emu_region_map``:
+  - ``index`` is the region index.
+  - ``region_size`` is for saving the size of mapped region.
+  - ``base_addr`` is for saving the address of mapped region.
+
+  In ``rte_emu_get_attr`` and ``rte_emu_set_attr``:
+  - ``attr_name`` is the name of attribute. Note that attribute names are aligned
+  between device provider and data path provider for the same device.
+  - ``attr`` is the attribute value.
+
+4. Queue-related
+
+* ``rte_emu_get_queue_info(dev_id, queue, info)``
+* ``rte_emu_get_irq_info(dev_id, irq, info)``
+* ``rte_emu_get_db_info(dev_id, doorbell, info)``
+
+  Above APIs are for data path provider to get queue/interrupt/doorbell information.
+
+  - ``queue``, ``irq`` and ``doorbell`` are respectively the queue/interrupt/doorbell
+  index.
+  - ``info`` is for saving the queue/interrupt/doorbell info.
+
+5. Direct Memory Access
+
+* ``rte_emu_get_mem_table(dev_id, tb)``
+
+  Above APIs are for data path provider to get the information of DMA memory table.
+  The memory table implementation varies between different devices and memory table
+  operations should better be helper functions exposed by device provider. Because
+  address translation make a difference in data path performance, the memory table
+  implementation should have high efficiency.
+
+  ``tb`` is for saving the DMA memory table.
doc/guides/prog_guide/index.rst +1 -0
index f9847b1058..0ed15a0995 100644
@@ -71,3 +71,4 @@ Programmer's Guide
     profile_app
     glossary
     vfio_user_lib
+    emudev
doc/guides/rel_notes/release_21_02.rst +12 -0
index 6fbb6e8c39..3d26b6b580 100644
@@ -67,6 +67,18 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
+* **Added emudev Library.**
+
+  Added an experimental library ``librte_emudev`` to provide device abstraction
+  for an emulated device.
+
+  The library abstracts an emulated device and provides several categories of
+  device-level APIs. The specific device type could be general (e.g, network,
+  crypto and etc.). It can be attached to another data path driver (e.g, ethdev
+  driver) to leverage the high performance of DPDK data path driver.
+
+  See :doc:`../prog_guide/emudev` for more information.
+
 Removed Items
 -------------
 
Signature · 1 line
2.17.1
author
#3 lore raw

PATCH 3/8 emu: introduce emulated iavf driver

In reply to Chenbo Xia

This patch introduces emulated iavf driver. It is a vdev driver emulating all iavf device behavior except data path handling.

---
 MAINTAINERS                          |   7 +
 drivers/emu/iavf/iavf_emu.c          |  29 ++++
 drivers/emu/iavf/iavf_emu_internal.h |  49 +++++++
 drivers/emu/iavf/iavf_emudev.c       | 207 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   8 ++
 drivers/emu/iavf/rte_iavf_emu.h      |  43 ++++++
 drivers/emu/iavf/version.map         |   3 +
 drivers/emu/meson.build              |   6 +
 drivers/meson.build                  |   1 +
 9 files changed, 353 insertions(+)
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
MAINTAINERS +7 -0
index 1b395e181d..bca206ba8f 100644
@@ -1275,6 +1275,13 @@ F: doc/guides/rawdevs/ntb.rst
 F: examples/ntb/
 F: doc/guides/sample_app_ug/ntb.rst
 
+Emudev Drivers
+--------------
+
+Intel iavf
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: drivers/emulation/iavf/
 
 Packet processing
 -----------------
drivers/emu/iavf/iavf_emu.c new file +29 -0
new file mode 100644
index 0000000000..68d2c440e3
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "iavf_emu_internal.h"
+
+static int iavf_emu_dev_close(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf;
+
+	/* For now, we don't support device close when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Close failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_device(iavf);
+	dev->priv_data = NULL;
+
+	return 0;
+}
+
+struct rte_emudev_ops emu_iavf_ops = {
+	.dev_close = iavf_emu_dev_close,
+};
drivers/emu/iavf/iavf_emu_internal.h new file +49 -0

49 lines not on this page · raw message

drivers/emu/iavf/iavf_emudev.c new file +207 -0

207 lines not on this page · raw message

drivers/emu/iavf/meson.build new file +8 -0
new file mode 100644
index 0000000000..58c2a90383
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('iavf_emu.c', 'iavf_emudev.c')
+
+deps += ['bus_vdev', 'emudev']
+
+headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h new file +43 -0

43 lines not on this page · raw message

drivers/emu/iavf/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
drivers/emu/meson.build new file +6 -0
new file mode 100644
index 0000000000..acc8c395ef
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+drivers = ['iavf']
+std_deps = ['emudev']
+config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_EMUDEV'
drivers/meson.build +1 -0
index f9febc579e..4579832459 100644
@@ -16,6 +16,7 @@ subdirs = [
 	'vdpa',    # depends on common, bus and mempool.
 	'event',   # depends on common, bus, mempool and net.
 	'baseband', # depends on common and bus.
+	'emu', # depends on common and bus.
 ]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
Signature · 1 line
2.17.1
author
#4 lore raw

PATCH 4/8 emu/iavf: add vfio-user device register and unregister

In reply to Chenbo Xia

This patch adds vfio-user APIs call in driver probe and remove. rte_vfio_user_register() and rte_vfio_user_unregister() are called to create/destroy a vfio-user device. Notify callbacks that libvfio_user defines are also implemented.

---
 drivers/emu/iavf/iavf_emu.c          |   3 +-
 drivers/emu/iavf/iavf_emu_internal.h |  19 ++
 drivers/emu/iavf/iavf_emudev.c       |  12 +-
 drivers/emu/iavf/iavf_vfio_user.c    | 384 +++++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h    |  16 ++
 drivers/emu/iavf/meson.build         |   5 +-
 drivers/emu/iavf/rte_iavf_emu.h      |  17 ++
 7 files changed, 452 insertions(+), 4 deletions(-)
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
drivers/emu/iavf/iavf_emu.c +2 -1
index 68d2c440e3..dfd9796920 100644
@@ -2,7 +2,7 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 static int iavf_emu_dev_close(struct rte_emudev *dev)
 {
@@ -18,6 +18,7 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
 
drivers/emu/iavf/iavf_emu_internal.h +19 -0
index a726bfe577..10197c00ba 100644
@@ -17,6 +17,13 @@ extern int emu_iavf_logtype;
 #define EMU_IAVF_LOG(level, ...) \
 	rte_log(RTE_LOG_ ## level, emu_iavf_logtype, "EMU_IAVF: " __VA_ARGS__)
 
+struct iavf_emu_vfio_user {
+	int dev_id;
+	struct vfio_device_info *dev_info;
+	struct rte_vfio_user_regions *reg;
+	struct rte_vfio_user_irq_info *irq;
+};
+
 struct iavf_emu_intr_info {
 	int enable;
 	int fd;
@@ -27,6 +34,14 @@ struct iavf_emu_intr {
 	struct iavf_emu_intr_info info[RTE_IAVF_EMU_MAX_INTR];
 };
 
+struct iavf_emu_adminQ {
+	uint32_t *ring_addr_lo;
+	uint32_t *ring_addr_hi;
+	uint32_t *ring_sz;
+	uint16_t db_size;
+	void *doorbell;
+};
+
 struct iavf_emu_lanQ {
 	uint16_t db_size;
 	void *doorbell;
@@ -34,14 +49,18 @@ struct iavf_emu_lanQ {
 
 struct iavf_emudev {
 	struct rte_emudev *edev;
+	struct iavf_emu_vfio_user *vfio;
 	/* Maximum LANQ queue pair that this emulated iavf has */
 	uint16_t max_lanqp;
 	/* Maximum LANQ queue pair number that back-end driver can use */
 	uint16_t max_be_lanqp;
 	unsigned int numa_node;
+	int ready;
 	char *sock_addr;
+	struct rte_iavf_emu_notify_ops *ops;
 	struct rte_iavf_emu_mem *mem;
 	struct iavf_emu_intr *intr;
+	struct iavf_emu_adminQ adq[RTE_IAVF_EMU_ADMINQ_NUM];
 	struct iavf_emu_lanQ *lanq;
 };
 
drivers/emu/iavf/iavf_emudev.c +11 -1
index a4cd2deb06..fbbe3d95a7 100644
@@ -6,7 +6,7 @@
 #include <rte_emudev.h>
 #include <rte_emudev_vdev.h>
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 #define EMU_IAVF_SOCK_ARG "sock"
 #define EMU_IAVF_QUEUES_ARG "queues"
@@ -170,10 +170,20 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 	iavf->max_lanqp = queues;
 	edev->priv_data = (void *)iavf;
 
+	ret = iavf_emu_register_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to register vfio user.\n");
+		ret = -1;
+		goto err_reg;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_reg:
+	iavf_emu_uninit_device(iavf);
 err_ndev:
 	rte_emudev_release(edev);
 err:
drivers/emu/iavf/iavf_vfio_user.c new file +384 -0

384 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h new file +16 -0
new file mode 100644
index 0000000000..aa2f3edc87
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IAVF_VFIO_USER_H
+#define _IAVF_VFIO_USER_H
+
+#include <rte_vfio_user.h>
+
+#include "iavf_emu_internal.h"
+
+int iavf_emu_register_vfio_user(struct iavf_emudev *dev);
+
+int iavf_emu_unregister_vfio_user(struct iavf_emudev *dev);
+
+#endif
drivers/emu/iavf/meson.build +3 -2
index 58c2a90383..4f651258c2 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
-sources = files('iavf_emu.c', 'iavf_emudev.c')
+sources = files('iavf_emu.c', 'iavf_vfio_user.c',
+	'iavf_emudev.c')
 
-deps += ['bus_vdev', 'emudev']
+deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
 headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h +17 -0
index 623c3c5d99..6de0989f0b 100644
@@ -40,4 +40,21 @@ struct rte_iavf_emu_mem {
 	struct rte_iavf_emu_mem_reg regions[RTE_IAVF_EMU_MAX_MEM_REGIONS];
 };
 
+struct rte_iavf_emu_notify_ops {
+	/* Device is ready */
+	int (*device_ready)(struct rte_emudev *dev);
+	/* Device is destroyed */
+	void (*device_destroy)(struct rte_emudev *dev);
+	/* Update device status */
+	int (*update_status)(struct rte_emudev *dev);
+	/* Start device */
+	int (*device_start)(struct rte_emudev *dev);
+	/* Stop device */
+	int (*device_stop)(struct rte_emudev *dev);
+	/* Lock or unlock data path */
+	int (*lock_dp)(struct rte_emudev *dev, int lock);
+	/* Reset device */
+	int (*reset_device)(struct rte_emudev *dev);
+};
+
 #endif
Signature · 1 line
2.17.1
#5 lore raw

Re: [dpdk-dev] [PATCH 4/8] emu/iavf: add vfio-user device register and unregister

Quoted 17 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Chenbo Xia
Sent: Friday, December 18, 2020 3:48 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: [dpdk-dev] [PATCH 4/8] emu/iavf: add vfio-user device register and unregister

This patch adds vfio-user APIs call in driver probe and remove. rte_vfio_user_register() and rte_vfio_user_unregister() are called to create/destroy a vfio-user device. Notify callbacks that libvfio_user defines are also implemented.

Quoted 6 lines

---

+static struct iavf_emudev *find_iavf_with_dev_id(int vfio_dev_id) {

It's better to change the function name to follow other function names' style. iavf_emu_xxx

Quoted 9 lines
drivers/emu/iavf/iavf_vfio_user.c new file +9 -0
+	struct iavf_emu_sock_list *list;
+	char sock_addr[PATH_MAX];
+	int ret;
+
+	ret = rte_vfio_get_sock_addr(vfio_dev_id, sock_addr,
+		sizeof(sock_addr));
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Can not find vfio device %d "
+			"sock_addr.\n", vfio_dev_id);

drivers/emu/iavf/iavf_vfio_user.c

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+		return NULL;
+	}
+
+	list = iavf_emu_find_sock_list(sock_addr);
+	if (!list) {
+		EMU_IAVF_LOG(ERR, "Can not find sock list.\n");
+		return NULL;
+	}
+
+	return (struct iavf_emudev *)list->emu_dev->priv_data; }

It's better to check if list->emu_dev is NULL first.

Quoted 23 lines
drivers/emu/iavf/iavf_vfio_user.c new file +23 -0
+
+static int iavf_emu_new_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+	int ret;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	dev->vfio->dev_id = vfio_dev_id;
+
+	ret = iavf_emu_setup_mem_table(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up memtable for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = iavf_emu_setup_irq(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up irq for "
+			"device %d", dev->vfio->dev_id);
+		return ret;

drivers/emu/iavf/iavf_vfio_user.c line 279

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	}
+
+	ret = iavf_emu_setup_queues(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up queues for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = dev->ops->device_ready(dev->edev);

Same as above, and please also check other functions, such as device_destroy...

Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c new file +5 -0
+	if (ret)
+		return ret;
+
+	dev->ready = 1;
+	return 0;

drivers/emu/iavf/iavf_vfio_user.c line 295

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+}
+
+static void iavf_emu_destroy_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return;
+
+	iavf_emu_reset_all_resources(dev);

Should we add 'dev->ready = 0' here?

Quoted 14 lines
drivers/emu/iavf/iavf_vfio_user.c new file +11 -0
+
+	dev->ops->device_destroy(dev->edev);
+}
+
 
 
 
+static int iavf_emu_lock_datapath(int vfio_dev_id, int lock) {
+	struct iavf_emudev *dev;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+

drivers/emu/iavf/iavf_vfio_user.c line 347

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	return dev->ops->lock_dp(dev->edev, lock); }
+
+static int iavf_emu_reset_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	iavf_emu_reset_all_resources(dev);

Should we add 'dev->ready = 0' here?

Quoted 4 lines
drivers/emu/iavf/iavf_vfio_user.c new file +4 -0
+
+	return dev->ops->reset_device(dev->edev);
+}
+
#6 lore raw

Re: [dpdk-dev] [PATCH 4/8] emu/iavf: add vfio-user device register and unregister

Hi Beilei,

Quoted 34 lines

-----Original Message-----
From: Xing, Beilei <[email protected]>
Sent: Thursday, January 7, 2021 3:19 PM
To: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: RE: [dpdk-dev] [PATCH 4/8] emu/iavf: add vfio-user device register and unregister

Quoted 10 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Chenbo Xia
Sent: Friday, December 18, 2020 3:48 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: [dpdk-dev] [PATCH 4/8] emu/iavf: add vfio-user device register and unregister

Quoted 3 lines

This patch adds vfio-user APIs call in driver probe and remove. rte_vfio_user_register() and rte_vfio_user_unregister() are called to create/destroy a vfio-user device. Notify callbacks that libvfio_user

defines are

Quoted 5 lines

also implemented.

---

Quoted 4 lines
Quoted 1 line

+static struct iavf_emudev *find_iavf_with_dev_id(int vfio_dev_id) {

It's better to change the function name to follow other function names' style. iavf_emu_xxx

OK, thanks!

Quoted 14 lines
Quoted 14 lines
drivers/emu/iavf/iavf_vfio_user.c new file +13 -0
+	struct iavf_emu_sock_list *list;
+	char sock_addr[PATH_MAX];
+	int ret;
+
+	ret = rte_vfio_get_sock_addr(vfio_dev_id, sock_addr,
+		sizeof(sock_addr));
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Can not find vfio device %d "
+			"sock_addr.\n", vfio_dev_id);
+		return NULL;
+	}
+
+	list = iavf_emu_find_sock_list(sock_addr);
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c

Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c new file +6 -0
+	if (!list) {
+		EMU_IAVF_LOG(ERR, "Can not find sock list.\n");
+		return NULL;
+	}
+
+	return (struct iavf_emudev *)list->emu_dev->priv_data; }

It's better to check if list->emu_dev is NULL first.

'list->emu_dev->priv_data' is already used in iavf_emu_find_sock_list. And based on the way we add to the list, emu_dev will not be NULL. But thanks to your comment, I noticed I should just return struct iavf_emudev in iavf_emu_find_sock_list. And the linked list may just store 'struct iavf_emudev' to make it easier.

Quoted 28 lines
Quoted 28 lines
drivers/emu/iavf/iavf_vfio_user.c new file +27 -0
+
+static int iavf_emu_new_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+	int ret;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	dev->vfio->dev_id = vfio_dev_id;
+
+	ret = iavf_emu_setup_mem_table(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up memtable for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = iavf_emu_setup_irq(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up irq for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = iavf_emu_setup_queues(dev);
+	if (ret) {
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c line 279

Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c new file +6 -0
+		EMU_IAVF_LOG(ERR, "Failed to set up queues for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = dev->ops->device_ready(dev->edev);

Same as above, and please also check other functions, such as device_destroy...

Yes! Will add check then.

Quoted 10 lines
Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +9 -0
+	if (ret)
+		return ret;
+
+	dev->ready = 1;
+	return 0;
+}
+
+static void iavf_emu_destroy_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c line 295

Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c new file +6 -0
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return;
+
+	iavf_emu_reset_all_resources(dev);

Should we add 'dev->ready = 0' here?

Yes. Will fix it.

Quoted 19 lines
Quoted 19 lines
drivers/emu/iavf/iavf_vfio_user.c new file +15 -0
+
+	dev->ops->device_destroy(dev->edev);
+}
+
 
 
 
+static int iavf_emu_lock_datapath(int vfio_dev_id, int lock) {
+	struct iavf_emudev *dev;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	return dev->ops->lock_dp(dev->edev, lock); }
+
+static int iavf_emu_reset_device(int vfio_dev_id) {
+	struct iavf_emudev *dev;
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c line 295

Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c new file +6 -0
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	iavf_emu_reset_all_resources(dev);

Should we add 'dev->ready = 0' here?

Yes. Will fix it.

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c new file +4 -0
+
+	return dev->ops->reset_device(dev->edev);
+}
+
author
#7 lore raw

PATCH 5/8 emu/iavf: add resource management and internal logic of iavf

In reply to Chenbo Xia

This patch adds the allocation and release of device resources. Device resources include PCI BARs' memory and interrupt related resources. Device internal logic is also added.

---
 drivers/emu/iavf/iavf_emu.c       |   1 +
 drivers/emu/iavf/iavf_emudev.c    |  20 +
 drivers/emu/iavf/iavf_vfio_user.c | 669 ++++++++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h |  41 ++
 drivers/emu/iavf/meson.build      |   8 +
 5 files changed, 739 insertions(+)
drivers/emu/iavf/iavf_emu.c +1 -0
index dfd9796920..c1a702d744 100644
@@ -18,6 +18,7 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_vfio_user(iavf);
 	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
drivers/emu/iavf/iavf_emudev.c +20 -0
index fbbe3d95a7..70cf558eef 100644
@@ -178,10 +178,30 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 		goto err_reg;
 	}
 
+	ret = iavf_emu_init_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to init vfio user.\n");
+		ret = -1;
+		goto err_init;
+	}
+
+	ret = iavf_emu_start_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to start vfio user.\n");
+		ret = -1;
+		goto err_start;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_start:
+	iavf_emu_uninit_vfio_user(iavf);
+err_init:
+	iavf_emu_unregister_vfio_user(iavf);
 err_reg:
 	iavf_emu_uninit_device(iavf);
 err_ndev:
drivers/emu/iavf/iavf_vfio_user.c +669 -0

697 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h +41 -0

53 lines not on this page · raw message

drivers/emu/iavf/meson.build +8 -0
index 4f651258c2..3cab2226b7 100644
@@ -1,6 +1,14 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
+librt = cc.find_library('rt', required: false)
+if not librt.found()
+	build = false
+	subdir_done()
+endif
+
+ext_deps += librt
+
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
 	'iavf_emudev.c')
 
Signature · 1 line
2.17.1
author
#8 lore raw

PATCH 6/8 emu/iavf: add emudev operations to fit in emudev framework

In reply to Chenbo Xia

This patch implements emudev opertions to make emulated iavf fit into rte_emudev framework. Lifecycle related and device resource related operations are both implemented.

---
 drivers/emu/iavf/iavf_emu.c     | 218 ++++++++++++++++++++++++++++++++
 drivers/emu/iavf/rte_iavf_emu.h |  59 +++++++++
 2 files changed, 277 insertions(+)
drivers/emu/iavf/iavf_emu.c +218 -0
index c1a702d744..9ad371ca98 100644
@@ -2,7 +2,72 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_malloc.h>
+#include <rte_emudev.h>
+
 #include "iavf_vfio_user.h"
+#include <iavf_type.h>
+
+static int iavf_emu_dev_start(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_start != NULL)
+		iavf->ops->device_start(dev);
+
+	return 0;
+}
+
+static void iavf_emu_dev_stop(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_stop != NULL)
+		iavf->ops->device_stop(dev);
+}
+
+static int iavf_emu_dev_configure(struct rte_emudev *dev,
+		struct rte_emudev_info *dev_conf)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf =
+		(struct rte_iavf_emu_config *)dev_conf->dev_priv;
+
+	if (!dev_conf->dev_priv)
+		return -EINVAL;
+
+	/* Currently emulated iavf does not support max_qp_num
+	 * and region num configuration
+	 */
+	if (dev->dev_info.max_qp_num != dev_conf->max_qp_num ||
+		dev->dev_info.region_num != dev_conf->region_num) {
+		EMU_IAVF_LOG(ERR,
+			"Configure max_qp_num/region num not supported\n");
+		return -ENOTSUP;
+	}
+
+	if (conf->qp_num >  RTE_MAX_QUEUES_PER_PORT ||
+		conf->qp_num > RTE_IAVF_EMU_MAX_QP_NUM) {
+		EMU_IAVF_LOG(ERR, "Queue pair num exceeds max\n");
+		return -EINVAL;
+	}
+
+	/* For now, we don't support device configure when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Configure failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf->max_be_lanqp = conf->qp_num;
+	return 0;
+}
 
 static int iavf_emu_dev_close(struct rte_emudev *dev)
 {
@@ -26,6 +91,159 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	return 0;
 }
 
+static int iavf_emu_get_dev_info(struct rte_emudev *dev,
+	rte_emudev_obj_t info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf = (struct rte_iavf_emu_config *)info;
+
+	if (!info)
+		return -EINVAL;
+
+	conf->qp_num = iavf->max_be_lanqp;
+	return 0;
+}
+
+static int iavf_emu_get_mem_table(struct rte_emudev *dev,
+	rte_emudev_mem_table_t *tb)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	*tb = iavf->mem;
+
+	return 0;
+}
+
+static int iavf_emu_get_queue_info(struct rte_emudev *dev, uint32_t queue,
+		struct rte_emudev_q_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (queue < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[queue];
+		uint64_t base, size;
+
+		if (adq->ring_addr_lo == NULL ||
+		    adq->ring_addr_hi == NULL ||
+		    adq->ring_sz == NULL)
+			return -1;
+		base = RTE_IAVF_EMU_32_TO_64(*adq->ring_addr_hi,
+			*adq->ring_addr_lo);
+		size = *adq->ring_sz;
+		info->base = base;
+		info->size = size;
+		info->doorbell_id = queue;
+		/* RX AdminQ should have IRQ vector 0 */
+		info->irq_vector = queue - 1;
+	} else {
+		info->base = 0;
+		info->size = 0;
+		info->doorbell_id = queue;
+		info->irq_vector = -1;
+	}
+
+	return 0;
+}
+
+static int iavf_emu_get_irq_info(struct rte_emudev *dev, uint32_t vector,
+		struct rte_emudev_irq_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct iavf_emu_intr *intr = iavf->intr;
+	struct iavf_emu_intr_info *intr_info = &intr->info[vector];
+
+	if (vector >= intr->intr_num)
+		return -EINVAL;
+
+	info->eventfd = intr_info->fd;
+	info->enable = intr_info->enable;
+	info->vector = vector;
+
+	return 0;
+}
+
+static int iavf_emu_get_db_info(struct rte_emudev *dev, uint32_t doorbell,
+		struct rte_emudev_db_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (doorbell < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[doorbell];
+
+		info->data.mem.base = (uint64_t)adq->doorbell;
+		info->data.mem.size = adq->db_size;
+	} else {
+		struct iavf_emu_lanQ *lanq =
+			&iavf->lanq[doorbell - RTE_IAVF_EMU_ADMINQ_NUM];
+
+		info->data.mem.base = (uint64_t)lanq->doorbell;
+		info->data.mem.size = lanq->db_size;
+	}
+
+	info->flag |= RTE_EMUDEV_DB_MEM;
+	info->id = doorbell;
+
+	return 0;
+}
+
+static int iavf_emu_subs_ev(struct rte_emudev *dev,
+	rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	iavf->ops = (struct rte_iavf_emu_notify_ops *)ev_chnl;
+
+	return 0;
+}
+
+static int iavf_emu_unsubs_ev(struct rte_emudev *dev,
+		rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if ((struct rte_iavf_emu_notify_ops *)ev_chnl == iavf->ops) {
+		iavf->ops = NULL;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
+static int iavf_emu_get_attr(struct rte_emudev *dev, const char *attr_name,
+		rte_emudev_attr_t attr)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_vfio_user_reg_info *info;
+	int ret = 0;
+
+	info = &iavf->vfio->reg->reg_info[0];
+
+	if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ASQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ATQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ARQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ARQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_RESET))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VFGEN_RSTAT;
+	else
+		ret = -EINVAL;
+
+	return ret;
+}
+
 struct rte_emudev_ops emu_iavf_ops = {
+	.dev_start = iavf_emu_dev_start,
+	.dev_stop = iavf_emu_dev_stop,
+	.dev_configure = iavf_emu_dev_configure,
 	.dev_close = iavf_emu_dev_close,
+	.dev_info_get = iavf_emu_get_dev_info,
+	.get_mem_table = iavf_emu_get_mem_table,
+	.get_queue_info = iavf_emu_get_queue_info,
+	.get_irq_info = iavf_emu_get_irq_info,
+	.get_db_info = iavf_emu_get_db_info,
+	.subscribe_event = iavf_emu_subs_ev,
+	.unsubscribe_event = iavf_emu_unsubs_ev,
+	.get_attr = iavf_emu_get_attr,
 };
drivers/emu/iavf/rte_iavf_emu.h +59 -0

84 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#9 lore raw

PATCH 7/8 test/emudev: introduce functional test

In reply to Chenbo Xia

This patch introduces functional test for emudev. The implementation of iavf emudev selftest is also added.

---
 app/test/meson.build                 |   5 +-
 app/test/test_emudev.c               |  29 +++++
 drivers/emu/iavf/iavf_emu.c          |   1 +
 drivers/emu/iavf/iavf_emu_internal.h |   1 +
 drivers/emu/iavf/iavf_emu_test.c     | 174 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   2 +-
 6 files changed, 210 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
app/test/meson.build +4 -1
index f5b15ac44c..b8b79bbc8b 100644
@@ -139,6 +139,7 @@ test_sources = files('commands.c',
 	'test_trace_register.c',
 	'test_trace_perf.c',
 	'test_vfio_user.c',
+	'test_emudev.c',
 	'test_version.c',
 	'virtual_pmd.c'
 )
@@ -176,7 +177,8 @@ test_deps = ['acl',
 	'stack',
 	'vfio_user',
 	'telemetry',
-	'timer'
+	'timer',
+	'emudev'
 ]
 
 # Each test is marked with flag true/false
@@ -327,6 +329,7 @@ driver_test_names = [
         'eventdev_selftest_octeontx',
         'eventdev_selftest_sw',
         'rawdev_autotest',
+        'emudev_autotest',
 ]
 
 dump_test_names = [
app/test/test_emudev.c new file +29 -0
new file mode 100644
index 0000000000..0dfce162ed
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_emudev.h>
+#include <rte_bus_vdev.h>
+
+#include "test.h"
+
+static int
+test_emudev_selftest_impl(const char *pmd, const char *opts)
+{
+	int ret = 0;
+
+	if (rte_emudev_get_dev_id(pmd) == -ENODEV)
+		ret = rte_vdev_init(pmd, opts);
+	if (ret)
+		return TEST_SKIPPED;
+
+	return rte_emudev_selftest(rte_emudev_get_dev_id(pmd));
+}
+
+static int
+test_emudev_selftest(void)
+{
+	return test_emudev_selftest_impl("emu_iavf", "sock=/tmp/sock1");
+}
+
+REGISTER_TEST_COMMAND(emudev_autotest, test_emudev_selftest);
drivers/emu/iavf/iavf_emu.c +1 -0
index 9ad371ca98..88bf2bdf94 100644
@@ -246,4 +246,5 @@ struct rte_emudev_ops emu_iavf_ops = {
 	.subscribe_event = iavf_emu_subs_ev,
 	.unsubscribe_event = iavf_emu_unsubs_ev,
 	.get_attr = iavf_emu_get_attr,
+	.dev_selftest = iavf_emu_selftest,
 };
drivers/emu/iavf/iavf_emu_internal.h +1 -0
index 10197c00ba..1ac7f96566 100644
@@ -65,4 +65,5 @@ struct iavf_emudev {
 };
 
 void iavf_emu_uninit_device(struct iavf_emudev *dev);
+int iavf_emu_selftest(uint16_t dev_id);
 #endif
drivers/emu/iavf/iavf_emu_test.c new file +174 -0

174 lines not on this page · raw message

drivers/emu/iavf/meson.build +1 -1
index 3cab2226b7..613783e407 100644
@@ -10,7 +10,7 @@ endif
 ext_deps += librt
 
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
-	'iavf_emudev.c')
+	'iavf_emu_test.c', 'iavf_emudev.c')
 
 deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
Signature · 1 line
2.17.1
author
#10 lore raw

PATCH 8/8 doc: update release notes for iavf emudev driver

In reply to Chenbo Xia

Update release notes for emulated iavf driver.

---
 doc/guides/rel_notes/release_21_02.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_21_02.rst +5 -1
index 3d26b6b580..b310b67b7d 100644
@@ -67,7 +67,7 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
-* **Added emudev Library.**
+* **Added emudev Library and first emudev driver.**
 
   Added an experimental library ``librte_emudev`` to provide device abstraction
   for an emulated device.
@@ -77,6 +77,10 @@ New Features
   crypto and etc.). It can be attached to another data path driver (e.g, ethdev
   driver) to leverage the high performance of DPDK data path driver.
 
+  The first emudev driver is also introduced, which emulates software iavf
+  devices. This driver emulates all iavf device behavior except data path
+  handling.
+
   See :doc:`../prog_guide/emudev` for more information.
 
 Removed Items
Signature · 1 line
2.17.1
#11 lore raw
In reply to Chenbo Xia
Quoted 9 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 13 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

Signature · 1 line
David Marchand
#12 lore raw

Hi David,

Quoted 31 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 5 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 8 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 8 lines
Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

implementing

Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its data path handling is). A first try on iavf spec shows better performance than virtio in our simple tests.

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

#13 lore raw

Hi Chenbo,

Quoted 37 lines · Chenbo

Hi David,

Quoted 35 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 5 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 8 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

implementing

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its data path handling is). A first try on iavf spec shows better performance than virtio in our simple tests.

That's interesting! How big is the performance difference? And how do we explain it?

If there are improvements that could be done in the Virtio specification, it would be great to know and work on their implementations. It worries me a bit that every one is coming with his new device emulation every release, making things like live- migration difficult to achieve in the future.

Regards,
Maxime

Quoted 8 lines

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

#14 lore raw
Quoted 49 lines · Maxime Coquelin

Hi Chenbo,

Quoted 37 lines · Chenbo

Hi David,

Quoted 35 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 5 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 8 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

implementing

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its data path handling is). A first try on iavf spec shows better performance than virtio in our simple tests.

Quoted 8 lines

That's interesting! How big is the performance difference? And how do we explain it?

If there are improvements that could be done in the Virtio specification, it would be great to know and work on their implementations. It worries me a bit that every one is coming with his new device emulation every release, making things like live- migration difficult to achieve in the future.

I did a quick review of the IAVF emudev driver to understand what other factors than ring layout could explain a performance gain.

My understanding is that part of the performance gain may come from following things that are supported/implemented in Vhost-user backend and not in IAVF driver:
1. Memory hotplug. It seems the datapath is not safe against memory hotplug in the VM, which causes the memory tables to be updated asynchronously from the datapath. In order to support it in Vhost-user library, we had to introduce locks to ensure the datapath isn't accessing the shared memory while it is being remapped.

2. Live-migration. This feature does not seem supported in the driver, as I could not find dirty pages tracking mechanism. On Vhost-user side, supporting implies adding more branch conditions in the hot path, even when it is not used.

3. No IOMMU support. Same here, this is supported in Vhost-user library, and adding its support in IAVF driver would introduce some more branches in the hot path even when not used.

4. Out of bound memory accesses checks. While rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested length is mapped, the data path does not use it. It does not even ensure the translated address is non-NULL. It makes it trivial for a malicious guest to make the hypervisor's vSwitch to crash by simply passing random values as buffer's address and length. Fixing it is trivial, but it will add several more checks and loops (if a buffer is spanned across two pages) in the hot path.

Other than that, there is for sure a performance gain due to all the features Virtio-net supports that we have to check and handle in the hotpath, like indirect descriptors or mergeable buffers for example.

Best regards,
Maxime

Quoted 11 lines

Regards,
Maxime

Quoted 8 lines

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

#15 lore raw

Hi Maxime,

Quoted 76 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Monday, December 21, 2020 8:02 PM
To: Xia, Chenbo <[email protected]>; Thomas Monjalon <[email protected]>; David Marchand <[email protected]>
Cc: dev <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 46 lines · Maxime Coquelin

Hi Chenbo,

Quoted 43 lines · Chenbo

Hi David,

Quoted 31 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 5 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 8 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 8 lines
Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

implementing

Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its data

path

Quoted 11 lines
Quoted 2 lines

handling is). A first try on iavf spec shows better performance than virtio in our simple tests.

That's interesting! How big is the performance difference? And how do we explain it?

If there are improvements that could be done in the Virtio specification, it would be great to know and work on their implementations. It worries me a bit that every one is coming with his new device emulation every release, making things like live- migration difficult to achieve in the future.

I did a quick review of the IAVF emudev driver to understand what other factors than ring layout could explain a performance gain.

Quoted 8 lines

My understanding is that part of the performance gain may come from following things that are supported/implemented in Vhost-user backend and not in IAVF driver:
1. Memory hotplug. It seems the datapath is not safe against memory hotplug in the VM, which causes the memory tables to be updated asynchronously from the datapath. In order to support it in Vhost-user library, we had to introduce locks to ensure the datapath isn't accessing the shared memory while it is being remapped.

I think now it uses the similar way that vhost-user does.

First, in the vfio-user patch series, we introduce a callback lock_dp to lock the data path when messages like DMA MAP/UNMAP come. It will lock datapath in our data path driver.

Note that the data path handling is in our data path driver:
http://patchwork.dpdk.org/cover/85500/

For modular design, iavf_emu driver emulates the device but the iavf back-end driver does data path handling.

Quoted 5 lines

2. Live-migration. This feature does not seem supported in the driver, as I could not find dirty pages tracking mechanism. On Vhost-user side, supporting implies adding more branch conditions in the hot path, even when it is not used.

Yes, we don't support this now in this version. And yes, when we support this feature, it will introduce complexity in data path.

Quoted 4 lines

3. No IOMMU support. Same here, this is supported in Vhost-user library, and adding its support in IAVF driver would introduce some more branches in the hot path even when not used.

Yes, vIOMMU is not fully supported in vfio-user spec for now and I also agree when we have to support it, it will slow down the data path.

Quoted 4 lines

4. Out of bound memory accesses checks. While rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested length is mapped, the data path does not use it. It does not even ensure

In fact, it uses it 😊. I think you may miss our data path driver. Here's a
use: http://patchwork.dpdk.org/patch/85504/

Quoted 5 lines

the translated address is non-NULL. It makes it trivial for a malicious guest to make the hypervisor's vSwitch to crash by simply passing random values as buffer's address and length. Fixing it is trivial, but it will add several more checks and loops (if a buffer is spanned across two pages) in the hot path.

I don't quite understand this one. First, rte_iavf_emu_get_dma_vaddr() is the only way to translate address. And I think this function will ensure the input address is valid. Looking at the vhost side, vhost_iova_to_vva() does similar things when vIOMMU is not used. Do I miss something? Just correct me if I am wrong.

Quoted 4 lines

Other than that, there is for sure a performance gain due to all the features Virtio-net supports that we have to check and handle in the hotpath, like indirect descriptors or mergeable buffers for example.

I think iavf has similar features like indirect and mergeable to recv/xmit large pkts. But I believe there will be some feature difference between iavf and virtio/vhost.

I think you are correct that for this version, it's not fair to compare virtio/vhost with iavf back-end because iavf back-end has not supported some features, and besides, we have not optimized the data path of iavf back-end. We expect the performance of iavf back-end in the same level of virtio 1.1 and hopefully better because of the ring layout. But let's see when we can do complete performance analysis 😊.

Thanks!
Chenbo

Quoted 15 lines

Best regards,
Maxime

Quoted 11 lines

Regards,
Maxime

Quoted 8 lines

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

#16 lore raw

Hi Chenbo,

Thanks for the detailed reply.

Quoted 93 lines · Chenbo

Hi Maxime,

Quoted 76 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Monday, December 21, 2020 8:02 PM
To: Xia, Chenbo <[email protected]>; Thomas Monjalon <[email protected]>; David Marchand <[email protected]>
Cc: dev <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 46 lines · Maxime Coquelin

Hi Chenbo,

Quoted 43 lines · Chenbo

Hi David,

Quoted 31 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 5 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 8 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 8 lines
Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

implementing

Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its data

path

Quoted 11 lines
Quoted 2 lines

handling is). A first try on iavf spec shows better performance than virtio in our simple tests.

That's interesting! How big is the performance difference? And how do we explain it?

If there are improvements that could be done in the Virtio specification, it would be great to know and work on their implementations. It worries me a bit that every one is coming with his new device emulation every release, making things like live- migration difficult to achieve in the future.

I did a quick review of the IAVF emudev driver to understand what other factors than ring layout could explain a performance gain.

Quoted 8 lines

My understanding is that part of the performance gain may come from following things that are supported/implemented in Vhost-user backend and not in IAVF driver:
1. Memory hotplug. It seems the datapath is not safe against memory hotplug in the VM, which causes the memory tables to be updated asynchronously from the datapath. In order to support it in Vhost-user library, we had to introduce locks to ensure the datapath isn't accessing the shared memory while it is being remapped.

I think now it uses the similar way that vhost-user does.

First, in the vfio-user patch series, we introduce a callback lock_dp to lock the data path when messages like DMA MAP/UNMAP come. It will lock datapath in our data path driver.

Quoted 5 lines

Note that the data path handling is in our data path driver:
http://patchwork.dpdk.org/cover/85500/

For modular design, iavf_emu driver emulates the device but the iavf back-end driver does data path handling.

My analysis was actually based on the data path driver series.

My point was that iavfbe_recv_pkts() and iavfbe_xmit_pkts() are not safe against asynchronous changes like memory table updates.

As far as I can see, the access_lock aren't taken by these functions, so if for example a memory table update happen during the these functions execution, it could lead to undefined behaviour. Only things checked there is whether the queue is enabled when entering the function, but this value can be changed right after having being checked.

For example, in Vhost-user lib, we protected rte_vhost_dequeue_burst() and rte_vhost_enqueue_burst() with a spinlock. Note that this spinlock is per-virtqueue in order to avoid contention between the different queues.

Quoted 18 lines
Quoted 5 lines

2. Live-migration. This feature does not seem supported in the driver, as I could not find dirty pages tracking mechanism. On Vhost-user side, supporting implies adding more branch conditions in the hot path, even when it is not used.

Yes, we don't support this now in this version. And yes, when we support this feature, it will introduce complexity in data path.

Quoted 4 lines

3. No IOMMU support. Same here, this is supported in Vhost-user library, and adding its support in IAVF driver would introduce some more branches in the hot path even when not used.

Yes, vIOMMU is not fully supported in vfio-user spec for now and I also agree when we have to support it, it will slow down the data path.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

4. Out of bound memory accesses checks. While rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested length is mapped, the data path does not use it. It does not even ensure

In fact, it uses it 😊. I think you may miss our data path driver. Here's a
use: http://patchwork.dpdk.org/patch/85504/

Sorry, I was not clear. What I meant is that rte_iavf_emu_get_dma_vaddr() is indeed used, but its outputs aren't checked properly.

Quoted 11 lines
Quoted 5 lines

the translated address is non-NULL. It makes it trivial for a malicious guest to make the hypervisor's vSwitch to crash by simply passing random values as buffer's address and length. Fixing it is trivial, but it will add several more checks and loops (if a buffer is spanned across two pages) in the hot path.

I don't quite understand this one. First, rte_iavf_emu_get_dma_vaddr() is the only way to translate address. And I think this function will ensure the input address is valid. Looking at the vhost side, vhost_iova_to_vva() does similar things when vIOMMU is not used. Do I miss something? Just correct me if I am wrong.

I think rte_iavf_emu_get_dma_vaddr() is good, the issue is the way it is used in iavfbe_recv_pkts() and iavfbe_xmit_pkts().

First the returned address (desc_addr) is not checked. So if the start address of the buffer is out of guest memory ranges, the host app will crash with a segmentation fault. It can happen in case of bug in the guest driver, or with a maliciously crafted descriptor.

Then, rte_iavf_emu_get_dma_vaddr() len parameter is a pointer. The caller has to pass the length of the buffer it wants to map. Once rte_iavf_emu_get_dma_vaddr() is updated with the contiguous length that is mapped.

The caller needs to check whether all requested length is mapped, and loop until it is all mapped:
http://code.dpdk.org/dpdk/latest/source/lib/librte_vhost/virtio_net.c#L484

This is required because a buffer contiguous in IOVA (guest physical address space in this case) is not necessarily contiguous in Host virtual address space. Also, this is required to be safe against untrusted guests, as a malicious guest could pass ~0ULL in the descriptor buffer len of the Tx path to crash the host application or overwrite its memory.

Currently, in the driver, descriptor buf len is passed to the DMA map function, but its value is not checked afterward, so it only ensure first byte is mapped. For Tx, 1 bytes length is requested at DMA map, so also only first byte of the buffer is ensured to be mapped (if desc_addr is not NULL, which is not checked).

Quoted 8 lines
Quoted 4 lines

Other than that, there is for sure a performance gain due to all the features Virtio-net supports that we have to check and handle in the hotpath, like indirect descriptors or mergeable buffers for example.

I think iavf has similar features like indirect and mergeable to recv/xmit large pkts. But I believe there will be some feature difference between iavf and virtio/vhost.

Sure, but I meant it might not be necessary to implement all those features (if they are optional in the spec) in your driver, just that in the case of Vhost we have a legacy to support.

Quoted 5 lines

I think you are correct that for this version, it's not fair to compare virtio/vhost with iavf back-end because iavf back-end has not supported some features, and besides, we have not optimized the data path of iavf back-end. We expect the performance of iavf back-end in the same level of virtio 1.1 and hopefully better because of the ring layout. But let's see when we can do complete performance analysis 😊.

Thanks!
Maxime

Quoted 19 lines

Thanks!
Chenbo

Quoted 15 lines

Best regards,
Maxime

Quoted 11 lines

Regards,
Maxime

Quoted 8 lines

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

#17 lore raw

Hi Maxime,

Quoted 134 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, December 22, 2020 4:49 PM
To: Xia, Chenbo <[email protected]>; Thomas Monjalon <[email protected]>; David Marchand <[email protected]>
Cc: dev <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 0/8] Introduce emudev library and iavf emudev driver

Hi Chenbo,

Thanks for the detailed reply.

Quoted 6 lines · Chenbo

Hi Maxime,

Quoted 4 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Monday, December 21, 2020 8:02 PM
To: Xia, Chenbo <[email protected]>; Thomas Monjalon

<[email protected]>;

Quoted 2 lines
Quoted 2 lines

David Marchand <[email protected]>
Cc: dev <[email protected]>; Stephen Hemminger <[email protected]>;

Liang,

Quoted 3 lines
Quoted 3 lines

Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 0/8] Introduce emudev library and iavf

emudev

Quoted 27 lines
Quoted 27 lines

driver

Quoted 22 lines · Maxime Coquelin

Hi Chenbo,

Quoted 19 lines · Chenbo

Hi David,

Quoted 17 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, December 18, 2020 5:54 PM
To: Xia, Chenbo <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Stephen Hemminger <[email protected]>; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>
Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver

Quoted 2 lines · Chenbo Xia

This series introduces a new device abstraction called emudev for

emulated

Quoted 2 lines

devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual

Function

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

(iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-

user

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction

will

Quoted 3 lines

be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

implementing

Quoted 2 lines

data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Since you mention performance, how does it compare to vhost-user/virtio?

I think it depends on the device specification (i.e., how complex its

data

Quoted 2 lines
Quoted 2 lines

path

Quoted 1 line
Quoted 1 line

handling is). A first try on iavf spec shows better performance than

virtio

Quoted 23 lines
Quoted 14 lines
Quoted 10 lines
Quoted 1 line

in our simple tests.

That's interesting! How big is the performance difference? And how do we explain it?

If there are improvements that could be done in the Virtio specification, it would be great to know and work on their implementations. It worries me a bit that every one is coming with his new device emulation every release, making things like live- migration difficult to achieve in the future.

I did a quick review of the IAVF emudev driver to understand what other factors than ring layout could explain a performance gain.

Quoted 8 lines

My understanding is that part of the performance gain may come from following things that are supported/implemented in Vhost-user backend and not in IAVF driver:
1. Memory hotplug. It seems the datapath is not safe against memory hotplug in the VM, which causes the memory tables to be updated asynchronously from the datapath. In order to support it in Vhost-user library, we had to introduce locks to ensure the datapath isn't accessing the shared memory while it is being remapped.

Quoted 3 lines

I think now it uses the similar way that vhost-user does.

First, in the vfio-user patch series, we introduce a callback lock_dp to

lock

Quoted 7 lines

the data path when messages like DMA MAP/UNMAP come. It will lock datapath in our data path driver.

Note that the data path handling is in our data path driver:
http://patchwork.dpdk.org/cover/85500/

For modular design, iavf_emu driver emulates the device but the iavf back-

end

Quoted 1 line

driver does data path handling.

My analysis was actually based on the data path driver series.

My point was that iavfbe_recv_pkts() and iavfbe_xmit_pkts() are not safe against asynchronous changes like memory table updates.

As far as I can see, the access_lock aren't taken by these functions, so if for example a memory table update happen during the these functions execution, it could lead to undefined behaviour. Only things checked there is whether the queue is enabled when entering the function, but this value can be changed right after having being checked.

Quoted 5 lines

For example, in Vhost-user lib, we protected rte_vhost_dequeue_burst() and rte_vhost_enqueue_burst() with a spinlock. Note that this spinlock is per-virtqueue in order to avoid contention between the different queues.

Oops, I didn't realize the data path driver missed that. And yes, the data path driver should do that like you said.

Quoted 70 lines
Quoted 7 lines
Quoted 5 lines

2. Live-migration. This feature does not seem supported in the driver, as I could not find dirty pages tracking mechanism. On Vhost-user side, supporting implies adding more branch conditions in the hot path, even when it is not used.

Yes, we don't support this now in this version. And yes, when we support

this

Quoted 8 lines

feature, it will introduce complexity in data path.

Quoted 4 lines

3. No IOMMU support. Same here, this is supported in Vhost-user library, and adding its support in IAVF driver would introduce some more branches in the hot path even when not used.

Yes, vIOMMU is not fully supported in vfio-user spec for now and I also

agree

Quoted 9 lines

when we have to support it, it will slow down the data path.

Quoted 4 lines

4. Out of bound memory accesses checks. While rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested length is mapped, the data path does not use it. It does not even ensure

In fact, it uses it 😊. I think you may miss our data path driver. Here's a
use: http://patchwork.dpdk.org/patch/85504/

Sorry, I was not clear. What I meant is that rte_iavf_emu_get_dma_vaddr() is indeed used, but its outputs aren't checked properly.

Quoted 7 lines
Quoted 5 lines

the translated address is non-NULL. It makes it trivial for a malicious guest to make the hypervisor's vSwitch to crash by simply passing random values as buffer's address and length. Fixing it is trivial, but it will add several more checks and loops (if a buffer is spanned across two pages) in the hot path.

I don't quite understand this one. First, rte_iavf_emu_get_dma_vaddr() is

the

Quoted 1 line

only way to translate address. And I think this function will ensure the

input

Quoted 1 line

address is valid. Looking at the vhost side, vhost_iova_to_vva() does

similar

Quoted 2 lines

things when vIOMMU is not used. Do I miss something? Just correct me if I am wrong.

I think rte_iavf_emu_get_dma_vaddr() is good, the issue is the way it is used in iavfbe_recv_pkts() and iavfbe_xmit_pkts().

First the returned address (desc_addr) is not checked. So if the start address of the buffer is out of guest memory ranges, the host app will crash with a segmentation fault. It can happen in case of bug in the guest driver, or with a maliciously crafted descriptor.

Then, rte_iavf_emu_get_dma_vaddr() len parameter is a pointer. The caller has to pass the length of the buffer it wants to map. Once rte_iavf_emu_get_dma_vaddr() is updated with the contiguous length that is mapped.

The caller needs to check whether all requested length is mapped, and loop until it is all mapped:
http://code.dpdk.org/dpdk/latest/source/lib/librte_vhost/virtio_net.c#L484

This is required because a buffer contiguous in IOVA (guest physical address space in this case) is not necessarily contiguous in Host virtual address space. Also, this is required to be safe against untrusted guests, as a malicious guest could pass ~0ULL in the descriptor buffer len of the Tx path to crash the host application or overwrite its memory.

Quoted 6 lines

Currently, in the driver, descriptor buf len is passed to the DMA map function, but its value is not checked afterward, so it only ensure first byte is mapped. For Tx, 1 bytes length is requested at DMA map, so also only first byte of the buffer is ensured to be mapped (if desc_addr is not NULL, which is not checked).

Correct! I also missed that rte_iavf_emu_get_dma_vaddr() is not properly used in data path driver. For the lock issue and this, @Wu, Jingjing could we fix in V2?

Thanks for the good catches!
Chenbo

Quoted 45 lines
Quoted 8 lines
Quoted 4 lines

Other than that, there is for sure a performance gain due to all the features Virtio-net supports that we have to check and handle in the hotpath, like indirect descriptors or mergeable buffers for example.

I think iavf has similar features like indirect and mergeable to recv/xmit large pkts. But I believe there will be some feature difference between iavf and virtio/vhost.

Sure, but I meant it might not be necessary to implement all those features (if they are optional in the spec) in your driver, just that in the case of Vhost we have a legacy to support.

Quoted 1 line

I think you are correct that for this version, it's not fair to compare

virtio/vhost

Quoted 1 line

with iavf back-end because iavf back-end has not supported some features,

and besides,

Quoted 1 line

we have not optimized the data path of iavf back-end. We expect the

performance of

Quoted 1 line

iavf back-end in the same level of virtio 1.1 and hopefully better because

of the ring

Quoted 1 line

layout. But let's see when we can do complete performance analysis 😊.

Thanks!
Maxime

Quoted 19 lines

Thanks!
Chenbo

Quoted 15 lines

Best regards,
Maxime

Quoted 11 lines

Regards,
Maxime

Quoted 8 lines

Thanks!
Chenbo

Quoted 4 lines

--
David Marchand

author
#18 lore raw

v2 cover Introduce emudev library and iavf emudev driver

In reply to Chenbo Xia

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Design overview
---------------

                    +---------------------------------------+
                    |   +---------------+    +-----------+  |
                    |   |  iavf_emudev  |<-->| data path |  |
                    |   |    driver     |    |   driver  |  |
                    |   +---------------+    +-----------+  |
                    |           |                           |
                    | --------------------------- VDEV BUS  |
                    |           |                           |
                    |   +---------------+                   |
+--------------+    |   | vdev:         |                   |
| +----------+ |    |   | /path/to/vfio |                   |
| | Generic  | |    |   +---------------+                   |
| | vfio-dev | |    |           |                           |
| +----------+ |    |           |                           |
| +----------+ |    |      +----------+                     |
| | vfio-user| |    |      | vfio-user|                     |
| | client   | |<---|----->| server   |                     |
| +----------+ |    |      +----------+                     |
| QEMU/DPDK    |    | DPDK                                  |
+--------------+    +---------------------------------------+

- Generic vfio-dev/vfio-user client/vfio-user server
Above concepts are all introduced in librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

- vdev:/path/to/vfio.
It binds to vdev bus driver. The vdev device is defined by DPDK applications through command line as '--vdev=emu_iavf, path=/path/to/socket' in iavf_emudev case. Parameters in command line include device name (emu_iavf) which is used to identify corresponding driver (in this case, iavf_emudev driver), path=/path/to/socket which is used to open the transport interface to vfio-user client in QEMU/DPDK.

- data path driver.
The data path handling is splited to another standalone driver for modular design.

Why not rawdev for emulated device
----------------------------------
Instead of introducing new class emudev, emulated device could be presented as rawdev. However, existing rawdev APIs cannot meet the requirements of emulated device. There are three API categories for emudev. They are emudev device lifecycle management, backend facing APIs, and emudev device provider facing APIs respectively. Existing rawdev APIs could only cover lifecycle management APIs and some of backend facing APIs. Other APIs, even if added to rawdev API are not required by other rawdev applications.

----------------------------------
v2:
- fix driver meson build file

Chenbo Xia (8):
lib: introduce emudev library
doc: add emudev library guide
emu: introduce emulated iavf driver
emu/iavf: add vfio-user device register and unregister emu/iavf: add resource management and internal logic of iavf emu/iavf: add emudev operations to fit in emudev framework test/emudev: introduce functional test
doc: update release notes for iavf emudev driver

 MAINTAINERS                            |   12 +
 app/test/meson.build                   |    5 +-
 app/test/test_emudev.c                 |   29 +
 doc/guides/prog_guide/emudev.rst       |  122 +++
 doc/guides/prog_guide/index.rst        |    1 +
 doc/guides/rel_notes/release_21_02.rst |   16 +
 drivers/emu/iavf/iavf_emu.c            |  250 ++++++
 drivers/emu/iavf/iavf_emu_internal.h   |   69 ++
 drivers/emu/iavf/iavf_emu_test.c       |  174 ++++
 drivers/emu/iavf/iavf_emudev.c         |  237 ++++++
 drivers/emu/iavf/iavf_vfio_user.c      | 1053 ++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h      |   57 ++
 drivers/emu/iavf/meson.build           |   17 +
 drivers/emu/iavf/rte_iavf_emu.h        |  119 +++
 drivers/emu/iavf/version.map           |    3 +
 drivers/emu/meson.build                |    6 +
 drivers/meson.build                    |    1 +
 lib/librte_emudev/meson.build          |    5 +
 lib/librte_emudev/rte_emudev.c         |  486 +++++++++++
 lib/librte_emudev/rte_emudev.h         |  410 +++++++++
 lib/librte_emudev/rte_emudev_vdev.h    |   53 ++
 lib/librte_emudev/version.map          |   27 +
 lib/meson.build                        |    2 +-
 23 files changed, 3152 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 doc/guides/prog_guide/emudev.rst
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
Signature · 1 line
2.17.1
author
#19 lore raw

PATCH v2 1/8 lib: introduce emudev library

This patch introduces the emudev library. Emudev library is used to abstract an emulated device, whose type could be general (e.g., network, crypto and etc.). Several device-level APIs are implemented to use or manipulate the device. It can be attached to another data path driver (e.g., ethdev driver) to plug in its high performance data path.

---
 MAINTAINERS                         |   5 +
 lib/librte_emudev/meson.build       |   5 +
 lib/librte_emudev/rte_emudev.c      | 486 ++++++++++++++++++++++++++++
 lib/librte_emudev/rte_emudev.h      | 410 +++++++++++++++++++++++
 lib/librte_emudev/rte_emudev_vdev.h |  53 +++
 lib/librte_emudev/version.map       |  27 ++
 lib/meson.build                     |   2 +-
 7 files changed, 987 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
MAINTAINERS +5 -0
index 5fb4880758..1b395e181d 100644
@@ -1545,6 +1545,11 @@ M: Chenbo Xia <[email protected]>
 M: Xiuchun Lu <[email protected]>
 F: lib/librte_vfio_user/
 
+Emudev - EXPERIMENTAL
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: lib/librte_emudev/
+
 Test Applications
 -----------------
 
lib/librte_emudev/meson.build new file +5 -0
new file mode 100644
index 0000000000..4e16cecbaf
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('rte_emudev.c')
+headers = files('rte_emudev.h', 'rte_emudev_vdev.h')
lib/librte_emudev/rte_emudev.c new file +486 -0

486 lines not on this page · raw message

lib/librte_emudev/rte_emudev.h new file +410 -0

410 lines not on this page · raw message

lib/librte_emudev/rte_emudev_vdev.h new file +53 -0

53 lines not on this page · raw message

lib/librte_emudev/version.map new file +27 -0
new file mode 100644
index 0000000000..f800b4c21c
@@ -0,0 +1,27 @@
+EXPERIMENTAL {
+	global:
+
+	rte_emudev_allocate;
+	rte_emudev_release;
+	rte_emudev_allocated;
+	rte_emudev_start;
+	rte_emudev_stop;
+	rte_emudev_configure;
+	rte_emudev_close;
+	rte_emudev_subscribe_event;
+	rte_emudev_unsubscribe_event;
+	rte_emudev_count;
+	rte_emudev_get_dev_id;
+	rte_emudev_get_dev_info;
+	rte_emudev_get_mem_table;
+	rte_emudev_get_queue_info;
+	rte_emudev_get_irq_info;
+	rte_emudev_get_db_info;
+	rte_emudev_set_attr;
+	rte_emudev_get_attr;
+	rte_emudev_region_map;
+	rte_emudev_selftest;
+	rte_emudev_is_valid_id;
+
+	local: *;
+};
lib/meson.build +1 -1
index b7fbfcc95b..6dd07fb73e 100644
@@ -28,7 +28,7 @@ libraries = [
 	'rib', 'reorder', 'sched', 'security', 'stack', 'vhost',
 	# ipsec lib depends on net, crypto and security
 	'ipsec',
-	'vfio_user',
+	'vfio_user', 'emudev',
 	#fib lib depends on rib
 	'fib',
 	# add pkt framework libs which use other libs from above
Signature · 1 line
2.17.1
author
#20 lore raw

PATCH v2 2/8 doc: add emudev library guide

In reply to Chenbo Xia

Add emudev library guide and update release notes.

---
 doc/guides/prog_guide/emudev.rst       | 122 +++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/rel_notes/release_21_02.rst |  12 +++
 3 files changed, 135 insertions(+)
 create mode 100644 doc/guides/prog_guide/emudev.rst
doc/guides/prog_guide/emudev.rst new file +122 -0
new file mode 100644
index 0000000000..91ad520de7
@@ -0,0 +1,122 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Emulated Device Library
+=================
+
+Introduction
+------------
+
+The DPDK Emudev library is an abstraction for emulated device. This library
+provides a generic set of APIs for device provider, data path provider and
+applications to use.
+
+A device provider could be implemented as a driver on vdev bus. It should
+expose itself as an emudev for applications to use. It is responsible for the
+device resource management and the device's internal logic. All specifics of a
+device, except data path handling, should be implemented in the device
+provider. The device provider uses emudev APIs mainly for create/destroy an
+emudev instance. The device provider should also use a tranport to communicate
+with device consumer (e.g., virtual machine monitor or container). A potential
+choice could be vfio-user library, which implements the vfio-user protocol for
+emulating devices outside of a virtual machine monitor.
+
+A data path provider could be implemented as any type of driver on vdev bus.
+If the device you want to emulate is a network device, you could implement
+it as an ethdev driver. It is responsible for all data path handling. The data
+path provider uses emudev APIs mainly for getting device-related information
+from the device provider.
+
+Applications uses emudev APIs for device lifecycle management and configuration.
+
+Design
+------------
+
+Some key objects are designed in emudev.
+
+  ``Regions`` are the device layout exposed to the data path provider.
+
+  ``Queues`` are the data path queues that the data path provider needs. Queue
+  information includes queue base address, queue size, queue-related doorbell
+  and interrupt information.
+
+  ``Memory Table`` is the DMA mapping table. The data path provider could use
+  it to perform DMA read/write on device consumer's memory.
+
+Information of above key objects could be acquired through emudev APIs. The
+following will introduce the emudev APIs which are used by data path provider
+and applications. The APIs for device provider to use are allocate/release APIs
+and will not be listed because it's similar to other device abstraction.
+
+There are five categories of APIs:
+
+1. Lifecycle management
+
+* ``rte_emu_dev_start(dev_id)``
+* ``rte_emu_dev_stop(dev_id)``
+* ``rte_emu_dev_configure(dev_id)``
+* ``rte_emu_dev_close(dev_id)``
+
+  Above APIs are respectively for device start/stop/configure/close and mainly
+  for applications to use.
+
+  ``dev_id`` is the emudev device ID.
+
+2. Notification
+
+* ``rte_emu_subscribe_event(dev_id, ev_chnl)``
+* ``rte_emu_unsubscribe_event(dev_id, ev_chnl)``
+
+  Above APIs are for data path provider and applications to register events.
+  The mechanism of event notification could be different in different device
+  providers. A possbile implementation could be event callbacks.
+
+  ``ev_chnl`` is the event channel pointer. The definition varies between
+  different devices.
+
+3. Region-related
+
+* ``rte_emu_region_map(dev_id, index, region_size, base_addr)``
+* ``rte_emu_get_attr(dev_id, attr_name, attr)``
+* ``rte_emu_set_attr(dev_id, attr_name, attr)``
+
+  Above APIs are for data path provider and applications to read/write regions.
+  ``rte_emu_region_map`` is for directly mapping the region and use the mapped
+  address to read/write it. ``rte_emu_get_attr`` and ``rte_emu_set_attr`` are
+  respectively for getting/setting certain attributes in all regions.
+
+  Applications will set attributes or write regions for device configuration.
+
+  In ``rte_emu_region_map``:
+  - ``index`` is the region index.
+  - ``region_size`` is for saving the size of mapped region.
+  - ``base_addr`` is for saving the address of mapped region.
+
+  In ``rte_emu_get_attr`` and ``rte_emu_set_attr``:
+  - ``attr_name`` is the name of attribute. Note that attribute names are aligned
+  between device provider and data path provider for the same device.
+  - ``attr`` is the attribute value.
+
+4. Queue-related
+
+* ``rte_emu_get_queue_info(dev_id, queue, info)``
+* ``rte_emu_get_irq_info(dev_id, irq, info)``
+* ``rte_emu_get_db_info(dev_id, doorbell, info)``
+
+  Above APIs are for data path provider to get queue/interrupt/doorbell information.
+
+  - ``queue``, ``irq`` and ``doorbell`` are respectively the queue/interrupt/doorbell
+  index.
+  - ``info`` is for saving the queue/interrupt/doorbell info.
+
+5. Direct Memory Access
+
+* ``rte_emu_get_mem_table(dev_id, tb)``
+
+  Above APIs are for data path provider to get the information of DMA memory table.
+  The memory table implementation varies between different devices and memory table
+  operations should better be helper functions exposed by device provider. Because
+  address translation make a difference in data path performance, the memory table
+  implementation should have high efficiency.
+
+  ``tb`` is for saving the DMA memory table.
doc/guides/prog_guide/index.rst +1 -0
index f9847b1058..0ed15a0995 100644
@@ -71,3 +71,4 @@ Programmer's Guide
     profile_app
     glossary
     vfio_user_lib
+    emudev
doc/guides/rel_notes/release_21_02.rst +12 -0
index 6fbb6e8c39..3d26b6b580 100644
@@ -67,6 +67,18 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
+* **Added emudev Library.**
+
+  Added an experimental library ``librte_emudev`` to provide device abstraction
+  for an emulated device.
+
+  The library abstracts an emulated device and provides several categories of
+  device-level APIs. The specific device type could be general (e.g, network,
+  crypto and etc.). It can be attached to another data path driver (e.g, ethdev
+  driver) to leverage the high performance of DPDK data path driver.
+
+  See :doc:`../prog_guide/emudev` for more information.
+
 Removed Items
 -------------
 
Signature · 1 line
2.17.1
author
#21 lore raw

PATCH v2 3/8 emu: introduce emulated iavf driver

In reply to Chenbo Xia

This patch introduces emulated iavf driver. It is a vdev driver emulating all iavf device behavior except data path handling.

---
 MAINTAINERS                          |   7 +
 drivers/emu/iavf/iavf_emu.c          |  29 ++++
 drivers/emu/iavf/iavf_emu_internal.h |  49 +++++++
 drivers/emu/iavf/iavf_emudev.c       | 207 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   8 ++
 drivers/emu/iavf/rte_iavf_emu.h      |  43 ++++++
 drivers/emu/iavf/version.map         |   3 +
 drivers/emu/meson.build              |   6 +
 drivers/meson.build                  |   1 +
 9 files changed, 353 insertions(+)
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
MAINTAINERS +7 -0
index 1b395e181d..bca206ba8f 100644
@@ -1275,6 +1275,13 @@ F: doc/guides/rawdevs/ntb.rst
 F: examples/ntb/
 F: doc/guides/sample_app_ug/ntb.rst
 
+Emudev Drivers
+--------------
+
+Intel iavf
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: drivers/emulation/iavf/
 
 Packet processing
 -----------------
drivers/emu/iavf/iavf_emu.c new file +29 -0
new file mode 100644
index 0000000000..68d2c440e3
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "iavf_emu_internal.h"
+
+static int iavf_emu_dev_close(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf;
+
+	/* For now, we don't support device close when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Close failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_device(iavf);
+	dev->priv_data = NULL;
+
+	return 0;
+}
+
+struct rte_emudev_ops emu_iavf_ops = {
+	.dev_close = iavf_emu_dev_close,
+};
drivers/emu/iavf/iavf_emu_internal.h new file +49 -0

49 lines not on this page · raw message

drivers/emu/iavf/iavf_emudev.c new file +207 -0

207 lines not on this page · raw message

drivers/emu/iavf/meson.build new file +8 -0
new file mode 100644
index 0000000000..58c2a90383
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('iavf_emu.c', 'iavf_emudev.c')
+
+deps += ['bus_vdev', 'emudev']
+
+headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h new file +43 -0

43 lines not on this page · raw message

drivers/emu/iavf/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
drivers/emu/meson.build new file +6 -0
new file mode 100644
index 0000000000..acc8c395ef
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+drivers = ['iavf']
+std_deps = ['emudev']
+config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_EMUDEV'
drivers/meson.build +1 -0
index f9febc579e..64c34d2f9f 100644
@@ -8,6 +8,7 @@ subdirs = [
 	'common/mlx5', # depends on bus.
 	'common/qat', # depends on bus.
 	'mempool', # depends on common and bus.
+	'emu', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
 	'crypto',  # depends on common, bus and mempool (net in future).
Signature · 1 line
2.17.1
author
#22 lore raw

PATCH v2 4/8 emu/iavf: add vfio-user device register and unregister

In reply to Chenbo Xia

This patch adds vfio-user APIs call in driver probe and remove. rte_vfio_user_register() and rte_vfio_user_unregister() are called to create/destroy a vfio-user device. Notify callbacks that libvfio_user defines are also implemented.

---
 drivers/emu/iavf/iavf_emu.c          |   3 +-
 drivers/emu/iavf/iavf_emu_internal.h |  19 ++
 drivers/emu/iavf/iavf_emudev.c       |  12 +-
 drivers/emu/iavf/iavf_vfio_user.c    | 384 +++++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h    |  16 ++
 drivers/emu/iavf/meson.build         |   5 +-
 drivers/emu/iavf/rte_iavf_emu.h      |  17 ++
 7 files changed, 452 insertions(+), 4 deletions(-)
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
drivers/emu/iavf/iavf_emu.c +2 -1
index 68d2c440e3..dfd9796920 100644
@@ -2,7 +2,7 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 static int iavf_emu_dev_close(struct rte_emudev *dev)
 {
@@ -18,6 +18,7 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
 
drivers/emu/iavf/iavf_emu_internal.h +19 -0
index a726bfe577..10197c00ba 100644
@@ -17,6 +17,13 @@ extern int emu_iavf_logtype;
 #define EMU_IAVF_LOG(level, ...) \
 	rte_log(RTE_LOG_ ## level, emu_iavf_logtype, "EMU_IAVF: " __VA_ARGS__)
 
+struct iavf_emu_vfio_user {
+	int dev_id;
+	struct vfio_device_info *dev_info;
+	struct rte_vfio_user_regions *reg;
+	struct rte_vfio_user_irq_info *irq;
+};
+
 struct iavf_emu_intr_info {
 	int enable;
 	int fd;
@@ -27,6 +34,14 @@ struct iavf_emu_intr {
 	struct iavf_emu_intr_info info[RTE_IAVF_EMU_MAX_INTR];
 };
 
+struct iavf_emu_adminQ {
+	uint32_t *ring_addr_lo;
+	uint32_t *ring_addr_hi;
+	uint32_t *ring_sz;
+	uint16_t db_size;
+	void *doorbell;
+};
+
 struct iavf_emu_lanQ {
 	uint16_t db_size;
 	void *doorbell;
@@ -34,14 +49,18 @@ struct iavf_emu_lanQ {
 
 struct iavf_emudev {
 	struct rte_emudev *edev;
+	struct iavf_emu_vfio_user *vfio;
 	/* Maximum LANQ queue pair that this emulated iavf has */
 	uint16_t max_lanqp;
 	/* Maximum LANQ queue pair number that back-end driver can use */
 	uint16_t max_be_lanqp;
 	unsigned int numa_node;
+	int ready;
 	char *sock_addr;
+	struct rte_iavf_emu_notify_ops *ops;
 	struct rte_iavf_emu_mem *mem;
 	struct iavf_emu_intr *intr;
+	struct iavf_emu_adminQ adq[RTE_IAVF_EMU_ADMINQ_NUM];
 	struct iavf_emu_lanQ *lanq;
 };
 
drivers/emu/iavf/iavf_emudev.c +11 -1
index a4cd2deb06..fbbe3d95a7 100644
@@ -6,7 +6,7 @@
 #include <rte_emudev.h>
 #include <rte_emudev_vdev.h>
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 #define EMU_IAVF_SOCK_ARG "sock"
 #define EMU_IAVF_QUEUES_ARG "queues"
@@ -170,10 +170,20 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 	iavf->max_lanqp = queues;
 	edev->priv_data = (void *)iavf;
 
+	ret = iavf_emu_register_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to register vfio user.\n");
+		ret = -1;
+		goto err_reg;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_reg:
+	iavf_emu_uninit_device(iavf);
 err_ndev:
 	rte_emudev_release(edev);
 err:
drivers/emu/iavf/iavf_vfio_user.c new file +384 -0

384 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h new file +16 -0
new file mode 100644
index 0000000000..aa2f3edc87
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IAVF_VFIO_USER_H
+#define _IAVF_VFIO_USER_H
+
+#include <rte_vfio_user.h>
+
+#include "iavf_emu_internal.h"
+
+int iavf_emu_register_vfio_user(struct iavf_emudev *dev);
+
+int iavf_emu_unregister_vfio_user(struct iavf_emudev *dev);
+
+#endif
drivers/emu/iavf/meson.build +3 -2
index 58c2a90383..4f651258c2 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
-sources = files('iavf_emu.c', 'iavf_emudev.c')
+sources = files('iavf_emu.c', 'iavf_vfio_user.c',
+	'iavf_emudev.c')
 
-deps += ['bus_vdev', 'emudev']
+deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
 headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h +17 -0
index 623c3c5d99..6de0989f0b 100644
@@ -40,4 +40,21 @@ struct rte_iavf_emu_mem {
 	struct rte_iavf_emu_mem_reg regions[RTE_IAVF_EMU_MAX_MEM_REGIONS];
 };
 
+struct rte_iavf_emu_notify_ops {
+	/* Device is ready */
+	int (*device_ready)(struct rte_emudev *dev);
+	/* Device is destroyed */
+	void (*device_destroy)(struct rte_emudev *dev);
+	/* Update device status */
+	int (*update_status)(struct rte_emudev *dev);
+	/* Start device */
+	int (*device_start)(struct rte_emudev *dev);
+	/* Stop device */
+	int (*device_stop)(struct rte_emudev *dev);
+	/* Lock or unlock data path */
+	int (*lock_dp)(struct rte_emudev *dev, int lock);
+	/* Reset device */
+	int (*reset_device)(struct rte_emudev *dev);
+};
+
 #endif
Signature · 1 line
2.17.1
#23 lore raw

Re: [dpdk-dev] [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

drivers/emu/iavf/iavf_vfio_user.c line 199

Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c new file +5 -0
+static inline struct iavf_emu_sock_list * iavf_emu_find_sock_list(char
+*sock_addr) {
+	struct iavf_emu_sock_list *list;
+	struct iavf_emudev *dev;
+	int list_exist;

Initialize list_exist to 0?

Quoted 4 lines
drivers/emu/iavf/iavf_vfio_user.c new file +4 -0
+
+	if (!sock_addr)
+		return NULL;
+

drivers/emu/iavf/iavf_vfio_user.c line 213

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	pthread_mutex_lock(&sock_list_lock);
+
+	TAILQ_FOREACH(list, &sock_list, next) {
+		dev = (struct iavf_emudev *)list->emu_dev->priv_data;
+
+		if (!strcmp(dev->sock_addr, sock_addr)) {
+			list_exist = 1;
+			break;
+		}
+		break;

This "break" need to be removed.

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	}
+
+	pthread_mutex_unlock(&sock_list_lock);
+
+	if (!list_exist)
+		return NULL;
+
+	return list;
+}
+
#24 lore raw

Re: [dpdk-dev] [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

Hi Jingjing,

Quoted 10 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Monday, January 4, 2021 2:45 PM
To: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>
Subject: RE: [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

Quoted 7 lines

drivers/emu/iavf/iavf_vfio_user.c line 199

Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c new file +5 -0
+static inline struct iavf_emu_sock_list * iavf_emu_find_sock_list(char
+*sock_addr) {
+	struct iavf_emu_sock_list *list;
+	struct iavf_emudev *dev;
+	int list_exist;

Initialize list_exist to 0?

Yes, will fix in next version.

Quoted 7 lines
Quoted 7 lines
drivers/emu/iavf/iavf_vfio_user.c new file +7 -0
+
+	if (!sock_addr)
+		return NULL;
+
+	pthread_mutex_lock(&sock_list_lock);
+
+	TAILQ_FOREACH(list, &sock_list, next) {
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c line 213

Quoted 7 lines
drivers/emu/iavf/iavf_vfio_user.c new file +7 -0
+		dev = (struct iavf_emudev *)list->emu_dev->priv_data;
+
+		if (!strcmp(dev->sock_addr, sock_addr)) {
+			list_exist = 1;
+			break;
+		}
+		break;

This "break" need to be removed.

Yes, will fix in next version.

Thanks,
Chenbo

Quoted 11 lines
Quoted 11 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	}
+
+	pthread_mutex_unlock(&sock_list_lock);
+
+	if (!list_exist)
+		return NULL;
+
+	return list;
+}
+
#25 lore raw

Re: [dpdk-dev] [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

In reply to Chenbo Xia
Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c new file +6 -0
+static int iavf_emu_update_status(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+	int ret;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)

drivers/emu/iavf/iavf_vfio_user.c line 265

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+		return -1;
+
+	ret = iavf_emu_setup_mem_table(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up memtable for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = iavf_emu_setup_irq(dev);

In update callback, irq fds will be reinitialized here. Think about if the update happening during mailbox communication, the eventfd of mailbox will be cleared without notify.

Quoted 10 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up irq for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	dev->ops->update_status(dev->edev);
+
+	return 0;
+}
#26 lore raw

Re: [dpdk-dev] [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

Hi Jingjing,

Quoted 21 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Tuesday, January 5, 2021 9:42 PM
To: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>
Subject: RE: [PATCH v2 4/8] emu/iavf: add vfio-user device register and unregister

Quoted 11 lines
drivers/emu/iavf/iavf_vfio_user.c new file +11 -0
+static int iavf_emu_update_status(int vfio_dev_id) {
+	struct iavf_emudev *dev;
+	int ret;
+
+	dev = find_iavf_with_dev_id(vfio_dev_id);
+	if (!dev)
+		return -1;
+
+	ret = iavf_emu_setup_mem_table(dev);
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up memtable for "
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c line 265

Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c new file +5 -0
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	ret = iavf_emu_setup_irq(dev);

In update callback, irq fds will be reinitialized here. Think about if the update happening during mailbox communication, the eventfd of mailbox will be cleared without notify.

Correct! Will fix this in next version.

Thanks!
Chenbo

Quoted 11 lines
Quoted 11 lines
drivers/emu/iavf/iavf_vfio_user.c new file +10 -0
+	if (ret) {
+		EMU_IAVF_LOG(ERR, "Failed to set up irq for "
+			"device %d", dev->vfio->dev_id);
+		return ret;
+	}
+
+	dev->ops->update_status(dev->edev);
+
+	return 0;
+}
author
#27 lore raw

PATCH v2 5/8 emu/iavf: add resource management and internal logic of iavf

In reply to Chenbo Xia

This patch adds the allocation and release of device resources. Device resources include PCI BARs' memory and interrupt related resources. Device internal logic is also added.

---
 drivers/emu/iavf/iavf_emu.c       |   1 +
 drivers/emu/iavf/iavf_emudev.c    |  20 +
 drivers/emu/iavf/iavf_vfio_user.c | 669 ++++++++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h |  41 ++
 drivers/emu/iavf/meson.build      |   8 +
 5 files changed, 739 insertions(+)
drivers/emu/iavf/iavf_emu.c +1 -0
index dfd9796920..c1a702d744 100644
@@ -18,6 +18,7 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_vfio_user(iavf);
 	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
drivers/emu/iavf/iavf_emudev.c +20 -0
index fbbe3d95a7..70cf558eef 100644
@@ -178,10 +178,30 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 		goto err_reg;
 	}
 
+	ret = iavf_emu_init_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to init vfio user.\n");
+		ret = -1;
+		goto err_init;
+	}
+
+	ret = iavf_emu_start_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to start vfio user.\n");
+		ret = -1;
+		goto err_start;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_start:
+	iavf_emu_uninit_vfio_user(iavf);
+err_init:
+	iavf_emu_unregister_vfio_user(iavf);
 err_reg:
 	iavf_emu_uninit_device(iavf);
 err_ndev:
drivers/emu/iavf/iavf_vfio_user.c +669 -0

697 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h +41 -0

53 lines not on this page · raw message

drivers/emu/iavf/meson.build +8 -0
index 4f651258c2..3cab2226b7 100644
@@ -1,6 +1,14 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
+librt = cc.find_library('rt', required: false)
+if not librt.found()
+	build = false
+	subdir_done()
+endif
+
+ext_deps += librt
+
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
 	'iavf_emudev.c')
 
Signature · 1 line
2.17.1
#28 lore raw

Re: [dpdk-dev] [PATCH v2 5/8] emu/iavf: add resource management and internal logic of iavf

Quoted 36 lines

+static ssize_t iavf_emu_bar0_rw(struct rte_vfio_user_reg_info *reg, char *buf,

drivers/emu/iavf/iavf_vfio_user.c +33 -0
+	size_t count, loff_t pos, bool iswrite) {
+	struct iavf_emudev *dev = (struct iavf_emudev *)reg->priv;
+	char *reg_pos;
+
+	if (!reg->base) {
+		EMU_IAVF_LOG(ERR, "BAR 0 does not exist\n");
+		return -EFAULT;
+	}
+
+	if (pos + count > reg->info->size) {
+		EMU_IAVF_LOG(ERR, "Access exceeds BAR 0 size\n");
+		return -EINVAL;
+	}
+
+	reg_pos = (char *)reg->base + pos;
+
+	if (!iswrite) {
+		rte_memcpy(buf, reg_pos, count);
+	} else {
+		int tmp;
+		uint32_t val;
+		int idx = -1;
+
+		if (count != 4)
+			return -EINVAL;
+
+		val = *(uint32_t *)buf;
+		/* Only handle interrupt enable/disable for now */
+		if (pos == IAVF_VFINT_DYN_CTL01) {
+			tmp = val & IAVF_VFINT_DYN_CTL01_INTENA_MASK;
+			idx = 0;
+		} else if ((pos >= IAVF_VFINT_DYN_CTLN1(0)) && pos <=
+
	IAVF_VFINT_DYN_CTLN1(RTE_IAVF_EMU_MAX_INTR - 1)) {

drivers/emu/iavf/iavf_vfio_user.c static void iavf_emu_reset_all_resources(struct iavf_emudev *dev) · line 661

Quoted 5 lines
drivers/emu/iavf/iavf_vfio_user.c +5 -0
+			tmp = val & IAVF_VFINT_DYN_CTLN1_INTENA_MASK;
+			idx = pos - IAVF_VFINT_DYN_CTLN1(0);
+			if (idx % 4)
+				return -EINVAL;
+			idx = idx / 4;

Should be idx = idx / 4 + 1; ?

drivers/emu/iavf/iavf_vfio_user.c static void iavf_emu_reset_all_resources(struct iavf_emudev *dev) · line 667

Quoted 6 lines
drivers/emu/iavf/iavf_vfio_user.c +6 -0
+		}
+
+		if (idx != -1 &&
+			tmp != dev->intr->info[idx].enable && dev->ready) {
+			dev->ops->update_status(dev->edev);
+			dev->intr->info[idx].enable = tmp;

dev->intr->info[idx].enable need to be set before update_status callback is called.

Quoted 7 lines
drivers/emu/iavf/iavf_vfio_user.c +7 -0
+		}
+
+		rte_memcpy(reg_pos, buf, count);
+	}
+
+	return count;
+}
#29 lore raw

Re: [dpdk-dev] [PATCH v2 5/8] emu/iavf: add resource management and internal logic of iavf

Hi Jingjing,

Quoted 44 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Tuesday, December 29, 2020 2:05 PM
To: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>
Subject: RE: [PATCH v2 5/8] emu/iavf: add resource management and internal logic of iavf

Quoted 34 lines

+static ssize_t iavf_emu_bar0_rw(struct rte_vfio_user_reg_info *reg, char *buf,

drivers/emu/iavf/iavf_vfio_user.c +32 -0
+	size_t count, loff_t pos, bool iswrite) {
+	struct iavf_emudev *dev = (struct iavf_emudev *)reg->priv;
+	char *reg_pos;
+
+	if (!reg->base) {
+		EMU_IAVF_LOG(ERR, "BAR 0 does not exist\n");
+		return -EFAULT;
+	}
+
+	if (pos + count > reg->info->size) {
+		EMU_IAVF_LOG(ERR, "Access exceeds BAR 0 size\n");
+		return -EINVAL;
+	}
+
+	reg_pos = (char *)reg->base + pos;
+
+	if (!iswrite) {
+		rte_memcpy(buf, reg_pos, count);
+	} else {
+		int tmp;
+		uint32_t val;
+		int idx = -1;
+
+		if (count != 4)
+			return -EINVAL;
+
+		val = *(uint32_t *)buf;
+		/* Only handle interrupt enable/disable for now */
+		if (pos == IAVF_VFINT_DYN_CTL01) {
+			tmp = val & IAVF_VFINT_DYN_CTL01_INTENA_MASK;
+			idx = 0;
+		} else if ((pos >= IAVF_VFINT_DYN_CTLN1(0)) && pos <=
Quoted 8 lines

drivers/emu/iavf/iavf_vfio_user.c static void iavf_emu_reset_all_resources(struct iavf_emudev *dev) · line 661

Quoted 7 lines

+

	IAVF_VFINT_DYN_CTLN1(RTE_IAVF_EMU_MAX_INTR - 1)) {
drivers/emu/iavf/iavf_vfio_user.c +5 -0
+			tmp = val & IAVF_VFINT_DYN_CTLN1_INTENA_MASK;
+			idx = pos - IAVF_VFINT_DYN_CTLN1(0);
+			if (idx % 4)
+				return -EINVAL;
+			idx = idx / 4;

Should be idx = idx / 4 + 1; ?

Yes! Will fix in v3.

Quoted 9 lines

drivers/emu/iavf/iavf_vfio_user.c static void iavf_emu_reset_all_resources(struct iavf_emudev *dev) · line 667

Quoted 7 lines
drivers/emu/iavf/iavf_vfio_user.c +6 -0
+		}
+
+		if (idx != -1 &&
+			tmp != dev->intr->info[idx].enable && dev->ready) {
+			dev->ops->update_status(dev->edev);
+			dev->intr->info[idx].enable = tmp;

dev->intr->info[idx].enable need to be set before update_status callback is called.

Yes. Will fix this too.

Thanks!
Chenbo

Quoted 8 lines
Quoted 8 lines
drivers/emu/iavf/iavf_vfio_user.c +7 -0
+		}
+
+		rte_memcpy(reg_pos, buf, count);
+	}
+
+	return count;
+}
author
#30 lore raw

PATCH v2 6/8 emu/iavf: add emudev operations to fit in emudev framework

In reply to Chenbo Xia

This patch implements emudev opertions to make emulated iavf fit into rte_emudev framework. Lifecycle related and device resource related operations are both implemented.

---
 drivers/emu/iavf/iavf_emu.c     | 218 ++++++++++++++++++++++++++++++++
 drivers/emu/iavf/rte_iavf_emu.h |  59 +++++++++
 2 files changed, 277 insertions(+)
drivers/emu/iavf/iavf_emu.c +218 -0
index c1a702d744..9ad371ca98 100644
@@ -2,7 +2,72 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_malloc.h>
+#include <rte_emudev.h>
+
 #include "iavf_vfio_user.h"
+#include <iavf_type.h>
+
+static int iavf_emu_dev_start(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_start != NULL)
+		iavf->ops->device_start(dev);
+
+	return 0;
+}
+
+static void iavf_emu_dev_stop(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_stop != NULL)
+		iavf->ops->device_stop(dev);
+}
+
+static int iavf_emu_dev_configure(struct rte_emudev *dev,
+		struct rte_emudev_info *dev_conf)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf =
+		(struct rte_iavf_emu_config *)dev_conf->dev_priv;
+
+	if (!dev_conf->dev_priv)
+		return -EINVAL;
+
+	/* Currently emulated iavf does not support max_qp_num
+	 * and region num configuration
+	 */
+	if (dev->dev_info.max_qp_num != dev_conf->max_qp_num ||
+		dev->dev_info.region_num != dev_conf->region_num) {
+		EMU_IAVF_LOG(ERR,
+			"Configure max_qp_num/region num not supported\n");
+		return -ENOTSUP;
+	}
+
+	if (conf->qp_num >  RTE_MAX_QUEUES_PER_PORT ||
+		conf->qp_num > RTE_IAVF_EMU_MAX_QP_NUM) {
+		EMU_IAVF_LOG(ERR, "Queue pair num exceeds max\n");
+		return -EINVAL;
+	}
+
+	/* For now, we don't support device configure when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Configure failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf->max_be_lanqp = conf->qp_num;
+	return 0;
+}
 
 static int iavf_emu_dev_close(struct rte_emudev *dev)
 {
@@ -26,6 +91,159 @@ static int iavf_emu_dev_close(struct rte_emudev *dev)
 	return 0;
 }
 
+static int iavf_emu_get_dev_info(struct rte_emudev *dev,
+	rte_emudev_obj_t info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf = (struct rte_iavf_emu_config *)info;
+
+	if (!info)
+		return -EINVAL;
+
+	conf->qp_num = iavf->max_be_lanqp;
+	return 0;
+}
+
+static int iavf_emu_get_mem_table(struct rte_emudev *dev,
+	rte_emudev_mem_table_t *tb)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	*tb = iavf->mem;
+
+	return 0;
+}
+
+static int iavf_emu_get_queue_info(struct rte_emudev *dev, uint32_t queue,
+		struct rte_emudev_q_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (queue < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[queue];
+		uint64_t base, size;
+
+		if (adq->ring_addr_lo == NULL ||
+		    adq->ring_addr_hi == NULL ||
+		    adq->ring_sz == NULL)
+			return -1;
+		base = RTE_IAVF_EMU_32_TO_64(*adq->ring_addr_hi,
+			*adq->ring_addr_lo);
+		size = *adq->ring_sz;
+		info->base = base;
+		info->size = size;
+		info->doorbell_id = queue;
+		/* RX AdminQ should have IRQ vector 0 */
+		info->irq_vector = queue - 1;
+	} else {
+		info->base = 0;
+		info->size = 0;
+		info->doorbell_id = queue;
+		info->irq_vector = -1;
+	}
+
+	return 0;
+}
+
+static int iavf_emu_get_irq_info(struct rte_emudev *dev, uint32_t vector,
+		struct rte_emudev_irq_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct iavf_emu_intr *intr = iavf->intr;
+	struct iavf_emu_intr_info *intr_info = &intr->info[vector];
+
+	if (vector >= intr->intr_num)
+		return -EINVAL;
+
+	info->eventfd = intr_info->fd;
+	info->enable = intr_info->enable;
+	info->vector = vector;
+
+	return 0;
+}
+
+static int iavf_emu_get_db_info(struct rte_emudev *dev, uint32_t doorbell,
+		struct rte_emudev_db_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (doorbell < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[doorbell];
+
+		info->data.mem.base = (uint64_t)adq->doorbell;
+		info->data.mem.size = adq->db_size;
+	} else {
+		struct iavf_emu_lanQ *lanq =
+			&iavf->lanq[doorbell - RTE_IAVF_EMU_ADMINQ_NUM];
+
+		info->data.mem.base = (uint64_t)lanq->doorbell;
+		info->data.mem.size = lanq->db_size;
+	}
+
+	info->flag |= RTE_EMUDEV_DB_MEM;
+	info->id = doorbell;
+
+	return 0;
+}
+
+static int iavf_emu_subs_ev(struct rte_emudev *dev,
+	rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	iavf->ops = (struct rte_iavf_emu_notify_ops *)ev_chnl;
+
+	return 0;
+}
+
+static int iavf_emu_unsubs_ev(struct rte_emudev *dev,
+		rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if ((struct rte_iavf_emu_notify_ops *)ev_chnl == iavf->ops) {
+		iavf->ops = NULL;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
+static int iavf_emu_get_attr(struct rte_emudev *dev, const char *attr_name,
+		rte_emudev_attr_t attr)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_vfio_user_reg_info *info;
+	int ret = 0;
+
+	info = &iavf->vfio->reg->reg_info[0];
+
+	if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ASQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ATQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ARQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ARQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_RESET))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VFGEN_RSTAT;
+	else
+		ret = -EINVAL;
+
+	return ret;
+}
+
 struct rte_emudev_ops emu_iavf_ops = {
+	.dev_start = iavf_emu_dev_start,
+	.dev_stop = iavf_emu_dev_stop,
+	.dev_configure = iavf_emu_dev_configure,
 	.dev_close = iavf_emu_dev_close,
+	.dev_info_get = iavf_emu_get_dev_info,
+	.get_mem_table = iavf_emu_get_mem_table,
+	.get_queue_info = iavf_emu_get_queue_info,
+	.get_irq_info = iavf_emu_get_irq_info,
+	.get_db_info = iavf_emu_get_db_info,
+	.subscribe_event = iavf_emu_subs_ev,
+	.unsubscribe_event = iavf_emu_unsubs_ev,
+	.get_attr = iavf_emu_get_attr,
 };
drivers/emu/iavf/rte_iavf_emu.h +59 -0

84 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#31 lore raw

PATCH v2 7/8 test/emudev: introduce functional test

In reply to Chenbo Xia

This patch introduces functional test for emudev. The implementation of iavf emudev selftest is also added.

---
 app/test/meson.build                 |   5 +-
 app/test/test_emudev.c               |  29 +++++
 drivers/emu/iavf/iavf_emu.c          |   1 +
 drivers/emu/iavf/iavf_emu_internal.h |   1 +
 drivers/emu/iavf/iavf_emu_test.c     | 174 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   2 +-
 6 files changed, 210 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
app/test/meson.build +4 -1
index f5b15ac44c..b8b79bbc8b 100644
@@ -139,6 +139,7 @@ test_sources = files('commands.c',
 	'test_trace_register.c',
 	'test_trace_perf.c',
 	'test_vfio_user.c',
+	'test_emudev.c',
 	'test_version.c',
 	'virtual_pmd.c'
 )
@@ -176,7 +177,8 @@ test_deps = ['acl',
 	'stack',
 	'vfio_user',
 	'telemetry',
-	'timer'
+	'timer',
+	'emudev'
 ]
 
 # Each test is marked with flag true/false
@@ -327,6 +329,7 @@ driver_test_names = [
         'eventdev_selftest_octeontx',
         'eventdev_selftest_sw',
         'rawdev_autotest',
+        'emudev_autotest',
 ]
 
 dump_test_names = [
app/test/test_emudev.c new file +29 -0
new file mode 100644
index 0000000000..0dfce162ed
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_emudev.h>
+#include <rte_bus_vdev.h>
+
+#include "test.h"
+
+static int
+test_emudev_selftest_impl(const char *pmd, const char *opts)
+{
+	int ret = 0;
+
+	if (rte_emudev_get_dev_id(pmd) == -ENODEV)
+		ret = rte_vdev_init(pmd, opts);
+	if (ret)
+		return TEST_SKIPPED;
+
+	return rte_emudev_selftest(rte_emudev_get_dev_id(pmd));
+}
+
+static int
+test_emudev_selftest(void)
+{
+	return test_emudev_selftest_impl("emu_iavf", "sock=/tmp/sock1");
+}
+
+REGISTER_TEST_COMMAND(emudev_autotest, test_emudev_selftest);
drivers/emu/iavf/iavf_emu.c +1 -0
index 9ad371ca98..88bf2bdf94 100644
@@ -246,4 +246,5 @@ struct rte_emudev_ops emu_iavf_ops = {
 	.subscribe_event = iavf_emu_subs_ev,
 	.unsubscribe_event = iavf_emu_unsubs_ev,
 	.get_attr = iavf_emu_get_attr,
+	.dev_selftest = iavf_emu_selftest,
 };
drivers/emu/iavf/iavf_emu_internal.h +1 -0
index 10197c00ba..1ac7f96566 100644
@@ -65,4 +65,5 @@ struct iavf_emudev {
 };
 
 void iavf_emu_uninit_device(struct iavf_emudev *dev);
+int iavf_emu_selftest(uint16_t dev_id);
 #endif
drivers/emu/iavf/iavf_emu_test.c new file +174 -0

174 lines not on this page · raw message

drivers/emu/iavf/meson.build +1 -1
index 3cab2226b7..613783e407 100644
@@ -10,7 +10,7 @@ endif
 ext_deps += librt
 
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
-	'iavf_emudev.c')
+	'iavf_emu_test.c', 'iavf_emudev.c')
 
 deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
Signature · 1 line
2.17.1
author
#32 lore raw

PATCH v2 8/8 doc: update release notes for iavf emudev driver

In reply to Chenbo Xia

Update release notes for emulated iavf driver.

---
 doc/guides/rel_notes/release_21_02.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_21_02.rst +5 -1
index 3d26b6b580..b310b67b7d 100644
@@ -67,7 +67,7 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
-* **Added emudev Library.**
+* **Added emudev Library and first emudev driver.**
 
   Added an experimental library ``librte_emudev`` to provide device abstraction
   for an emulated device.
@@ -77,6 +77,10 @@ New Features
   crypto and etc.). It can be attached to another data path driver (e.g, ethdev
   driver) to leverage the high performance of DPDK data path driver.
 
+  The first emudev driver is also introduced, which emulates software iavf
+  devices. This driver emulates all iavf device behavior except data path
+  handling.
+
   See :doc:`../prog_guide/emudev` for more information.
 
 Removed Items
Signature · 1 line
2.17.1
#33 lore raw
In reply to Chenbo Xia

19/12/2020 07:27, Chenbo Xia:

Quoted 8 lines

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 13 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

As voted today in the Technical Board meeting, there will be a draft repository to host this initiative. A proposed name for the git tree is:
dpdk-draft-emudev
Feel free to propose a better name.

This series cannot be merged in the main repository until the Qemu specification is made official.
When Qemu part will be complete, we can assume more reviews of the DPDK related code.
In the meantime, the draft repository will help getting interest, testing, reviews and opinions.

#34 lore raw

Hi Thomas,

Quoted 35 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, January 14, 2021 12:52 AM
To: Xia, Chenbo <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Liang, Cunming <[email protected]>; Lu, Xiuchun <[email protected]>; Li, Miao <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH v2 0/8] Introduce emudev library and iavf emudev driver

19/12/2020 07:27, Chenbo Xia:

Quoted 8 lines

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Quoted 13 lines

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Quoted 5 lines

As voted today in the Technical Board meeting, there will be a draft repository to host this initiative. A proposed name for the git tree is:
dpdk-draft-emudev
Feel free to propose a better name.

This works fine.

Quoted 5 lines

This series cannot be merged in the main repository until the Qemu specification is made official. When Qemu part will be complete, we can assume more reviews of the DPDK related code.

OK.

Quoted 2 lines

In the meantime, the draft repository will help getting interest, testing, reviews and opinions.

Thanks for doing this 😊

Cheers,
Chenbo

Quoted 2 lines
author
#35 lore raw

v3 cover Introduce emudev library and iavf emudev driver

In reply to Chenbo Xia

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/86498/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Design overview
---------------

                    +---------------------------------------+
                    |   +---------------+    +-----------+  |
                    |   |  iavf_emudev  |<-->| data path |  |
                    |   |    driver     |    |   driver  |  |
                    |   +---------------+    +-----------+  |
                    |           |                           |
                    | --------------------------- VDEV BUS  |
                    |           |                           |
                    |   +---------------+                   |
+--------------+    |   | vdev:         |                   |
| +----------+ |    |   | /path/to/vfio |                   |
| | Generic  | |    |   +---------------+                   |
| | vfio-dev | |    |           |                           |
| +----------+ |    |           |                           |
| +----------+ |    |      +----------+                     |
| | vfio-user| |    |      | vfio-user|                     |
| | client   | |<---|----->| server   |                     |
| +----------+ |    |      +----------+                     |
| QEMU/DPDK    |    | DPDK                                  |
+--------------+    +---------------------------------------+

- Generic vfio-dev/vfio-user client/vfio-user server
Above concepts are all introduced in librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/86498/

- vdev:/path/to/vfio.
It binds to vdev bus driver. The vdev device is defined by DPDK applications through command line as '--vdev=emu_iavf, path=/path/to/socket' in iavf_emudev case. Parameters in command line include device name (emu_iavf) which is used to identify corresponding driver (in this case, iavf_emudev driver), path=/path/to/socket which is used to open the transport interface to vfio-user client in QEMU/DPDK.

- data path driver.
The data path handling is splited to another standalone driver for modular design.

Why not rawdev for emulated device
----------------------------------
Instead of introducing new class emudev, emulated device could be presented as rawdev. However, existing rawdev APIs cannot meet the requirements of emulated device. There are three API categories for emudev. They are emudev device lifecycle management, backend facing APIs, and emudev device provider facing APIs respectively. Existing rawdev APIs could only cover lifecycle management APIs and some of backend facing APIs. Other APIs, even if added to rawdev API are not required by other rawdev applications.

----------------------------------
v3:
- fix interrupt issue in iavf emudev driver (Jingjing)

v2:
- fix driver meson build file

Chenbo Xia (8):
lib: introduce emudev library
doc: add emudev library guide
emu: introduce emulated iavf driver
emu/iavf: add vfio-user device register and unregister emu/iavf: add resource management and internal logic of iavf emu/iavf: add emudev operations to fit in emudev framework test/emudev: introduce functional test
doc: update release notes for iavf emudev driver

 MAINTAINERS                            |   12 +
 app/test/meson.build                   |    5 +-
 app/test/test_emudev.c                 |   29 +
 doc/guides/prog_guide/emudev.rst       |  122 +++
 doc/guides/prog_guide/index.rst        |    1 +
 doc/guides/rel_notes/release_21_02.rst |   16 +
 drivers/emu/iavf/iavf_emu.c            |  262 ++++++
 drivers/emu/iavf/iavf_emu_internal.h   |   69 ++
 drivers/emu/iavf/iavf_emu_test.c       |  174 ++++
 drivers/emu/iavf/iavf_emudev.c         |  239 ++++++
 drivers/emu/iavf/iavf_vfio_user.c      | 1076 ++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h      |   57 ++
 drivers/emu/iavf/meson.build           |   17 +
 drivers/emu/iavf/rte_iavf_emu.h        |  119 +++
 drivers/emu/iavf/version.map           |    3 +
 drivers/emu/meson.build                |    6 +
 drivers/meson.build                    |    1 +
 lib/librte_emudev/meson.build          |    5 +
 lib/librte_emudev/rte_emudev.c         |  502 +++++++++++
 lib/librte_emudev/rte_emudev.h         |  431 ++++++++++
 lib/librte_emudev/rte_emudev_vdev.h    |   53 ++
 lib/librte_emudev/version.map          |   27 +
 lib/meson.build                        |    2 +-
 23 files changed, 3226 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 doc/guides/prog_guide/emudev.rst
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
Signature · 1 line
2.17.1
author
#36 lore raw

PATCH v3 1/8 lib: introduce emudev library

This patch introduces the emudev library. Emudev library is used to abstract an emulated device, whose type could be general (e.g., network, crypto and etc.). Several device-level APIs are implemented to use or manipulate the device. It can be attached to another data path driver (e.g., ethdev driver) to plug in its high performance data path.

---
 MAINTAINERS                         |   5 +
 lib/librte_emudev/meson.build       |   5 +
 lib/librte_emudev/rte_emudev.c      | 502 ++++++++++++++++++++++++++++
 lib/librte_emudev/rte_emudev.h      | 431 ++++++++++++++++++++++++
 lib/librte_emudev/rte_emudev_vdev.h |  53 +++
 lib/librte_emudev/version.map       |  27 ++
 lib/meson.build                     |   2 +-
 7 files changed, 1024 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map
MAINTAINERS +5 -0
index 91b8b2ccc1..f5f2c4fe15 100644
@@ -1546,6 +1546,11 @@ M: Chenbo Xia <[email protected]>
 M: Xiuchun Lu <[email protected]>
 F: lib/librte_vfio_user/
 
+Emudev - EXPERIMENTAL
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: lib/librte_emudev/
+
 Test Applications
 -----------------
 
lib/librte_emudev/meson.build new file +5 -0
new file mode 100644
index 0000000000..4e16cecbaf
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('rte_emudev.c')
+headers = files('rte_emudev.h', 'rte_emudev_vdev.h')
lib/librte_emudev/rte_emudev.c new file +502 -0

502 lines not on this page · raw message

lib/librte_emudev/rte_emudev.h new file +431 -0

431 lines not on this page · raw message

lib/librte_emudev/rte_emudev_vdev.h new file +53 -0

53 lines not on this page · raw message

lib/librte_emudev/version.map new file +27 -0
new file mode 100644
index 0000000000..f800b4c21c
@@ -0,0 +1,27 @@
+EXPERIMENTAL {
+	global:
+
+	rte_emudev_allocate;
+	rte_emudev_release;
+	rte_emudev_allocated;
+	rte_emudev_start;
+	rte_emudev_stop;
+	rte_emudev_configure;
+	rte_emudev_close;
+	rte_emudev_subscribe_event;
+	rte_emudev_unsubscribe_event;
+	rte_emudev_count;
+	rte_emudev_get_dev_id;
+	rte_emudev_get_dev_info;
+	rte_emudev_get_mem_table;
+	rte_emudev_get_queue_info;
+	rte_emudev_get_irq_info;
+	rte_emudev_get_db_info;
+	rte_emudev_set_attr;
+	rte_emudev_get_attr;
+	rte_emudev_region_map;
+	rte_emudev_selftest;
+	rte_emudev_is_valid_id;
+
+	local: *;
+};
lib/meson.build +1 -1
index b7fbfcc95b..6dd07fb73e 100644
@@ -28,7 +28,7 @@ libraries = [
 	'rib', 'reorder', 'sched', 'security', 'stack', 'vhost',
 	# ipsec lib depends on net, crypto and security
 	'ipsec',
-	'vfio_user',
+	'vfio_user', 'emudev',
 	#fib lib depends on rib
 	'fib',
 	# add pkt framework libs which use other libs from above
Signature · 1 line
2.17.1
author
#37 lore raw

PATCH v3 2/8 doc: add emudev library guide

In reply to Chenbo Xia

Add emudev library guide and update release notes.

---
 doc/guides/prog_guide/emudev.rst       | 122 +++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/rel_notes/release_21_02.rst |  12 +++
 3 files changed, 135 insertions(+)
 create mode 100644 doc/guides/prog_guide/emudev.rst
doc/guides/prog_guide/emudev.rst new file +122 -0
new file mode 100644
index 0000000000..e40213bb5e
@@ -0,0 +1,122 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Emulated Device Library
+=================
+
+Introduction
+------------
+
+The DPDK Emudev library is an abstraction for emulated device. This library
+provides a generic set of APIs for device provider, data path provider and
+applications to use.
+
+A device provider could be implemented as a driver on vdev bus. It should
+expose itself as an emudev for applications to use. It is responsible for the
+device resource management and the device's internal logic. All specifics of a
+device, except data path handling, should be implemented in the device
+provider. The device provider uses emudev APIs mainly for create/destroy an
+emudev instance. The device provider should also use a transport to communicate
+with device consumer (e.g., virtual machine monitor or container). A potential
+choice could be vfio-user library, which implements the vfio-user protocol for
+emulating devices outside of a virtual machine monitor.
+
+A data path provider could be implemented as any type of driver on vdev bus.
+If the device you want to emulate is a network device, you could implement
+it as an ethdev driver. It is responsible for all data path handling. The data
+path provider uses emudev APIs mainly for getting device-related information
+from the device provider.
+
+Applications uses emudev APIs for device lifecycle management and configuration.
+
+Design
+------------
+
+Some key objects are designed in emudev.
+
+  ``Regions`` are the device layout exposed to the data path provider.
+
+  ``Queues`` are the data path queues that the data path provider needs. Queue
+  information includes queue base address, queue size, queue-related doorbell
+  and interrupt information.
+
+  ``Memory Table`` is the DMA mapping table. The data path provider could use
+  it to perform DMA read/write on device consumer's memory.
+
+Information of above key objects could be acquired through emudev APIs. The
+following will introduce the emudev APIs which are used by data path provider
+and applications. The APIs for device provider to use are allocate/release APIs
+and will not be listed because it's similar to other device abstraction.
+
+There are five categories of APIs:
+
+1. Lifecycle management
+
+* ``rte_emu_dev_start(dev_id)``
+* ``rte_emu_dev_stop(dev_id)``
+* ``rte_emu_dev_configure(dev_id)``
+* ``rte_emu_dev_close(dev_id)``
+
+  Above APIs are respectively for device start/stop/configure/close and mainly
+  for applications to use.
+
+  ``dev_id`` is the emudev device ID.
+
+2. Notification
+
+* ``rte_emu_subscribe_event(dev_id, ev_chnl)``
+* ``rte_emu_unsubscribe_event(dev_id, ev_chnl)``
+
+  Above APIs are for data path provider and applications to register events.
+  The mechanism of event notification could be different in different device
+  providers. A possbile implementation could be event callbacks.
+
+  ``ev_chnl`` is the event channel pointer. The definition varies between
+  different devices.
+
+3. Region-related
+
+* ``rte_emu_region_map(dev_id, index, region_size, base_addr)``
+* ``rte_emu_get_attr(dev_id, attr_name, attr)``
+* ``rte_emu_set_attr(dev_id, attr_name, attr)``
+
+  Above APIs are for data path provider and applications to read/write regions.
+  ``rte_emu_region_map`` is for directly mapping the region and use the mapped
+  address to read/write it. ``rte_emu_get_attr`` and ``rte_emu_set_attr`` are
+  respectively for getting/setting certain attributes in all regions.
+
+  Applications will set attributes or write regions for device configuration.
+
+  In ``rte_emu_region_map``:
+  - ``index`` is the region index.
+  - ``region_size`` is for saving the size of mapped region.
+  - ``base_addr`` is for saving the address of mapped region.
+
+  In ``rte_emu_get_attr`` and ``rte_emu_set_attr``:
+  - ``attr_name`` is the name of attribute. Note that attribute names are aligned
+  between device provider and data path provider for the same device.
+  - ``attr`` is the attribute value.
+
+4. Queue-related
+
+* ``rte_emu_get_queue_info(dev_id, queue, info)``
+* ``rte_emu_get_irq_info(dev_id, irq, info)``
+* ``rte_emu_get_db_info(dev_id, doorbell, info)``
+
+  Above APIs are for data path provider to get queue/interrupt/doorbell information.
+
+  - ``queue``, ``irq`` and ``doorbell`` are respectively the queue/interrupt/doorbell
+  index.
+  - ``info`` is for saving the queue/interrupt/doorbell info.
+
+5. Direct Memory Access
+
+* ``rte_emu_get_mem_table(dev_id, tb)``
+
+  Above APIs are for data path provider to get the information of DMA memory table.
+  The memory table implementation varies between different devices and memory table
+  operations should better be helper functions exposed by device provider. Because
+  address translation make a difference in data path performance, the memory table
+  implementation should have high efficiency.
+
+  ``tb`` is for saving the DMA memory table.
doc/guides/prog_guide/index.rst +1 -0
index f9847b1058..0ed15a0995 100644
@@ -71,3 +71,4 @@ Programmer's Guide
     profile_app
     glossary
     vfio_user_lib
+    emudev
doc/guides/rel_notes/release_21_02.rst +12 -0
index eabbbfebef..9686930de3 100644
@@ -67,6 +67,18 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
+* **Added emudev Library.**
+
+  Added an experimental library ``librte_emudev`` to provide device abstraction
+  for an emulated device.
+
+  The library abstracts an emulated device and provides several categories of
+  device-level APIs. The specific device type could be general (e.g, network,
+  crypto and etc.). It can be attached to another data path driver (e.g, ethdev
+  driver) to leverage the high performance of DPDK data path driver.
+
+  See :doc:`../prog_guide/emudev` for more information.
+
 Removed Items
 -------------
 
Signature · 1 line
2.17.1
author
#38 lore raw

PATCH v3 3/8 emu: introduce emulated iavf driver

In reply to Chenbo Xia

This patch introduces emulated iavf driver. It is a vdev driver emulating all iavf device behavior except data path handling.

---
 MAINTAINERS                          |   7 +
 drivers/emu/iavf/iavf_emu.c          |  30 ++++
 drivers/emu/iavf/iavf_emu_internal.h |  49 +++++++
 drivers/emu/iavf/iavf_emudev.c       | 209 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   8 +
 drivers/emu/iavf/rte_iavf_emu.h      |  43 ++++++
 drivers/emu/iavf/version.map         |   3 +
 drivers/emu/meson.build              |   6 +
 drivers/meson.build                  |   1 +
 9 files changed, 356 insertions(+)
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/meson.build
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
MAINTAINERS +7 -0
index f5f2c4fe15..3ef8bd1999 100644
@@ -1276,6 +1276,13 @@ F: doc/guides/rawdevs/ntb.rst
 F: examples/ntb/
 F: doc/guides/sample_app_ug/ntb.rst
 
+Emudev Drivers
+--------------
+
+Intel iavf
+M: Chenbo Xia <[email protected]>
+M: Xiuchun Lu <[email protected]>
+F: drivers/emulation/iavf/
 
 Packet processing
 -----------------
drivers/emu/iavf/iavf_emu.c new file +30 -0
new file mode 100644
index 0000000000..98abfcdca2
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "iavf_emu_internal.h"
+
+static int
+iavf_emu_dev_close(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf;
+
+	/* For now, we don't support device close when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Close failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_device(iavf);
+	dev->priv_data = NULL;
+
+	return 0;
+}
+
+struct rte_emudev_ops emu_iavf_ops = {
+	.dev_close = iavf_emu_dev_close,
+};
drivers/emu/iavf/iavf_emu_internal.h new file +49 -0

49 lines not on this page · raw message

drivers/emu/iavf/iavf_emudev.c new file +209 -0

209 lines not on this page · raw message

drivers/emu/iavf/meson.build new file +8 -0
new file mode 100644
index 0000000000..58c2a90383
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+sources = files('iavf_emu.c', 'iavf_emudev.c')
+
+deps += ['bus_vdev', 'emudev']
+
+headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h new file +43 -0

43 lines not on this page · raw message

drivers/emu/iavf/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
drivers/emu/meson.build new file +6 -0
new file mode 100644
index 0000000000..acc8c395ef
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+drivers = ['iavf']
+std_deps = ['emudev']
+config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_EMUDEV'
drivers/meson.build +1 -0
index f9febc579e..64c34d2f9f 100644
@@ -8,6 +8,7 @@ subdirs = [
 	'common/mlx5', # depends on bus.
 	'common/qat', # depends on bus.
 	'mempool', # depends on common and bus.
+	'emu', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
 	'crypto',  # depends on common, bus and mempool (net in future).
Signature · 1 line
2.17.1
author
#39 lore raw

PATCH v3 4/8 emu/iavf: add vfio-user device register and unregister

In reply to Chenbo Xia

This patch adds vfio-user APIs call in driver probe and remove. rte_vfio_user_register() and rte_vfio_user_unregister() are called to create/destroy a vfio-user device. Notify callbacks that libvfio_user defines are also implemented.

---
 drivers/emu/iavf/iavf_emu.c          |   3 +-
 drivers/emu/iavf/iavf_emu_internal.h |  19 ++
 drivers/emu/iavf/iavf_emudev.c       |  12 +-
 drivers/emu/iavf/iavf_vfio_user.c    | 399 +++++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h    |  16 ++
 drivers/emu/iavf/meson.build         |   5 +-
 drivers/emu/iavf/rte_iavf_emu.h      |  17 ++
 7 files changed, 467 insertions(+), 4 deletions(-)
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
drivers/emu/iavf/iavf_emu.c +2 -1
index 98abfcdca2..2f1513137c 100644
@@ -2,7 +2,7 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 static int
 iavf_emu_dev_close(struct rte_emudev *dev)
@@ -19,6 +19,7 @@ iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
 
drivers/emu/iavf/iavf_emu_internal.h +19 -0
index a726bfe577..10197c00ba 100644
@@ -17,6 +17,13 @@ extern int emu_iavf_logtype;
 #define EMU_IAVF_LOG(level, ...) \
 	rte_log(RTE_LOG_ ## level, emu_iavf_logtype, "EMU_IAVF: " __VA_ARGS__)
 
+struct iavf_emu_vfio_user {
+	int dev_id;
+	struct vfio_device_info *dev_info;
+	struct rte_vfio_user_regions *reg;
+	struct rte_vfio_user_irq_info *irq;
+};
+
 struct iavf_emu_intr_info {
 	int enable;
 	int fd;
@@ -27,6 +34,14 @@ struct iavf_emu_intr {
 	struct iavf_emu_intr_info info[RTE_IAVF_EMU_MAX_INTR];
 };
 
+struct iavf_emu_adminQ {
+	uint32_t *ring_addr_lo;
+	uint32_t *ring_addr_hi;
+	uint32_t *ring_sz;
+	uint16_t db_size;
+	void *doorbell;
+};
+
 struct iavf_emu_lanQ {
 	uint16_t db_size;
 	void *doorbell;
@@ -34,14 +49,18 @@ struct iavf_emu_lanQ {
 
 struct iavf_emudev {
 	struct rte_emudev *edev;
+	struct iavf_emu_vfio_user *vfio;
 	/* Maximum LANQ queue pair that this emulated iavf has */
 	uint16_t max_lanqp;
 	/* Maximum LANQ queue pair number that back-end driver can use */
 	uint16_t max_be_lanqp;
 	unsigned int numa_node;
+	int ready;
 	char *sock_addr;
+	struct rte_iavf_emu_notify_ops *ops;
 	struct rte_iavf_emu_mem *mem;
 	struct iavf_emu_intr *intr;
+	struct iavf_emu_adminQ adq[RTE_IAVF_EMU_ADMINQ_NUM];
 	struct iavf_emu_lanQ *lanq;
 };
 
drivers/emu/iavf/iavf_emudev.c +11 -1
index 35c3557188..6ba1cc2a89 100644
@@ -6,7 +6,7 @@
 #include <rte_emudev.h>
 #include <rte_emudev_vdev.h>
 
-#include "iavf_emu_internal.h"
+#include "iavf_vfio_user.h"
 
 #define EMU_IAVF_SOCK_ARG "sock"
 #define EMU_IAVF_QUEUES_ARG "queues"
@@ -172,10 +172,20 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 	iavf->max_lanqp = queues;
 	edev->priv_data = (void *)iavf;
 
+	ret = iavf_emu_register_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to register vfio user.\n");
+		ret = -1;
+		goto err_reg;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_reg:
+	iavf_emu_uninit_device(iavf);
 err_ndev:
 	rte_emudev_release(edev);
 err:
drivers/emu/iavf/iavf_vfio_user.c new file +399 -0

399 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h new file +16 -0
new file mode 100644
index 0000000000..aa2f3edc87
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IAVF_VFIO_USER_H
+#define _IAVF_VFIO_USER_H
+
+#include <rte_vfio_user.h>
+
+#include "iavf_emu_internal.h"
+
+int iavf_emu_register_vfio_user(struct iavf_emudev *dev);
+
+int iavf_emu_unregister_vfio_user(struct iavf_emudev *dev);
+
+#endif
drivers/emu/iavf/meson.build +3 -2
index 58c2a90383..4f651258c2 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
-sources = files('iavf_emu.c', 'iavf_emudev.c')
+sources = files('iavf_emu.c', 'iavf_vfio_user.c',
+	'iavf_emudev.c')
 
-deps += ['bus_vdev', 'emudev']
+deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
 headers = files('rte_iavf_emu.h')
drivers/emu/iavf/rte_iavf_emu.h +17 -0
index 623c3c5d99..6de0989f0b 100644
@@ -40,4 +40,21 @@ struct rte_iavf_emu_mem {
 	struct rte_iavf_emu_mem_reg regions[RTE_IAVF_EMU_MAX_MEM_REGIONS];
 };
 
+struct rte_iavf_emu_notify_ops {
+	/* Device is ready */
+	int (*device_ready)(struct rte_emudev *dev);
+	/* Device is destroyed */
+	void (*device_destroy)(struct rte_emudev *dev);
+	/* Update device status */
+	int (*update_status)(struct rte_emudev *dev);
+	/* Start device */
+	int (*device_start)(struct rte_emudev *dev);
+	/* Stop device */
+	int (*device_stop)(struct rte_emudev *dev);
+	/* Lock or unlock data path */
+	int (*lock_dp)(struct rte_emudev *dev, int lock);
+	/* Reset device */
+	int (*reset_device)(struct rte_emudev *dev);
+};
+
 #endif
Signature · 1 line
2.17.1
author
#40 lore raw

PATCH v3 5/8 emu/iavf: add resource management and internal logic of iavf

In reply to Chenbo Xia

This patch adds the allocation and release of device resources. Device resources include PCI BARs' memory and interrupt related resources. Device internal logic is also added.

---
 drivers/emu/iavf/iavf_emu.c       |   1 +
 drivers/emu/iavf/iavf_emudev.c    |  20 +
 drivers/emu/iavf/iavf_vfio_user.c | 683 +++++++++++++++++++++++++++++-
 drivers/emu/iavf/iavf_vfio_user.h |  41 ++
 drivers/emu/iavf/meson.build      |   8 +
 5 files changed, 750 insertions(+), 3 deletions(-)
drivers/emu/iavf/iavf_emu.c +1 -0
index 2f1513137c..7506849e42 100644
@@ -19,6 +19,7 @@ iavf_emu_dev_close(struct rte_emudev *dev)
 	}
 
 	iavf = (struct iavf_emudev *)dev->priv_data;
+	iavf_emu_uninit_vfio_user(iavf);
 	iavf_emu_unregister_vfio_user(iavf);
 	iavf_emu_uninit_device(iavf);
 	dev->priv_data = NULL;
drivers/emu/iavf/iavf_emudev.c +20 -0
index 6ba1cc2a89..74f4829e7f 100644
@@ -180,10 +180,30 @@ rte_emu_iavf_probe(struct rte_vdev_device *dev)
 		goto err_reg;
 	}
 
+	ret = iavf_emu_init_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to init vfio user.\n");
+		ret = -1;
+		goto err_init;
+	}
+
+	ret = iavf_emu_start_vfio_user(iavf);
+	if (ret) {
+		EMU_IAVF_LOG(ERR,
+			"Emulated iavf failed to start vfio user.\n");
+		ret = -1;
+		goto err_start;
+	}
+
 	edev->started = 1;
 	rte_kvargs_free(kvlist);
 	return 0;
 
+err_start:
+	iavf_emu_uninit_vfio_user(iavf);
+err_init:
+	iavf_emu_unregister_vfio_user(iavf);
 err_reg:
 	iavf_emu_uninit_device(iavf);
 err_ndev:
drivers/emu/iavf/iavf_vfio_user.c +680 -3

712 lines not on this page · raw message

drivers/emu/iavf/iavf_vfio_user.h +41 -0

53 lines not on this page · raw message

drivers/emu/iavf/meson.build +8 -0
index 4f651258c2..3cab2226b7 100644
@@ -1,6 +1,14 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Intel Corporation
 
+librt = cc.find_library('rt', required: false)
+if not librt.found()
+	build = false
+	subdir_done()
+endif
+
+ext_deps += librt
+
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
 	'iavf_emudev.c')
 
Signature · 1 line
2.17.1
author
#41 lore raw

PATCH v3 6/8 emu/iavf: add emudev operations to fit in emudev framework

In reply to Chenbo Xia

This patch implements emudev operations to make emulated iavf fit into rte_emudev framework. Lifecycle related and device resource related operations are both implemented.

---
 drivers/emu/iavf/iavf_emu.c     | 229 ++++++++++++++++++++++++++++++++
 drivers/emu/iavf/rte_iavf_emu.h |  59 ++++++++
 2 files changed, 288 insertions(+)
drivers/emu/iavf/iavf_emu.c +229 -0
index 7506849e42..cc56ec1c4b 100644
@@ -2,7 +2,75 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_malloc.h>
+#include <rte_emudev.h>
+
 #include "iavf_vfio_user.h"
+#include <iavf_type.h>
+
+static int
+iavf_emu_dev_start(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_start != NULL)
+		iavf->ops->device_start(dev);
+
+	return 0;
+}
+
+static void
+iavf_emu_dev_stop(struct rte_emudev *dev)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (iavf->ops != NULL && iavf->ops->device_stop != NULL)
+		iavf->ops->device_stop(dev);
+}
+
+static int
+iavf_emu_dev_configure(struct rte_emudev *dev,
+		struct rte_emudev_info *dev_conf)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf =
+		(struct rte_iavf_emu_config *)dev_conf->dev_priv;
+
+	if (!dev_conf->dev_priv)
+		return -EINVAL;
+
+	/* Currently emulated iavf does not support max_qp_num
+	 * and region num configuration
+	 */
+	if (dev->dev_info.max_qp_num != dev_conf->max_qp_num ||
+		dev->dev_info.region_num != dev_conf->region_num) {
+		EMU_IAVF_LOG(ERR,
+			"Configure max_qp_num/region num not supported\n");
+		return -ENOTSUP;
+	}
+
+	if (conf->qp_num >  RTE_MAX_QUEUES_PER_PORT ||
+		conf->qp_num > RTE_IAVF_EMU_MAX_QP_NUM) {
+		EMU_IAVF_LOG(ERR, "Queue pair num exceeds max\n");
+		return -EINVAL;
+	}
+
+	/* For now, we don't support device configure when data
+	 * path driver is attached
+	 */
+	if (dev->backend_priv) {
+		EMU_IAVF_LOG(ERR, "Configure failed because of "
+			"data path attached\n");
+		return -EPERM;
+	}
+
+	iavf->max_be_lanqp = conf->qp_num;
+	return 0;
+}
 
 static int
 iavf_emu_dev_close(struct rte_emudev *dev)
@@ -27,6 +95,167 @@ iavf_emu_dev_close(struct rte_emudev *dev)
 	return 0;
 }
 
+static int
+iavf_emu_get_dev_info(struct rte_emudev *dev,
+	rte_emudev_obj_t info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_iavf_emu_config *conf = (struct rte_iavf_emu_config *)info;
+
+	if (!info)
+		return -EINVAL;
+
+	conf->qp_num = iavf->max_be_lanqp;
+	return 0;
+}
+
+static int
+iavf_emu_get_mem_table(struct rte_emudev *dev,
+	rte_emudev_mem_table_t *tb)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	*tb = iavf->mem;
+
+	return 0;
+}
+
+static int
+iavf_emu_get_queue_info(struct rte_emudev *dev, uint32_t queue,
+		struct rte_emudev_q_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (queue < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[queue];
+		uint64_t base, size;
+
+		if (adq->ring_addr_lo == NULL ||
+		    adq->ring_addr_hi == NULL ||
+		    adq->ring_sz == NULL)
+			return -1;
+		base = RTE_IAVF_EMU_32_TO_64(*adq->ring_addr_hi,
+			*adq->ring_addr_lo);
+		size = *adq->ring_sz;
+		info->base = base;
+		info->size = size;
+		info->doorbell_id = queue;
+		/* RX AdminQ should have IRQ vector 0 */
+		info->irq_vector = queue - 1;
+	} else {
+		info->base = 0;
+		info->size = 0;
+		info->doorbell_id = queue;
+		info->irq_vector = -1;
+	}
+
+	return 0;
+}
+
+static int
+iavf_emu_get_irq_info(struct rte_emudev *dev, uint32_t vector,
+		struct rte_emudev_irq_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct iavf_emu_intr *intr = iavf->intr;
+	struct iavf_emu_intr_info *intr_info = &intr->info[vector];
+
+	if (vector >= intr->intr_num)
+		return -EINVAL;
+
+	info->eventfd = intr_info->fd;
+	info->enable = intr_info->enable;
+	info->vector = vector;
+
+	return 0;
+}
+
+static int
+iavf_emu_get_db_info(struct rte_emudev *dev, uint32_t doorbell,
+		struct rte_emudev_db_info *info)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if (doorbell < RTE_IAVF_EMU_ADMINQ_NUM) {
+		struct iavf_emu_adminQ *adq = &iavf->adq[doorbell];
+
+		info->data.mem.base = (uint64_t)adq->doorbell;
+		info->data.mem.size = adq->db_size;
+	} else {
+		struct iavf_emu_lanQ *lanq =
+			&iavf->lanq[doorbell - RTE_IAVF_EMU_ADMINQ_NUM];
+
+		info->data.mem.base = (uint64_t)lanq->doorbell;
+		info->data.mem.size = lanq->db_size;
+	}
+
+	info->flag |= RTE_EMUDEV_DB_MEM;
+	info->id = doorbell;
+
+	return 0;
+}
+
+static int
+iavf_emu_subs_ev(struct rte_emudev *dev,
+	rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	iavf->ops = (struct rte_iavf_emu_notify_ops *)ev_chnl;
+
+	return 0;
+}
+
+static int
+iavf_emu_unsubs_ev(struct rte_emudev *dev,
+		rte_emudev_event_chnl_t ev_chnl)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+
+	if ((struct rte_iavf_emu_notify_ops *)ev_chnl == iavf->ops) {
+		iavf->ops = NULL;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
+static int
+iavf_emu_get_attr(struct rte_emudev *dev, const char *attr_name,
+		rte_emudev_attr_t attr)
+{
+	struct iavf_emudev *iavf = (struct iavf_emudev *)dev->priv_data;
+	struct rte_vfio_user_reg_info *info;
+	int ret = 0;
+
+	info = &iavf->vfio->reg->reg_info[0];
+
+	if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ASQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ATQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_ARQ_HEAD))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VF_ARQH1;
+	else if (!strcmp(attr_name, RTE_IAVF_EMU_ATTR_RESET))
+		*(uint64_t *)attr = (uint64_t)(uintptr_t)info->base
+			+ IAVF_VFGEN_RSTAT;
+	else
+		ret = -EINVAL;
+
+	return ret;
+}
+
 struct rte_emudev_ops emu_iavf_ops = {
+	.dev_start = iavf_emu_dev_start,
+	.dev_stop = iavf_emu_dev_stop,
+	.dev_configure = iavf_emu_dev_configure,
 	.dev_close = iavf_emu_dev_close,
+	.dev_info_get = iavf_emu_get_dev_info,
+	.get_mem_table = iavf_emu_get_mem_table,
+	.get_queue_info = iavf_emu_get_queue_info,
+	.get_irq_info = iavf_emu_get_irq_info,
+	.get_db_info = iavf_emu_get_db_info,
+	.subscribe_event = iavf_emu_subs_ev,
+	.unsubscribe_event = iavf_emu_unsubs_ev,
+	.get_attr = iavf_emu_get_attr,
 };
drivers/emu/iavf/rte_iavf_emu.h +59 -0

84 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#42 lore raw

PATCH v3 7/8 test/emudev: introduce functional test

In reply to Chenbo Xia

This patch introduces functional test for emudev. The implementation of iavf emudev selftest is also added.

---
 app/test/meson.build                 |   5 +-
 app/test/test_emudev.c               |  29 +++++
 drivers/emu/iavf/iavf_emu.c          |   1 +
 drivers/emu/iavf/iavf_emu_internal.h |   1 +
 drivers/emu/iavf/iavf_emu_test.c     | 174 +++++++++++++++++++++++++++
 drivers/emu/iavf/meson.build         |   2 +-
 6 files changed, 210 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
app/test/meson.build +4 -1
index f5b15ac44c..b8b79bbc8b 100644
@@ -139,6 +139,7 @@ test_sources = files('commands.c',
 	'test_trace_register.c',
 	'test_trace_perf.c',
 	'test_vfio_user.c',
+	'test_emudev.c',
 	'test_version.c',
 	'virtual_pmd.c'
 )
@@ -176,7 +177,8 @@ test_deps = ['acl',
 	'stack',
 	'vfio_user',
 	'telemetry',
-	'timer'
+	'timer',
+	'emudev'
 ]
 
 # Each test is marked with flag true/false
@@ -327,6 +329,7 @@ driver_test_names = [
         'eventdev_selftest_octeontx',
         'eventdev_selftest_sw',
         'rawdev_autotest',
+        'emudev_autotest',
 ]
 
 dump_test_names = [
app/test/test_emudev.c new file +29 -0
new file mode 100644
index 0000000000..0dfce162ed
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_emudev.h>
+#include <rte_bus_vdev.h>
+
+#include "test.h"
+
+static int
+test_emudev_selftest_impl(const char *pmd, const char *opts)
+{
+	int ret = 0;
+
+	if (rte_emudev_get_dev_id(pmd) == -ENODEV)
+		ret = rte_vdev_init(pmd, opts);
+	if (ret)
+		return TEST_SKIPPED;
+
+	return rte_emudev_selftest(rte_emudev_get_dev_id(pmd));
+}
+
+static int
+test_emudev_selftest(void)
+{
+	return test_emudev_selftest_impl("emu_iavf", "sock=/tmp/sock1");
+}
+
+REGISTER_TEST_COMMAND(emudev_autotest, test_emudev_selftest);
drivers/emu/iavf/iavf_emu.c +1 -0
index cc56ec1c4b..8439677b65 100644
@@ -258,4 +258,5 @@ struct rte_emudev_ops emu_iavf_ops = {
 	.subscribe_event = iavf_emu_subs_ev,
 	.unsubscribe_event = iavf_emu_unsubs_ev,
 	.get_attr = iavf_emu_get_attr,
+	.dev_selftest = iavf_emu_selftest,
 };
drivers/emu/iavf/iavf_emu_internal.h +1 -0
index 10197c00ba..1ac7f96566 100644
@@ -65,4 +65,5 @@ struct iavf_emudev {
 };
 
 void iavf_emu_uninit_device(struct iavf_emudev *dev);
+int iavf_emu_selftest(uint16_t dev_id);
 #endif
drivers/emu/iavf/iavf_emu_test.c new file +174 -0

174 lines not on this page · raw message

drivers/emu/iavf/meson.build +1 -1
index 3cab2226b7..613783e407 100644
@@ -10,7 +10,7 @@ endif
 ext_deps += librt
 
 sources = files('iavf_emu.c', 'iavf_vfio_user.c',
-	'iavf_emudev.c')
+	'iavf_emu_test.c', 'iavf_emudev.c')
 
 deps += ['bus_vdev', 'emudev', 'vfio_user', 'common_iavf']
 
Signature · 1 line
2.17.1
author
#43 lore raw

PATCH v3 8/8 doc: update release notes for iavf emudev driver

In reply to Chenbo Xia

Update release notes for emulated iavf driver.

---
 doc/guides/rel_notes/release_21_02.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_21_02.rst +5 -1
index 9686930de3..90030079a5 100644
@@ -67,7 +67,7 @@ New Features
 
   See :doc:`../prog_guide/vfio_user_lib` for more information.
 
-* **Added emudev Library.**
+* **Added emudev Library and first emudev driver.**
 
   Added an experimental library ``librte_emudev`` to provide device abstraction
   for an emulated device.
@@ -77,6 +77,10 @@ New Features
   crypto and etc.). It can be attached to another data path driver (e.g, ethdev
   driver) to leverage the high performance of DPDK data path driver.
 
+  The first emudev driver is also introduced, which emulates software iavf
+  devices. This driver emulates all iavf device behavior except data path
+  handling.
+
   See :doc:`../prog_guide/emudev` for more information.
 
 Removed Items
Signature · 1 line
2.17.1
#44 lore raw
In reply to Chenbo Xia
Quoted 62 lines · On Thu, 14 Jan 2021 14:25:04 +0800 Chenbo Xia <[email protected]> wrote:

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/86498/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Design overview
---------------

                    +---------------------------------------+
                    |   +---------------+    +-----------+  |
                    |   |  iavf_emudev  |<-->| data path |  |
                    |   |    driver     |    |   driver  |  |
                    |   +---------------+    +-----------+  |
                    |           |                           |
                    | --------------------------- VDEV BUS  |
                    |           |                           |
                    |   +---------------+                   |
+--------------+    |   | vdev:         |                   |
| +----------+ |    |   | /path/to/vfio |                   |
| | Generic  | |    |   +---------------+                   |
| | vfio-dev | |    |           |                           |
| +----------+ |    |           |                           |
| +----------+ |    |      +----------+                     |
| | vfio-user| |    |      | vfio-user|                     |
| | client   | |<---|----->| server   |                     |
| +----------+ |    |      +----------+                     |
| QEMU/DPDK    |    | DPDK                                  |
+--------------+    +---------------------------------------+

- Generic vfio-dev/vfio-user client/vfio-user server
Above concepts are all introduced in librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/86498/

- vdev:/path/to/vfio.
It binds to vdev bus driver. The vdev device is defined by DPDK applications through command line as '--vdev=emu_iavf, path=/path/to/socket' in iavf_emudev case. Parameters in command line include device name (emu_iavf) which is used to identify corresponding driver (in this case, iavf_emudev driver), path=/path/to/socket which is used to open the transport interface to vfio-user client in QEMU/DPDK.

- data path driver.
The data path handling is splited to another standalone driver for modular design.

Quoted 9 lines

Why not rawdev for emulated device
----------------------------------
Instead of introducing new class emudev, emulated device could be presented as rawdev. However, existing rawdev APIs cannot meet the requirements of emulated device. There are three API categories for emudev. They are emudev device lifecycle management, backend facing APIs, and emudev device provider facing APIs respectively. Existing rawdev APIs could only cover lifecycle management APIs and some of backend facing APIs. Other APIs, even if added to rawdev API are not required by other rawdev applications.

This looks interesting but there has been on further work on it for three years. I recommend that the patch be dropped for now and resubmitted if/when the vfio_user patches get accepted.

#45 lore raw
In reply to Chenbo Xia
Quoted 106 lines · On Fri, 18 Dec 2020 15:47:28 +0800 Chenbo Xia <[email protected]> wrote:

This series introduces a new device abstraction called emudev for emulated devices. A new library (librte_emudev) is implemented. The first emudev driver is also introduced, which emulates Intel Adaptive Virtual Function (iavf) as a software network device.

This series has a dependency on librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

Background & Motivation
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU. Therefore, librte_vfio_user is introduced in DPDK to accommodate emulated devices for high performance I/O. Although vfio-user library provides possibility of emulating devices in DPDK, DPDK does not have a device abstraction for emulated devices. A good device abstraction will be useful for applications or high performance data path driver. With this consideration, emudev library is designed and implemented. It also make it possbile to keep modular design on emulated devices by implementing data path related logic in a standalone driver (e.g., an ethdev driver) and keeps the unrelated logic in the emudev driver.

Design overview
---------------

                    +---------------------------------------+
                    |   +---------------+    +-----------+  |
                    |   |  iavf_emudev  |<-->| data path |  |
                    |   |    driver     |    |   driver  |  |
                    |   +---------------+    +-----------+  |
                    |           |                           |
                    | --------------------------- VDEV BUS  |
                    |           |                           |
                    |   +---------------+                   |
+--------------+    |   | vdev:         |                   |
| +----------+ |    |   | /path/to/vfio |                   |
| | Generic  | |    |   +---------------+                   |
| | vfio-dev | |    |           |                           |
| +----------+ |    |           |                           |
| +----------+ |    |      +----------+                     |
| | vfio-user| |    |      | vfio-user|                     |
| | client   | |<---|----->| server   |                     |
| +----------+ |    |      +----------+                     |
| QEMU/DPDK    |    | DPDK                                  |
+--------------+    +---------------------------------------+

- Generic vfio-dev/vfio-user client/vfio-user server
Above concepts are all introduced in librte_vfio_user patch series:
http://patchwork.dpdk.org/cover/85389/

- vdev:/path/to/vfio.
It binds to vdev bus driver. The vdev device is defined by DPDK applications through command line as '--vdev=emu_iavf, path=/path/to/socket' in iavf_emudev case. Parameters in command line include device name (emu_iavf) which is used to identify corresponding driver (in this case, iavf_emudev driver), path=/path/to/socket which is used to open the transport interface to vfio-user client in QEMU/DPDK.

- data path driver.
The data path handling is splited to another standalone driver for modular design.

Chenbo Xia (8):
lib: introduce emudev library
doc: add emudev library guide
emu: introduce emulated iavf driver
emu/iavf: add vfio-user device register and unregister emu/iavf: add resource management and internal logic of iavf emu/iavf: add emudev operations to fit in emudev framework test/emudev: introduce functional test
doc: update release notes for iavf emudev driver

 MAINTAINERS                            |   12 +
 app/test/meson.build                   |    5 +-
 app/test/test_emudev.c                 |   29 +
 doc/guides/prog_guide/emudev.rst       |  122 +++
 doc/guides/prog_guide/index.rst        |    1 +
 doc/guides/rel_notes/release_21_02.rst |   16 +
 drivers/emu/iavf/iavf_emu.c            |  250 ++++++
 drivers/emu/iavf/iavf_emu_internal.h   |   69 ++
 drivers/emu/iavf/iavf_emu_test.c       |  174 ++++
 drivers/emu/iavf/iavf_emudev.c         |  237 ++++++
 drivers/emu/iavf/iavf_vfio_user.c      | 1053 ++++++++++++++++++++++++
 drivers/emu/iavf/iavf_vfio_user.h      |   57 ++
 drivers/emu/iavf/meson.build           |   17 +
 drivers/emu/iavf/rte_iavf_emu.h        |  119 +++
 drivers/emu/iavf/version.map           |    3 +
 drivers/emu/meson.build                |    6 +
 drivers/meson.build                    |    1 +
 lib/librte_emudev/meson.build          |    5 +
 lib/librte_emudev/rte_emudev.c         |  486 +++++++++++
 lib/librte_emudev/rte_emudev.h         |  410 +++++++++
 lib/librte_emudev/rte_emudev_vdev.h    |   53 ++
 lib/librte_emudev/version.map          |   27 +
 lib/meson.build                        |    2 +-
 23 files changed, 3152 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_emudev.c
 create mode 100644 doc/guides/prog_guide/emudev.rst
 create mode 100644 drivers/emu/iavf/iavf_emu.c
 create mode 100644 drivers/emu/iavf/iavf_emu_internal.h
 create mode 100644 drivers/emu/iavf/iavf_emu_test.c
 create mode 100644 drivers/emu/iavf/iavf_emudev.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.c
 create mode 100644 drivers/emu/iavf/iavf_vfio_user.h
 create mode 100644 drivers/emu/iavf/meson.build
Quoted 9 lines
 create mode 100644 drivers/emu/iavf/rte_iavf_emu.h
 create mode 100644 drivers/emu/iavf/version.map
 create mode 100644 drivers/emu/meson.build
 create mode 100644 lib/librte_emudev/meson.build
 create mode 100644 lib/librte_emudev/rte_emudev.c
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_emudev/rte_emudev_vdev.h
 create mode 100644 lib/librte_emudev/version.map

This patchset seems to have a dependency on a recent version of QEMU. Was it merged? The patchset would need to be rebased and retested on current DPDK. If still relevant please do that and resubmit.