Skip to contentSkip to conversation

ioring: network driver

RFC Superseded Stephen Hemminger 3 participants 72 messages Dec 10, 2024 – Feb 15

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover ioring: network driver

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statistics

 doc/guides/nics/features/ioring.ini |   14 +
 doc/guides/nics/index.rst           |    1 +
 doc/guides/nics/ioring.rst          |   66 ++
 drivers/net/ioring/meson.build      |   12 +
 drivers/net/ioring/rte_eth_ioring.c | 1067 +++++++++++++++++++++++++++
 drivers/net/meson.build             |    1 +
 6 files changed, 1161 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.45.2
#1 lore raw

RFC 1/8 net/ioring: introduce new driver

Add basic driver initialization, documentation, and device creation and basic documentation.

---
 doc/guides/nics/features/ioring.ini |   9 +
 doc/guides/nics/index.rst           |   1 +
 doc/guides/nics/ioring.rst          |  66 +++++++
 drivers/net/ioring/meson.build      |  12 ++
 drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++
 drivers/net/meson.build             |   1 +
 6 files changed, 351 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
new file mode 100644
index 0000000000..c4c57caaa4
@@ -0,0 +1,9 @@
+;
+; Supported features of the 'ioring' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux		     = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index 50688d9f64..e4d243622e 100644
@@ -41,6 +41,7 @@ Network Interface Controller Drivers
     igc
     intel_vf
     ionic
+    ioring
     ipn3ke
     ixgbe
     mana
doc/guides/nics/ioring.rst new file +66 -0

66 lines not on this page · raw message

drivers/net/ioring/meson.build new file +12 -0
new file mode 100644
index 0000000000..198063e51f
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+ext_deps += dependency('liburing', required:true)
+
+sources = files('rte_eth_ioring.c')
+require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0

262 lines not on this page · raw message

drivers/net/meson.build +1 -0
index dafd637ba4..b68a55e916 100644
@@ -33,6 +33,7 @@ drivers = [
         'idpf',
         'igc',
         'ionic',
+        'ioring',
         'ipn3ke',
         'ixgbe',
         'mana',
Signature · 1 line
2.45.2
#2 lore raw

RFC 2/8 net/ioring: implement link state

In reply to Stephen Hemminger

Add hooks to set kernel link up/down and report state.

---
 doc/guides/nics/features/ioring.ini |  1 +
 drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
index c4c57caaa4..d4bf70cb4f 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0

104 lines not on this page · raw message

Signature · 1 line
2.45.2
#3 lore raw

RFC 3/8 net/ioring: implement control functions

In reply to Stephen Hemminger

These internal ops, just force changes to kernel visible net device.

---
 doc/guides/nics/features/ioring.ini |  3 ++
 drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
index d4bf70cb4f..199c7cd31c 100644
@@ -5,6 +5,9 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0

93 lines not on this page · raw message

Signature · 1 line
2.45.2
#4 lore raw

RFC 4/8 net/ioring: implement management functions

In reply to Stephen Hemminger

Add start, stop, configure and info functions.

---
 drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
index de10a4d83f..b5d9c12bdf 100644
@@ -212,12 +212,16 @@ static int
 eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
+
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
 	struct ifreq ifr = { .ifr_mtu = mtu };
-	int ret;
 
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFMTU, &ifr);
+	int ret = ioctl(sock, SIOCSIFMTU, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFMTU) failed: %s", strerror(errno));
 		ret = -errno;
@@ -230,14 +234,17 @@ static int
 eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
-	struct ifreq ifr = { };
-	int ret;
 
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
+	struct ifreq ifr = { };
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 	ifr.ifr_hwaddr.sa_family = ARPHRD_ETHER;
 	memcpy(ifr.ifr_hwaddr.sa_data, addr, sizeof(*addr));
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFHWADDR, &ifr);
+	int ret = ioctl(sock, SIOCSIFHWADDR, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFHWADDR) failed: %s", strerror(errno));
 		ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 	return ret;
 }
 
+static int
+eth_dev_start(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
+	eth_dev_set_link_up(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_stop(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+	eth_dev_set_link_down(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_configure(struct rte_eth_dev *dev)
+{
+	/* rx/tx must be paired */
+	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int
+eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	dev_info->if_index = if_nametoindex(pmd->ifname);
+	dev_info->max_mac_addrs = 1;
+	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+
+	return 0;
+}
+
 static int
 eth_dev_close(struct rte_eth_dev *dev)
 {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 		close(pmd->keep_fd);
 		pmd->keep_fd = -1;
 	}
-
 	return 0;
 }
 
 static const struct eth_dev_ops ops = {
+	.dev_start		= eth_dev_start,
+	.dev_stop		= eth_dev_stop,
+	.dev_configure		= eth_dev_configure,
+	.dev_infos_get		= eth_dev_info,
 	.dev_close		= eth_dev_close,
 	.link_update		= eth_link_update,
 	.dev_set_link_up	= eth_dev_set_link_up,
Signature · 1 line
2.45.2
#5 lore raw

RFC 5/8 net/ioring: implement primary secondary fd passing

In reply to Stephen Hemminger

Add support for communicating fd's from primary to secondary.

---
 doc/guides/nics/features/ioring.ini |   1 +
 drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++-
 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
index 199c7cd31c..da47062adb 100644
@@ -8,6 +8,7 @@ Link status          = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
+Multiprocess aware   = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2

181 lines not on this page · raw message

Signature · 1 line
2.45.2
#6 lore raw

RFC 6/8 net/ioring: implement receive and transmit

In reply to Stephen Hemminger

Use io_uring to read and write from TAP device.

---
 drivers/net/ioring/rte_eth_ioring.c | 364 +++++++++++++++++++++++++++-
 1 file changed, 363 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +363 -1
index ddef57adfb..fa79bc5667 100644
@@ -2,6 +2,7 @@
  * Copyright (c) Stephen Hemminger
  */
 
+#include <assert.h>
 #include <ctype.h>
 #include <errno.h>
 #include <fcntl.h>
@@ -9,8 +10,10 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <liburing.h>
 #include <sys/ioctl.h>
 #include <sys/socket.h>
+#include <sys/uio.h>
 #include <net/if.h>
 #include <linux/if.h>
 #include <linux/if_arp.h>
@@ -27,6 +30,13 @@
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
+#define IORING_DEFAULT_BURST	64
+#define IORING_NUM_BUFFERS	1024
+#define IORING_MAX_QUEUES	128
+
+
+static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
+
 #define IORING_DEFAULT_IFNAME	"enio%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -34,6 +44,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);
 #define RTE_LOGTYPE_IORING ioring_logtype
 #define PMD_LOG(level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__)
 
+#ifdef RTE_ETHDEV_DEBUG_RX
+#define PMD_RX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_RX_LOG(...) do { } while (0)
+#endif
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+#define PMD_TX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_TX_LOG(...) do { } while (0)
+#endif
+
 #define IORING_IFACE_ARG	"iface"
 #define IORING_PERSIST_ARG	"persist"
 
@@ -43,6 +67,30 @@ static const char * const valid_arguments[] = {
 	NULL
 };
 
+struct rx_queue {
+	struct rte_mempool *mb_pool;	/* rx buffer pool */
+	struct io_uring io_ring;	/* queue of posted read's */
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint64_t rx_packets;
+	uint64_t rx_bytes;
+	uint64_t rx_nombuf;
+	uint64_t rx_errors;
+};
+
+struct tx_queue {
+	struct io_uring io_ring;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t free_thresh;
+
+	uint64_t tx_packets;
+	uint64_t tx_bytes;
+	uint64_t tx_errors;
+};
+
 struct pmd_internals {
 	int keep_fd;			/* keep alive file descriptor */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
@@ -300,6 +348,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->if_index = if_nametoindex(pmd->ifname);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+	dev_info->max_rx_queues = IORING_MAX_QUEUES;
+	dev_info->max_tx_queues = IORING_MAX_QUEUES;
+	dev_info->min_rx_bufsize = 0;
+
+	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
+		.burst_size = IORING_DEFAULT_BURST,
+		.ring_size = IORING_NUM_BUFFERS,
+		.nb_queues = 1,
+	};
 
 	return 0;
 }
@@ -311,6 +368,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 
 	PMD_LOG(INFO, "Closing %s", pmd->ifname);
 
+	int *fds = dev->process_private;
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		if (fds[i] == -1)
+			continue;
+		close(fds[i]);
+		fds[i] = -1;
+	}
+
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
@@ -324,6 +389,296 @@ eth_dev_close(struct rte_eth_dev *dev)
 	return 0;
 }
 
+/* Setup another fd to TAP device for the queue */
+static int
+eth_queue_setup(struct rte_eth_dev *dev, const char *name, uint16_t queue_id)
+{
+	int *fds = dev->process_private;
+
+	if (fds[queue_id] != -1)
+		return 0;	/* already setup */
+
+	struct ifreq ifr = { };
+	int tap_fd = tap_open(name, &ifr, 0);
+	if (tap_fd < 0) {
+		PMD_LOG(ERR, "tap_open failed");
+		return -1;
+	}
+
+	PMD_LOG(DEBUG, "opened %d for queue %u", tap_fd, queue_id);
+	fds[queue_id] = tap_fd;
+	return 0;
+}
+
+static int
+eth_queue_fd(uint16_t port_id, uint16_t queue_id)
+{
+	struct rte_eth_dev *dev = &rte_eth_devices[port_id];
+	int *fds = dev->process_private;
+
+	return fds[queue_id];
+}
+
+/* setup an submit queue to read mbuf */
+static inline void
+eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
+{
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+
+	if (unlikely(sqe == NULL)) {
+		PMD_LOG(DEBUG, "io_uring no rx sqe");
+		rxq->rx_errors++;
+	} else {
+		void *base = rte_pktmbuf_mtod(mb, void *);
+		size_t len = mb->buf_len;
+
+		io_uring_prep_read(sqe, fd, base, len, 0);
+		io_uring_sqe_set_data(sqe, mb);
+	}
+}
+
+static uint16_t
+eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct rx_queue *rxq = queue;
+	struct io_uring_cqe *cqe;
+	unsigned int head, num_cqe = 0;
+	uint16_t num_rx = 0;
+	uint32_t num_bytes = 0;
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	io_uring_for_each_cqe(&rxq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+		ssize_t len = cqe->res;
+
+		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
+		num_cqe++;
+
+		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
+			if (len < 0)
+				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
+			else
+				PMD_LOG(ERR, "io_uring_read missing hdr");
+
+			rxq->rx_errors++;
+			goto resubmit;
+		}
+
+		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(nmb == 0)) {
+			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+			++rxq->rx_nombuf;
+			goto resubmit;
+		}
+
+		mb->pkt_len = len;
+		mb->data_len = len;
+		mb->port = rxq->port_id;
+		__rte_mbuf_sanity_check(mb, 1);
+
+		num_bytes += len;
+		bufs[num_rx++] = mb;
+
+		mb = nmb;
+resubmit:
+		eth_rx_submit(rxq, fd, mb);
+
+		if (num_rx == nb_pkts)
+			break;
+	}
+	io_uring_cq_advance(&rxq->io_ring, num_cqe);
+
+	rxq->rx_packets += num_rx;
+	rxq->rx_bytes += num_bytes;
+	return num_rx;
+}
+
+static int
+eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
+		   unsigned int socket_id,
+		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   struct rte_mempool *mb_pool)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	PMD_LOG(DEBUG, "setup port %u queue %u rx_descriptors %u",
+		dev->data->port_id, queue_id, nb_rx_desc);
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct rx_queue *rxq = rte_zmalloc_socket(NULL, sizeof(*rxq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (rxq == NULL) {
+		PMD_LOG(ERR, "rxq alloc failed");
+		return -1;
+	}
+
+	rxq->mb_pool = mb_pool;
+	rxq->port_id = dev->data->port_id;
+	rxq->queue_id = queue_id;
+	dev->data->rx_queues[queue_id] = rxq;
+
+	struct rte_mbuf *mbufs[nb_rx_desc];
+	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
+		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
+		return -1;
+	}
+
+	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		goto error;
+	}
+
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	for (uint16_t i = 0; i < nb_rx_desc; i++) {
+		struct rte_mbuf *mb = mbufs[i];
+
+		eth_rx_submit(rxq, fd, mb);
+	}
+
+	io_uring_submit(&rxq->io_ring);
+	return 0;
+
+error:
+	rte_pktmbuf_free_bulk(mbufs, nb_rx_desc);
+	return -1;
+}
+
+static void
+eth_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct rx_queue *rxq = dev->data->rx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&rxq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&rxq->io_ring);
+}
+
+static int
+eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+		   uint16_t nb_tx_desc, unsigned int socket_id,
+		   const struct rte_eth_txconf *tx_conf)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct tx_queue *txq = rte_zmalloc_socket(NULL, sizeof(*txq), RTE_CACHE_LINE_SIZE, socket_id);
+	if (txq == NULL) {
+		PMD_LOG(ERR, "txq alloc failed");
+		return -1;
+	}
+
+	txq->port_id = dev->data->port_id;
+	txq->queue_id = queue_id;
+	txq->free_thresh = tx_conf->tx_free_thresh;
+	dev->data->tx_queues[queue_id] = txq;
+
+	if (io_uring_queue_init(nb_tx_desc, &txq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	return 0;
+}
+
+static void
+eth_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct tx_queue *txq = dev->data->tx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&txq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&txq->io_ring);
+}
+
+static void
+eth_ioring_tx_cleanup(struct tx_queue *txq)
+{
+	struct io_uring_cqe *cqe;
+	unsigned int head;
+	unsigned int tx_done = 0;
+	uint64_t tx_bytes = 0;
+
+	io_uring_for_each_cqe(&txq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+
+		PMD_TX_LOG(DEBUG, " mbuf len %u result: %d", mb->pkt_len, cqe->res);
+		if (unlikely(cqe->res < 0)) {
+			++txq->tx_errors;
+		} else {
+			++tx_done;
+			tx_bytes += mb->pkt_len;
+		}
+
+		rte_pktmbuf_free(mb);
+	}
+	io_uring_cq_advance(&txq->io_ring, tx_done);
+
+	txq->tx_packets += tx_done;
+	txq->tx_bytes += tx_bytes;
+}
+
+static uint16_t
+eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct tx_queue *txq = queue;
+	uint16_t num_tx;
+
+	if (unlikely(nb_pkts == 0))
+		return 0;
+
+	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
+
+	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
+		eth_ioring_tx_cleanup(txq);
+
+	int fd = eth_queue_fd(txq->port_id, txq->queue_id);
+
+	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
+		struct rte_mbuf *mb = bufs[num_tx];
+
+		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+		if (sqe == NULL)
+			break;	/* submit ring is full */
+
+		io_uring_sqe_set_data(sqe, mb);
+
+		if (rte_mbuf_refcnt_read(mb) == 1 &&
+		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
+			void *base = rte_pktmbuf_mtod(mb, void *);
+			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+
+			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
+		} else {
+			PMD_LOG(ERR, "Can't do mbuf without space yet!");
+			++txq->tx_errors;
+			continue;
+		}
+	}
+	if (num_tx > 0)
+		io_uring_submit(&txq->io_ring);
+
+	return num_tx;
+}
+
 static const struct eth_dev_ops ops = {
 	.dev_start		= eth_dev_start,
 	.dev_stop		= eth_dev_stop,
@@ -339,9 +694,12 @@ static const struct eth_dev_ops ops = {
 	.promiscuous_disable	= eth_dev_promiscuous_disable,
 	.allmulticast_enable	= eth_dev_allmulticast_enable,
 	.allmulticast_disable	= eth_dev_allmulticast_disable,
+	.rx_queue_setup		= eth_rx_queue_setup,
+	.rx_queue_release	= eth_rx_queue_release,
+	.tx_queue_setup		= eth_tx_queue_setup,
+	.tx_queue_release	= eth_tx_queue_release,
 };
 
-
 static int
 ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 {
@@ -379,6 +737,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	}
 
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
+
+	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_burst = eth_ioring_tx;
+
 	return 0;
 
 error:
Signature · 1 line
2.45.2
#7 lore raw

RFC 7/8 net/ioring: add VLAN support

In reply to Stephen Hemminger

Add support for VLAN insert and stripping.

---
 drivers/net/ioring/rte_eth_ioring.c | 45 +++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +43 -2
index fa79bc5667..a2bfefec45 100644
@@ -34,6 +34,8 @@
 #define IORING_NUM_BUFFERS	1024
 #define IORING_MAX_QUEUES	128
 
+#define IORING_TX_OFFLOAD	RTE_ETH_TX_OFFLOAD_VLAN_INSERT
+#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_VLAN_STRIP
 
 static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
 
@@ -70,6 +72,7 @@ static const char * const valid_arguments[] = {
 struct rx_queue {
 	struct rte_mempool *mb_pool;	/* rx buffer pool */
 	struct io_uring io_ring;	/* queue of posted read's */
+	uint64_t offloads;
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -81,6 +84,7 @@ struct rx_queue {
 
 struct tx_queue {
 	struct io_uring io_ring;
+	uint64_t offloads;
 
 	uint16_t port_id;
 	uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 			goto resubmit;
 		}
 
+		if (rxq->offloads & RTE_ETH_RX_OFFLOAD_VLAN_STRIP)
+			rte_vlan_strip(mb);
+
 		mb->pkt_len = len;
 		mb->data_len = len;
 		mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 static int
 eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
-		   unsigned int socket_id,
-		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,
 		   struct rte_mempool *mb_pool)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_de
 		return -1;
 	}
 
+	rxq->offloads = rx_conf->offloads;
 	rxq->mb_pool = mb_pool;
 	rxq->port_id = dev->data->port_id;
 	rxq->queue_id = queue_id;
@@ -582,6 +589,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	txq->port_id = dev->data->port_id;
 	txq->queue_id = queue_id;
+	txq->offloads = tx_conf->offloads;
 	txq->free_thresh = tx_conf->tx_free_thresh;
 	dev->data->tx_queues[queue_id] = txq;
 
@@ -636,6 +644,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq)
 	txq->tx_bytes += tx_bytes;
 }
 
+static uint16_t
+eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+	uint16_t nb_tx;
+	int error;
+
+	for (nb_tx = 0; nb_tx < nb_pkts; nb_tx++) {
+		struct rte_mbuf *m = tx_pkts[nb_tx];
+
+#ifdef RTE_LIBRTE_ETHDEV_DEBUG
+		error = rte_validate_tx_offload(m);
+		if (unlikely(error)) {
+			rte_errno = -error;
+			break;
+		}
+#endif
+		/* Do VLAN tag insertion */
+		if (unlikely(m->ol_flags & RTE_MBUF_F_TX_VLAN)) {
+			error = rte_vlan_insert(&m);
+			/* rte_vlan_insert() may change pointer */
+			tx_pkts[nb_tx] = m;
+
+			if (unlikely(error)) {
+				rte_errno = -error;
+				break;
+			}
+		}
+	}
+
+	return nb_tx;
+}
+
 static uint16_t
 eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 {
@@ -739,6 +779,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
 
 	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_prepare = eth_ioring_tx_prepare;
 	dev->tx_pkt_burst = eth_ioring_tx;
 
 	return 0;
Signature · 1 line
2.45.2
#8 lore raw

RFC 8/8 net/ioring: implement statistics

In reply to Stephen Hemminger

Add support for basic statistics

---
 drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
drivers/net/ioring/rte_eth_ioring.c +57 -0
index a2bfefec45..f58740197d 100644
@@ -365,6 +365,61 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	return 0;
 }
 
+static int
+eth_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
+{
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		const struct rx_queue *rxq = dev->data->rx_queues[i];
+
+		stats->ipackets += rxq->rx_packets;
+		stats->ibytes += rxq->rx_bytes;
+		stats->ierrors += rxq->rx_errors;
+		stats->rx_nombuf += rxq->rx_nombuf;
+
+		if (i < RTE_ETHDEV_QUEUE_STAT_CNTRS) {
+			stats->q_ipackets[i] = rxq->rx_packets;
+			stats->q_ibytes[i] = rxq->rx_bytes;
+		}
+	}
+
+	for (uint16_t i = 0; i < dev->data->nb_tx_queues; i++) {
+		const struct tx_queue *txq = dev->data->tx_queues[i];
+
+		stats->opackets += txq->tx_packets;
+		stats->obytes += txq->tx_bytes;
+		stats->oerrors += txq->tx_errors;
+
+		if (i < RTE_ETHDEV_QUEUE_STAT_CNTRS) {
+			stats->q_opackets[i] = txq->tx_packets;
+			stats->q_obytes[i] = txq->tx_bytes;
+		}
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_stats_reset(struct rte_eth_dev *dev)
+{
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		struct rx_queue *rxq = dev->data->rx_queues[i];
+
+		rxq->rx_packets = 0;
+		rxq->rx_bytes = 0;
+		rxq->rx_nombuf = 0;
+		rxq->rx_errors = 0;
+	}
+
+	for (uint16_t i = 0; i < dev->data->nb_tx_queues; i++) {
+		struct tx_queue *txq = dev->data->tx_queues[i];
+
+		txq->tx_packets = 0;
+		txq->tx_bytes = 0;
+		txq->tx_errors = 0;
+	}
+	return 0;
+}
+
 static int
 eth_dev_close(struct rte_eth_dev *dev)
 {
@@ -734,6 +789,8 @@ static const struct eth_dev_ops ops = {
 	.promiscuous_disable	= eth_dev_promiscuous_disable,
 	.allmulticast_enable	= eth_dev_allmulticast_enable,
 	.allmulticast_disable	= eth_dev_allmulticast_disable,
+	.stats_get              = eth_dev_stats_get,
+	.stats_reset            = eth_dev_stats_reset,
 	.rx_queue_setup		= eth_rx_queue_setup,
 	.rx_queue_release	= eth_rx_queue_release,
 	.tx_queue_setup		= eth_tx_queue_setup,
Signature · 1 line
2.45.2
#9 lore raw
In reply to Stephen Hemminger
Quoted 12 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

Quoted 4 lines

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

Quoted 24 lines

Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statistics

 doc/guides/nics/features/ioring.ini |   14 +
 doc/guides/nics/index.rst           |    1 +
 doc/guides/nics/ioring.rst          |   66 ++
 drivers/net/ioring/meson.build      |   12 +
 drivers/net/ioring/rte_eth_ioring.c | 1067 +++++++++++++++++++++++++++
 drivers/net/meson.build             |    1 +
 6 files changed, 1161 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c

--
2.45.2

#10 lore raw
Quoted 15 lines · On Wed, 11 Dec 2024 11:34:39 +0000 Konstantin Ananyev…
Quoted 10 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

Quoted 4 lines

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Quoted 4 lines

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

There are some userspaces that look for "e*" in name for some setups. But names are totally abitrary

#11 lore raw
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Quoted 6 lines
Quoted 4 lines

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

There are some userspaces that look for "e*" in name for some setups.

Didn't get you here, pls try to re-phrase.

Quoted 1 line

But names are totally abitrary

#12 lore raw
Quoted 21 lines

From: Konstantin Ananyev [mailto:[email protected]]

Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

Quoted 4 lines

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "net/tap_ioring".

Quoted 2 lines
Quoted 2 lines

There are some userspaces that look for "e*" in name for some setups.

Stephen is referring to the device name of an instantiated interface, e.g. "eth0".

And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)

Quoted 4 lines

Didn't get you here, pls try to re-phrase.

Quoted 1 line

But names are totally arbitrary

#13 lore raw
Quoted 19 lines
Quoted 19 lines

From: Konstantin Ananyev [mailto:[email protected]]

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "net/tap_ioring".

Yep, that what I meant.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

There are some userspaces that look for "e*" in name for some setups.

Stephen is referring to the device name of an instantiated interface, e.g. "eth0".

And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)

Ok... and why such practice should prevent us to name PMD itself in a way we think is appropriate?

Quoted 6 lines
Quoted 4 lines

Didn't get you here, pls try to re-phrase.

Quoted 1 line

But names are totally arbitrary

#14 lore raw
Quoted 17 lines · On Fri, 20 Dec 2024 14:34:27 +0000 Konstantin Ananyev…
Quoted 17 lines
Quoted 17 lines

From: Konstantin Ananyev [mailto:[email protected]]

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.

Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "net/tap_ioring".

Yep, that what I meant.

My thoughts, are shorter name is better, and avoids confusion. There are already multiple drivers that create tap devices: tap and virtio_user.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

There are some userspaces that look for "e*" in name for some setups.

Stephen is referring to the device name of an instantiated interface, e.g. "eth0".

And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)

Ok... and why such practice should prevent us to name PMD itself in a way we think is appropriate?

I am more leaning towards not having a default name at all. The policy should be done by Linux (udev) not DPDK. If user wants a name they can add it via devargs.

#15 lore raw

v2 cover ioring: network driver

In reply to Stephen Hemminger

This is initial work of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statistics

 doc/guides/nics/features/ioring.ini |   16 +
 doc/guides/nics/index.rst           |    1 +
 doc/guides/nics/ioring.rst          |   66 ++
 drivers/net/ioring/meson.build      |   15 +
 drivers/net/ioring/rte_eth_ioring.c | 1068 +++++++++++++++++++++++++++
 drivers/net/meson.build             |    1 +
 6 files changed, 1167 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.45.2
#16 lore raw

PATCH v2 1/8 net/ioring: introduce new driver

Add basic driver initialization, documentation, and device creation and basic documentation.

---
 doc/guides/nics/features/ioring.ini |   9 +
 doc/guides/nics/index.rst           |   1 +
 doc/guides/nics/ioring.rst          |  66 +++++++
 drivers/net/ioring/meson.build      |  15 ++
 drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++
 drivers/net/meson.build             |   1 +
 6 files changed, 354 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
new file mode 100644
index 0000000000..c4c57caaa4
@@ -0,0 +1,9 @@
+;
+; Supported features of the 'ioring' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux		     = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index 50688d9f64..e4d243622e 100644
@@ -41,6 +41,7 @@ Network Interface Controller Drivers
     igc
     intel_vf
     ionic
+    ioring
     ipn3ke
     ixgbe
     mana
doc/guides/nics/ioring.rst new file +66 -0

66 lines not on this page · raw message

drivers/net/ioring/meson.build new file +15 -0
new file mode 100644
index 0000000000..264554d069
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+dep = dependency('liburing', required:false)
+reason = 'missing dependency, "liburing"'
+build = dep.found()
+ext_deps += dep
+
+sources = files('rte_eth_ioring.c')
+require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0

262 lines not on this page · raw message

drivers/net/meson.build +1 -0
index dafd637ba4..b68a55e916 100644
@@ -33,6 +33,7 @@ drivers = [
         'idpf',
         'igc',
         'ionic',
+        'ioring',
         'ipn3ke',
         'ixgbe',
         'mana',
Signature · 1 line
2.45.2
#17 lore raw

RE: [PATCH v2 1/8] net/ioring: introduce new driver

drivers/net/ioring/rte_eth_ioring.c line 70

Quoted 8 lines
drivers/net/ioring/rte_eth_ioring.c new file +8 -0
+	int features = 0;
+	if (ioctl(tap_fd, TUNGETFEATURES, &features) < 0) {
+		PMD_LOG(ERR, "ioctl(TUNGETFEATURES) %s", strerror(errno));
+		goto error;
+	}
+
+	int flags = IFF_TAP | IFF_MULTI_QUEUE | IFF_NO_PI;
+	if ((features & flags) == 0) {

Comparison will only fail if all three flags are missing. Should be:
if ((features & flags) != flags) {

drivers/net/ioring/rte_eth_ioring.c line 72

Quoted 2 lines
drivers/net/ioring/rte_eth_ioring.c new file +2 -0
+		PMD_LOG(ERR, "TUN features %#x missing support for %#x",
+			features, features & flags);

Should be:
features, (features & flags) ^ flags);

Quoted 2 lines
(quoted diff) +2 -0
+		goto error;
+	}
#18 lore raw

PATCH v2 2/8 net/ioring: implement link state

In reply to Stephen Hemminger

Add hooks to set kernel link up/down and report state.

---
 doc/guides/nics/features/ioring.ini |  1 +
 drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
index c4c57caaa4..d4bf70cb4f 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0

104 lines not on this page · raw message

Signature · 1 line
2.45.2
#19 lore raw

PATCH v2 3/8 net/ioring: implement control functions

In reply to Stephen Hemminger

These internal ops, just force changes to kernel visible net device.

---
 doc/guides/nics/features/ioring.ini |  3 ++
 drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
index d4bf70cb4f..199c7cd31c 100644
@@ -5,6 +5,9 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0

93 lines not on this page · raw message

Signature · 1 line
2.45.2
#20 lore raw

PATCH v2 4/8 net/ioring: implement management functions

In reply to Stephen Hemminger

Add start, stop, configure and info functions.

---
 drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
index de10a4d83f..b5d9c12bdf 100644
@@ -212,12 +212,16 @@ static int
 eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
+
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
 	struct ifreq ifr = { .ifr_mtu = mtu };
-	int ret;
 
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFMTU, &ifr);
+	int ret = ioctl(sock, SIOCSIFMTU, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFMTU) failed: %s", strerror(errno));
 		ret = -errno;
@@ -230,14 +234,17 @@ static int
 eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
-	struct ifreq ifr = { };
-	int ret;
 
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
+	struct ifreq ifr = { };
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 	ifr.ifr_hwaddr.sa_family = ARPHRD_ETHER;
 	memcpy(ifr.ifr_hwaddr.sa_data, addr, sizeof(*addr));
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFHWADDR, &ifr);
+	int ret = ioctl(sock, SIOCSIFHWADDR, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFHWADDR) failed: %s", strerror(errno));
 		ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 	return ret;
 }
 
+static int
+eth_dev_start(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
+	eth_dev_set_link_up(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_stop(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+	eth_dev_set_link_down(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_configure(struct rte_eth_dev *dev)
+{
+	/* rx/tx must be paired */
+	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int
+eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	dev_info->if_index = if_nametoindex(pmd->ifname);
+	dev_info->max_mac_addrs = 1;
+	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+
+	return 0;
+}
+
 static int
 eth_dev_close(struct rte_eth_dev *dev)
 {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 		close(pmd->keep_fd);
 		pmd->keep_fd = -1;
 	}
-
 	return 0;
 }
 
 static const struct eth_dev_ops ops = {
+	.dev_start		= eth_dev_start,
+	.dev_stop		= eth_dev_stop,
+	.dev_configure		= eth_dev_configure,
+	.dev_infos_get		= eth_dev_info,
 	.dev_close		= eth_dev_close,
 	.link_update		= eth_link_update,
 	.dev_set_link_up	= eth_dev_set_link_up,
Signature · 1 line
2.45.2
#21 lore raw

PATCH v2 5/8 net/ioring: implement primary secondary fd passing

In reply to Stephen Hemminger

Add support for communicating fd's from primary to secondary.

---
 doc/guides/nics/features/ioring.ini |   1 +
 drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++-
 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
index 199c7cd31c..da47062adb 100644
@@ -8,6 +8,7 @@ Link status          = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
+Multiprocess aware   = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2

181 lines not on this page · raw message

Signature · 1 line
2.45.2
#22 lore raw

PATCH v2 6/8 net/ioring: implement receive and transmit

In reply to Stephen Hemminger

Use io_uring to read and write from TAP device.

---
 drivers/net/ioring/rte_eth_ioring.c | 365 +++++++++++++++++++++++++++-
 1 file changed, 364 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +364 -1
index ddef57adfb..8dd717cb9d 100644
@@ -2,6 +2,7 @@
  * Copyright (c) Stephen Hemminger
  */
 
+#include <assert.h>
 #include <ctype.h>
 #include <errno.h>
 #include <fcntl.h>
@@ -9,8 +10,10 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <liburing.h>
 #include <sys/ioctl.h>
 #include <sys/socket.h>
+#include <sys/uio.h>
 #include <net/if.h>
 #include <linux/if.h>
 #include <linux/if_arp.h>
@@ -27,6 +30,13 @@
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
+#define IORING_DEFAULT_BURST	64
+#define IORING_NUM_BUFFERS	1024
+#define IORING_MAX_QUEUES	128
+
+
+static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
+
 #define IORING_DEFAULT_IFNAME	"enio%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -34,6 +44,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);
 #define RTE_LOGTYPE_IORING ioring_logtype
 #define PMD_LOG(level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__)
 
+#ifdef RTE_ETHDEV_DEBUG_RX
+#define PMD_RX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_RX_LOG(...) do { } while (0)
+#endif
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+#define PMD_TX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_TX_LOG(...) do { } while (0)
+#endif
+
 #define IORING_IFACE_ARG	"iface"
 #define IORING_PERSIST_ARG	"persist"
 
@@ -43,6 +67,30 @@ static const char * const valid_arguments[] = {
 	NULL
 };
 
+struct rx_queue {
+	struct rte_mempool *mb_pool;	/* rx buffer pool */
+	struct io_uring io_ring;	/* queue of posted read's */
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint64_t rx_packets;
+	uint64_t rx_bytes;
+	uint64_t rx_nombuf;
+	uint64_t rx_errors;
+};
+
+struct tx_queue {
+	struct io_uring io_ring;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t free_thresh;
+
+	uint64_t tx_packets;
+	uint64_t tx_bytes;
+	uint64_t tx_errors;
+};
+
 struct pmd_internals {
 	int keep_fd;			/* keep alive file descriptor */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
@@ -300,6 +348,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->if_index = if_nametoindex(pmd->ifname);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+	dev_info->max_rx_queues = IORING_MAX_QUEUES;
+	dev_info->max_tx_queues = IORING_MAX_QUEUES;
+	dev_info->min_rx_bufsize = 0;
+
+	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
+		.burst_size = IORING_DEFAULT_BURST,
+		.ring_size = IORING_NUM_BUFFERS,
+		.nb_queues = 1,
+	};
 
 	return 0;
 }
@@ -311,6 +368,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 
 	PMD_LOG(INFO, "Closing %s", pmd->ifname);
 
+	int *fds = dev->process_private;
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		if (fds[i] == -1)
+			continue;
+		close(fds[i]);
+		fds[i] = -1;
+	}
+
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
@@ -324,6 +389,297 @@ eth_dev_close(struct rte_eth_dev *dev)
 	return 0;
 }
 
+/* Setup another fd to TAP device for the queue */
+static int
+eth_queue_setup(struct rte_eth_dev *dev, const char *name, uint16_t queue_id)
+{
+	int *fds = dev->process_private;
+
+	if (fds[queue_id] != -1)
+		return 0;	/* already setup */
+
+	struct ifreq ifr = { };
+	int tap_fd = tap_open(name, &ifr, 0);
+	if (tap_fd < 0) {
+		PMD_LOG(ERR, "tap_open failed");
+		return -1;
+	}
+
+	PMD_LOG(DEBUG, "opened %d for queue %u", tap_fd, queue_id);
+	fds[queue_id] = tap_fd;
+	return 0;
+}
+
+static int
+eth_queue_fd(uint16_t port_id, uint16_t queue_id)
+{
+	struct rte_eth_dev *dev = &rte_eth_devices[port_id];
+	int *fds = dev->process_private;
+
+	return fds[queue_id];
+}
+
+/* setup an submit queue to read mbuf */
+static inline void
+eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
+{
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+
+	if (unlikely(sqe == NULL)) {
+		PMD_LOG(DEBUG, "io_uring no rx sqe");
+		rxq->rx_errors++;
+	} else {
+		void *base = rte_pktmbuf_mtod(mb, void *);
+		size_t len = mb->buf_len;
+
+		io_uring_prep_read(sqe, fd, base, len, 0);
+		io_uring_sqe_set_data(sqe, mb);
+	}
+}
+
+static uint16_t
+eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct rx_queue *rxq = queue;
+	struct io_uring_cqe *cqe;
+	unsigned int head, num_cqe = 0;
+	uint16_t num_rx = 0;
+	uint32_t num_bytes = 0;
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	io_uring_for_each_cqe(&rxq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+		ssize_t len = cqe->res;
+
+		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
+		num_cqe++;
+
+		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
+			if (len < 0)
+				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
+			else
+				PMD_LOG(ERR, "io_uring_read missing hdr");
+
+			rxq->rx_errors++;
+			goto resubmit;
+		}
+
+		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(nmb == 0)) {
+			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+			++rxq->rx_nombuf;
+			goto resubmit;
+		}
+
+		mb->pkt_len = len;
+		mb->data_len = len;
+		mb->port = rxq->port_id;
+		__rte_mbuf_sanity_check(mb, 1);
+
+		num_bytes += len;
+		bufs[num_rx++] = mb;
+
+		mb = nmb;
+resubmit:
+		eth_rx_submit(rxq, fd, mb);
+
+		if (num_rx == nb_pkts)
+			break;
+	}
+	io_uring_cq_advance(&rxq->io_ring, num_cqe);
+
+	rxq->rx_packets += num_rx;
+	rxq->rx_bytes += num_bytes;
+	return num_rx;
+}
+
+static int
+eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
+		   unsigned int socket_id,
+		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   struct rte_mempool *mb_pool)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	PMD_LOG(DEBUG, "setup port %u queue %u rx_descriptors %u",
+		dev->data->port_id, queue_id, nb_rx_desc);
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct rx_queue *rxq = rte_zmalloc_socket(NULL, sizeof(*rxq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (rxq == NULL) {
+		PMD_LOG(ERR, "rxq alloc failed");
+		return -1;
+	}
+
+	rxq->mb_pool = mb_pool;
+	rxq->port_id = dev->data->port_id;
+	rxq->queue_id = queue_id;
+	dev->data->rx_queues[queue_id] = rxq;
+
+	struct rte_mbuf *mbufs[nb_rx_desc];
+	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
+		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
+		return -1;
+	}
+
+	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		goto error;
+	}
+
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	for (uint16_t i = 0; i < nb_rx_desc; i++) {
+		struct rte_mbuf *mb = mbufs[i];
+
+		eth_rx_submit(rxq, fd, mb);
+	}
+
+	io_uring_submit(&rxq->io_ring);
+	return 0;
+
+error:
+	rte_pktmbuf_free_bulk(mbufs, nb_rx_desc);
+	return -1;
+}
+
+static void
+eth_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct rx_queue *rxq = dev->data->rx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&rxq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&rxq->io_ring);
+}
+
+static int
+eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+		   uint16_t nb_tx_desc, unsigned int socket_id,
+		   const struct rte_eth_txconf *tx_conf)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct tx_queue *txq = rte_zmalloc_socket(NULL, sizeof(*txq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (txq == NULL) {
+		PMD_LOG(ERR, "txq alloc failed");
+		return -1;
+	}
+
+	txq->port_id = dev->data->port_id;
+	txq->queue_id = queue_id;
+	txq->free_thresh = tx_conf->tx_free_thresh;
+	dev->data->tx_queues[queue_id] = txq;
+
+	if (io_uring_queue_init(nb_tx_desc, &txq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	return 0;
+}
+
+static void
+eth_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct tx_queue *txq = dev->data->tx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&txq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&txq->io_ring);
+}
+
+static void
+eth_ioring_tx_cleanup(struct tx_queue *txq)
+{
+	struct io_uring_cqe *cqe;
+	unsigned int head;
+	unsigned int tx_done = 0;
+	uint64_t tx_bytes = 0;
+
+	io_uring_for_each_cqe(&txq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+
+		PMD_TX_LOG(DEBUG, " mbuf len %u result: %d", mb->pkt_len, cqe->res);
+		if (unlikely(cqe->res < 0)) {
+			++txq->tx_errors;
+		} else {
+			++tx_done;
+			tx_bytes += mb->pkt_len;
+		}
+
+		rte_pktmbuf_free(mb);
+	}
+	io_uring_cq_advance(&txq->io_ring, tx_done);
+
+	txq->tx_packets += tx_done;
+	txq->tx_bytes += tx_bytes;
+}
+
+static uint16_t
+eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct tx_queue *txq = queue;
+	uint16_t num_tx;
+
+	if (unlikely(nb_pkts == 0))
+		return 0;
+
+	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
+
+	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
+		eth_ioring_tx_cleanup(txq);
+
+	int fd = eth_queue_fd(txq->port_id, txq->queue_id);
+
+	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
+		struct rte_mbuf *mb = bufs[num_tx];
+
+		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+		if (sqe == NULL)
+			break;	/* submit ring is full */
+
+		io_uring_sqe_set_data(sqe, mb);
+
+		if (rte_mbuf_refcnt_read(mb) == 1 &&
+		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
+			void *base = rte_pktmbuf_mtod(mb, void *);
+			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+
+			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
+		} else {
+			PMD_LOG(ERR, "Can't do mbuf without space yet!");
+			++txq->tx_errors;
+			continue;
+		}
+	}
+	if (num_tx > 0)
+		io_uring_submit(&txq->io_ring);
+
+	return num_tx;
+}
+
 static const struct eth_dev_ops ops = {
 	.dev_start		= eth_dev_start,
 	.dev_stop		= eth_dev_stop,
@@ -339,9 +695,12 @@ static const struct eth_dev_ops ops = {
 	.promiscuous_disable	= eth_dev_promiscuous_disable,
 	.allmulticast_enable	= eth_dev_allmulticast_enable,
 	.allmulticast_disable	= eth_dev_allmulticast_disable,
+	.rx_queue_setup		= eth_rx_queue_setup,
+	.rx_queue_release	= eth_rx_queue_release,
+	.tx_queue_setup		= eth_tx_queue_setup,
+	.tx_queue_release	= eth_tx_queue_release,
 };
 
-
 static int
 ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 {
@@ -379,6 +738,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	}
 
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
+
+	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_burst = eth_ioring_tx;
+
 	return 0;
 
 error:
Signature · 1 line
2.45.2
#23 lore raw

PATCH v2 7/8 net/ioring: add VLAN support

In reply to Stephen Hemminger

Add support for VLAN insert and stripping.

---
 drivers/net/ioring/rte_eth_ioring.c | 45 +++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +43 -2
index 8dd717cb9d..a7f38e0d5c 100644
@@ -34,6 +34,8 @@
 #define IORING_NUM_BUFFERS	1024
 #define IORING_MAX_QUEUES	128
 
+#define IORING_TX_OFFLOAD	RTE_ETH_TX_OFFLOAD_VLAN_INSERT
+#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_VLAN_STRIP
 
 static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
 
@@ -70,6 +72,7 @@ static const char * const valid_arguments[] = {
 struct rx_queue {
 	struct rte_mempool *mb_pool;	/* rx buffer pool */
 	struct io_uring io_ring;	/* queue of posted read's */
+	uint64_t offloads;
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -81,6 +84,7 @@ struct rx_queue {
 
 struct tx_queue {
 	struct io_uring io_ring;
+	uint64_t offloads;
 
 	uint16_t port_id;
 	uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 			goto resubmit;
 		}
 
+		if (rxq->offloads & RTE_ETH_RX_OFFLOAD_VLAN_STRIP)
+			rte_vlan_strip(mb);
+
 		mb->pkt_len = len;
 		mb->data_len = len;
 		mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 static int
 eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
-		   unsigned int socket_id,
-		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,
 		   struct rte_mempool *mb_pool)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_de
 		return -1;
 	}
 
+	rxq->offloads = rx_conf->offloads;
 	rxq->mb_pool = mb_pool;
 	rxq->port_id = dev->data->port_id;
 	rxq->queue_id = queue_id;
@@ -583,6 +590,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	txq->port_id = dev->data->port_id;
 	txq->queue_id = queue_id;
+	txq->offloads = tx_conf->offloads;
 	txq->free_thresh = tx_conf->tx_free_thresh;
 	dev->data->tx_queues[queue_id] = txq;
 
@@ -637,6 +645,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq)
 	txq->tx_bytes += tx_bytes;
 }
 
+static uint16_t
+eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+	uint16_t nb_tx;
+	int error;
+
+	for (nb_tx = 0; nb_tx < nb_pkts; nb_tx++) {
+		struct rte_mbuf *m = tx_pkts[nb_tx];
+
+#ifdef RTE_LIBRTE_ETHDEV_DEBUG
+		error = rte_validate_tx_offload(m);
+		if (unlikely(error)) {
+			rte_errno = -error;
+			break;
+		}
+#endif
+		/* Do VLAN tag insertion */
+		if (unlikely(m->ol_flags & RTE_MBUF_F_TX_VLAN)) {
+			error = rte_vlan_insert(&m);
+			/* rte_vlan_insert() may change pointer */
+			tx_pkts[nb_tx] = m;
+
+			if (unlikely(error)) {
+				rte_errno = -error;
+				break;
+			}
+		}
+	}
+
+	return nb_tx;
+}
+
 static uint16_t
 eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 {
@@ -740,6 +780,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
 
 	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_prepare = eth_ioring_tx_prepare;
 	dev->tx_pkt_burst = eth_ioring_tx;
 
 	return 0;
Signature · 1 line
2.45.2
#24 lore raw

PATCH v2 8/8 net/ioring: implement statistics

In reply to Stephen Hemminger

Add support for basic statistics

---
 doc/guides/nics/features/ioring.ini |  2 +
 drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
index da47062adb..c9a4582d0e 100644
@@ -9,6 +9,8 @@ MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
 Multiprocess aware   = Y
+Basic stats          = Y
+Stats per queue      = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0

69 lines not on this page · raw message

Signature · 1 line
2.45.2
#25 lore raw

v3 cover ioring PMD device

In reply to Stephen Hemminger

This is a new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

v3 - add multi-segment support
review feedback

Stephen Hemminger (9):
net/ioring: introduce new driver
net/ioring: implement link state
net/ioring: implement control functions
net/ioring: implement management functions
net/ioring: implement secondary process support
net/ioring: implement receive and transmit
net/ioring: add VLAN support
net/ioring: implement statistics
net/ioring: support multi-segment Rx and Tx

 doc/guides/nics/features/ioring.ini |   16 +
 doc/guides/nics/index.rst           |    1 +
 doc/guides/nics/ioring.rst          |   66 ++
 drivers/net/ioring/meson.build      |   15 +
 drivers/net/ioring/rte_eth_ioring.c | 1129 +++++++++++++++++++++++++++
 drivers/net/meson.build             |    1 +
 6 files changed, 1228 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.47.2
#26 lore raw

PATCH v3 1/9 net/ioring: introduce new driver

Add basic driver initialization, documentation, and device creation and basic documentation.

---
 doc/guides/nics/features/ioring.ini |   9 +
 doc/guides/nics/index.rst           |   1 +
 doc/guides/nics/ioring.rst          |  66 +++++++
 drivers/net/ioring/meson.build      |  15 ++
 drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++
 drivers/net/meson.build             |   1 +
 6 files changed, 354 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
new file mode 100644
index 0000000000..c4c57caaa4
@@ -0,0 +1,9 @@
+;
+; Supported features of the 'ioring' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux		     = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index 10a2eca3b0..afb6bf289b 100644
@@ -41,6 +41,7 @@ Network Interface Controller Drivers
     igc
     intel_vf
     ionic
+    ioring
     ipn3ke
     ixgbe
     mana
doc/guides/nics/ioring.rst new file +66 -0

66 lines not on this page · raw message

drivers/net/ioring/meson.build new file +15 -0
new file mode 100644
index 0000000000..264554d069
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+dep = dependency('liburing', required:false)
+reason = 'missing dependency, "liburing"'
+build = dep.found()
+ext_deps += dep
+
+sources = files('rte_eth_ioring.c')
+require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0

262 lines not on this page · raw message

drivers/net/meson.build +1 -0
index 460eb69e5b..2e39136a5b 100644
@@ -34,6 +34,7 @@ drivers = [
         'intel/ixgbe',
         'intel/cpfl',  # depends on idpf, so must come after it
         'ionic',
+        'ioring',
         'mana',
         'memif',
         'mlx4',
Signature · 1 line
2.47.2
#27 lore raw

PATCH v3 2/9 net/ioring: implement link state

In reply to Stephen Hemminger

Add hooks to set kernel link up/down and report state.

---
 doc/guides/nics/features/ioring.ini |  1 +
 drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
index c4c57caaa4..d4bf70cb4f 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0

104 lines not on this page · raw message

Signature · 1 line
2.47.2
#28 lore raw

PATCH v3 3/9 net/ioring: implement control functions

In reply to Stephen Hemminger

These internal ops, just force changes to kernel visible net device.

---
 doc/guides/nics/features/ioring.ini |  3 ++
 drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
index d4bf70cb4f..199c7cd31c 100644
@@ -5,6 +5,9 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0

93 lines not on this page · raw message

Signature · 1 line
2.47.2
#29 lore raw

PATCH v3 4/9 net/ioring: implement management functions

In reply to Stephen Hemminger

Add start, stop, configure and info functions.

---
 drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
index fe3c72098c..b5b5ffdee3 100644
@@ -212,12 +212,16 @@ static int
 eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
+
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
 	struct ifreq ifr = { .ifr_mtu = mtu };
-	int ret;
 
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFMTU, &ifr);
+	int ret = ioctl(sock, SIOCSIFMTU, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFMTU) failed: %s", strerror(errno));
 		ret = -errno;
@@ -230,14 +234,17 @@ static int
 eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
-	struct ifreq ifr = { };
-	int ret;
 
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
+	struct ifreq ifr = { };
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 	ifr.ifr_hwaddr.sa_family = ARPHRD_ETHER;
 	memcpy(ifr.ifr_hwaddr.sa_data, addr, sizeof(*addr));
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFHWADDR, &ifr);
+	int ret = ioctl(sock, SIOCSIFHWADDR, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFHWADDR) failed: %s", strerror(errno));
 		ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 	return ret;
 }
 
+static int
+eth_dev_start(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
+	eth_dev_set_link_up(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_stop(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+	eth_dev_set_link_down(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_configure(struct rte_eth_dev *dev)
+{
+	/* rx/tx must be paired */
+	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int
+eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	dev_info->if_index = if_nametoindex(pmd->ifname);
+	dev_info->max_mac_addrs = 1;
+	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+
+	return 0;
+}
+
 static int
 eth_dev_close(struct rte_eth_dev *dev)
 {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 		close(pmd->keep_fd);
 		pmd->keep_fd = -1;
 	}
-
 	return 0;
 }
 
 static const struct eth_dev_ops ops = {
+	.dev_start		= eth_dev_start,
+	.dev_stop		= eth_dev_stop,
+	.dev_configure		= eth_dev_configure,
+	.dev_infos_get		= eth_dev_info,
 	.dev_close		= eth_dev_close,
 	.link_update		= eth_link_update,
 	.dev_set_link_up	= eth_dev_set_link_up,
Signature · 1 line
2.47.2
#30 lore raw

PATCH v3 5/9 net/ioring: implement secondary process support

In reply to Stephen Hemminger

Add support for communicating fd's from primary to secondary.

---
 doc/guides/nics/features/ioring.ini |   1 +
 drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++-
 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
index 199c7cd31c..da47062adb 100644
@@ -8,6 +8,7 @@ Link status          = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
+Multiprocess aware   = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2

181 lines not on this page · raw message

Signature · 1 line
2.47.2
#31 lore raw

PATCH v3 6/9 net/ioring: implement receive and transmit

In reply to Stephen Hemminger

Use io_uring to read and write from TAP device.

---
 drivers/net/ioring/rte_eth_ioring.c | 362 +++++++++++++++++++++++++++-
 1 file changed, 361 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +361 -1
index f01db960a7..4d064c2c22 100644
@@ -2,6 +2,7 @@
  * Copyright (c) Stephen Hemminger
  */
 
+#include <assert.h>
 #include <ctype.h>
 #include <errno.h>
 #include <fcntl.h>
@@ -9,8 +10,10 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <liburing.h>
 #include <sys/ioctl.h>
 #include <sys/socket.h>
+#include <sys/uio.h>
 #include <net/if.h>
 #include <linux/if.h>
 #include <linux/if_arp.h>
@@ -27,6 +30,12 @@
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
+#define IORING_DEFAULT_BURST	64
+#define IORING_NUM_BUFFERS	1024
+#define IORING_MAX_QUEUES	128
+
+static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
+
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -34,6 +43,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);
 #define RTE_LOGTYPE_IORING ioring_logtype
 #define PMD_LOG(level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__)
 
+#ifdef RTE_ETHDEV_DEBUG_RX
+#define PMD_RX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_RX_LOG(...) do { } while (0)
+#endif
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+#define PMD_TX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_TX_LOG(...) do { } while (0)
+#endif
+
 #define IORING_IFACE_ARG	"iface"
 #define IORING_PERSIST_ARG	"persist"
 
@@ -43,6 +66,30 @@ static const char * const valid_arguments[] = {
 	NULL
 };
 
+struct rx_queue {
+	struct rte_mempool *mb_pool;	/* rx buffer pool */
+	struct io_uring io_ring;	/* queue of posted read's */
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint64_t rx_packets;
+	uint64_t rx_bytes;
+	uint64_t rx_nombuf;
+	uint64_t rx_errors;
+};
+
+struct tx_queue {
+	struct io_uring io_ring;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t free_thresh;
+
+	uint64_t tx_packets;
+	uint64_t tx_bytes;
+	uint64_t tx_errors;
+};
+
 struct pmd_internals {
 	int keep_fd;			/* keep alive file descriptor */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
@@ -300,6 +347,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->if_index = if_nametoindex(pmd->ifname);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+	dev_info->max_rx_queues = IORING_MAX_QUEUES;
+	dev_info->max_tx_queues = IORING_MAX_QUEUES;
+	dev_info->min_rx_bufsize = 0;
+
+	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
+		.burst_size = IORING_DEFAULT_BURST,
+		.ring_size = IORING_NUM_BUFFERS,
+		.nb_queues = 1,
+	};
 
 	return 0;
 }
@@ -311,6 +367,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 
 	PMD_LOG(INFO, "Closing %s", pmd->ifname);
 
+	int *fds = dev->process_private;
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		if (fds[i] == -1)
+			continue;
+		close(fds[i]);
+		fds[i] = -1;
+	}
+
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
@@ -324,6 +388,295 @@ eth_dev_close(struct rte_eth_dev *dev)
 	return 0;
 }
 
+/* Setup another fd to TAP device for the queue */
+static int
+eth_queue_setup(struct rte_eth_dev *dev, const char *name, uint16_t queue_id)
+{
+	int *fds = dev->process_private;
+
+	if (fds[queue_id] != -1)
+		return 0;	/* already setup */
+
+	struct ifreq ifr = { };
+	int tap_fd = tap_open(name, &ifr, 0);
+	if (tap_fd < 0) {
+		PMD_LOG(ERR, "tap_open failed");
+		return -1;
+	}
+
+	PMD_LOG(DEBUG, "opened %d for queue %u", tap_fd, queue_id);
+	fds[queue_id] = tap_fd;
+	return 0;
+}
+
+static int
+eth_queue_fd(uint16_t port_id, uint16_t queue_id)
+{
+	struct rte_eth_dev *dev = &rte_eth_devices[port_id];
+	int *fds = dev->process_private;
+
+	return fds[queue_id];
+}
+
+/* setup an submit queue to read mbuf */
+static inline void
+eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
+{
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+
+	if (unlikely(sqe == NULL)) {
+		PMD_LOG(DEBUG, "io_uring no rx sqe");
+		rxq->rx_errors++;
+		rte_pktmbuf_free(mb);
+	} else {
+		void *base = rte_pktmbuf_mtod(mb, void *);
+		size_t len = mb->buf_len;
+
+		io_uring_prep_read(sqe, fd, base, len, 0);
+		io_uring_sqe_set_data(sqe, mb);
+	}
+}
+
+static uint16_t
+eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct rx_queue *rxq = queue;
+	struct io_uring_cqe *cqe;
+	unsigned int head, num_cqe = 0;
+	uint16_t num_rx = 0;
+	uint32_t num_bytes = 0;
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	io_uring_for_each_cqe(&rxq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+		ssize_t len = cqe->res;
+
+		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
+		num_cqe++;
+
+		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
+			if (len < 0)
+				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
+			else
+				PMD_LOG(ERR, "io_uring_read missing hdr");
+
+			rxq->rx_errors++;
+			goto resubmit;
+		}
+
+		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(nmb == 0)) {
+			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+			++rxq->rx_nombuf;
+			goto resubmit;
+		}
+
+		mb->pkt_len = len;
+		mb->data_len = len;
+		mb->port = rxq->port_id;
+		__rte_mbuf_sanity_check(mb, 1);
+
+		num_bytes += len;
+		bufs[num_rx++] = mb;
+
+		mb = nmb;
+resubmit:
+		eth_rx_submit(rxq, fd, mb);
+
+		if (num_rx == nb_pkts)
+			break;
+	}
+	io_uring_cq_advance(&rxq->io_ring, num_cqe);
+
+	rxq->rx_packets += num_rx;
+	rxq->rx_bytes += num_bytes;
+	return num_rx;
+}
+
+static int
+eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
+		   unsigned int socket_id,
+		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   struct rte_mempool *mb_pool)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	PMD_LOG(DEBUG, "setup port %u queue %u rx_descriptors %u",
+		dev->data->port_id, queue_id, nb_rx_desc);
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct rx_queue *rxq = rte_zmalloc_socket(NULL, sizeof(*rxq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (rxq == NULL) {
+		PMD_LOG(ERR, "rxq alloc failed");
+		return -1;
+	}
+
+	rxq->mb_pool = mb_pool;
+	rxq->port_id = dev->data->port_id;
+	rxq->queue_id = queue_id;
+	dev->data->rx_queues[queue_id] = rxq;
+
+	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	struct rte_mbuf **mbufs = alloca(nb_rx_desc * sizeof(struct rte_mbuf *));
+	if (mbufs == NULL) {
+		PMD_LOG(ERR, "alloca for %u failed", nb_rx_desc);
+		return -1;
+	}
+
+	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
+		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
+		return -1;
+	}
+
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+	for (uint16_t i = 0; i < nb_rx_desc; i++)
+		eth_rx_submit(rxq, fd, mbufs[i]);
+
+	io_uring_submit(&rxq->io_ring);
+	return 0;
+}
+
+static void
+eth_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct rx_queue *rxq = dev->data->rx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&rxq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&rxq->io_ring);
+}
+
+static int
+eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+		   uint16_t nb_tx_desc, unsigned int socket_id,
+		   const struct rte_eth_txconf *tx_conf)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct tx_queue *txq = rte_zmalloc_socket(NULL, sizeof(*txq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (txq == NULL) {
+		PMD_LOG(ERR, "txq alloc failed");
+		return -1;
+	}
+
+	txq->port_id = dev->data->port_id;
+	txq->queue_id = queue_id;
+	txq->free_thresh = tx_conf->tx_free_thresh;
+	dev->data->tx_queues[queue_id] = txq;
+
+	if (io_uring_queue_init(nb_tx_desc, &txq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	return 0;
+}
+
+static void
+eth_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct tx_queue *txq = dev->data->tx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&txq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&txq->io_ring);
+}
+
+static void
+eth_ioring_tx_cleanup(struct tx_queue *txq)
+{
+	struct io_uring_cqe *cqe;
+	unsigned int head;
+	unsigned int tx_done = 0;
+	uint64_t tx_bytes = 0;
+
+	io_uring_for_each_cqe(&txq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+
+		PMD_TX_LOG(DEBUG, " mbuf len %u result: %d", mb->pkt_len, cqe->res);
+		if (unlikely(cqe->res < 0)) {
+			++txq->tx_errors;
+		} else {
+			++tx_done;
+			tx_bytes += mb->pkt_len;
+		}
+
+		rte_pktmbuf_free(mb);
+	}
+	io_uring_cq_advance(&txq->io_ring, tx_done);
+
+	txq->tx_packets += tx_done;
+	txq->tx_bytes += tx_bytes;
+}
+
+static uint16_t
+eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct tx_queue *txq = queue;
+	uint16_t num_tx;
+
+	if (unlikely(nb_pkts == 0))
+		return 0;
+
+	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
+
+	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
+		eth_ioring_tx_cleanup(txq);
+
+	int fd = eth_queue_fd(txq->port_id, txq->queue_id);
+
+	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
+		struct rte_mbuf *mb = bufs[num_tx];
+
+		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+		if (sqe == NULL)
+			break;	/* submit ring is full */
+
+		io_uring_sqe_set_data(sqe, mb);
+
+		if (rte_mbuf_refcnt_read(mb) == 1 &&
+		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
+			void *base = rte_pktmbuf_mtod(mb, void *);
+			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+
+			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
+		} else {
+			PMD_LOG(ERR, "Can't do mbuf without space yet!");
+			++txq->tx_errors;
+			continue;
+		}
+	}
+	if (num_tx > 0)
+		io_uring_submit(&txq->io_ring);
+
+	return num_tx;
+}
+
 static const struct eth_dev_ops ops = {
 	.dev_start		= eth_dev_start,
 	.dev_stop		= eth_dev_stop,
@@ -339,9 +692,12 @@ static const struct eth_dev_ops ops = {
 	.promiscuous_disable	= eth_dev_promiscuous_disable,
 	.allmulticast_enable	= eth_dev_allmulticast_enable,
 	.allmulticast_disable	= eth_dev_allmulticast_disable,
+	.rx_queue_setup		= eth_rx_queue_setup,
+	.rx_queue_release	= eth_rx_queue_release,
+	.tx_queue_setup		= eth_tx_queue_setup,
+	.tx_queue_release	= eth_tx_queue_release,
 };
 
-
 static int
 ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 {
@@ -379,6 +735,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	}
 
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
+
+	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_burst = eth_ioring_tx;
+
 	return 0;
 
 error:
Signature · 1 line
2.47.2
#32 lore raw

PATCH v3 7/9 net/ioring: add VLAN support

In reply to Stephen Hemminger

Add support for VLAN insert and stripping.

---
 drivers/net/ioring/rte_eth_ioring.c | 47 +++++++++++++++++++++++++++--
 1 file changed, 44 insertions(+), 3 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +44 -3
index 4d064c2c22..d473c3fcbb 100644
@@ -33,9 +33,11 @@
 #define IORING_DEFAULT_BURST	64
 #define IORING_NUM_BUFFERS	1024
 #define IORING_MAX_QUEUES	128
-
 static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
 
+#define IORING_TX_OFFLOAD	RTE_ETH_TX_OFFLOAD_VLAN_INSERT
+#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_VLAN_STRIP
+
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -69,6 +71,7 @@ static const char * const valid_arguments[] = {
 struct rx_queue {
 	struct rte_mempool *mb_pool;	/* rx buffer pool */
 	struct io_uring io_ring;	/* queue of posted read's */
+	uint64_t offloads;
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -80,6 +83,7 @@ struct rx_queue {
 
 struct tx_queue {
 	struct io_uring io_ring;
+	uint64_t offloads;
 
 	uint16_t port_id;
 	uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 			goto resubmit;
 		}
 
+		if (rxq->offloads & RTE_ETH_RX_OFFLOAD_VLAN_STRIP)
+			rte_vlan_strip(mb);
+
 		mb->pkt_len = len;
 		mb->data_len = len;
 		mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 static int
 eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
-		   unsigned int socket_id,
-		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,
 		   struct rte_mempool *mb_pool)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_de
 		return -1;
 	}
 
+	rxq->offloads = rx_conf->offloads;
 	rxq->mb_pool = mb_pool;
 	rxq->port_id = dev->data->port_id;
 	rxq->queue_id = queue_id;
@@ -580,6 +587,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	txq->port_id = dev->data->port_id;
 	txq->queue_id = queue_id;
+	txq->offloads = tx_conf->offloads;
 	txq->free_thresh = tx_conf->tx_free_thresh;
 	dev->data->tx_queues[queue_id] = txq;
 
@@ -634,6 +642,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq)
 	txq->tx_bytes += tx_bytes;
 }
 
+static uint16_t
+eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+	uint16_t nb_tx;
+	int error;
+
+	for (nb_tx = 0; nb_tx < nb_pkts; nb_tx++) {
+		struct rte_mbuf *m = tx_pkts[nb_tx];
+
+#ifdef RTE_LIBRTE_ETHDEV_DEBUG
+		error = rte_validate_tx_offload(m);
+		if (unlikely(error)) {
+			rte_errno = -error;
+			break;
+		}
+#endif
+		/* Do VLAN tag insertion */
+		if (unlikely(m->ol_flags & RTE_MBUF_F_TX_VLAN)) {
+			error = rte_vlan_insert(&m);
+			/* rte_vlan_insert() may change pointer */
+			tx_pkts[nb_tx] = m;
+
+			if (unlikely(error)) {
+				rte_errno = -error;
+				break;
+			}
+		}
+	}
+
+	return nb_tx;
+}
+
 static uint16_t
 eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 {
@@ -737,6 +777,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
 
 	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_prepare = eth_ioring_tx_prepare;
 	dev->tx_pkt_burst = eth_ioring_tx;
 
 	return 0;
Signature · 1 line
2.47.2
#33 lore raw

PATCH v3 8/9 net/ioring: implement statistics

In reply to Stephen Hemminger

Add support for basic statistics

---
 doc/guides/nics/features/ioring.ini |  2 +
 drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
index da47062adb..c9a4582d0e 100644
@@ -9,6 +9,8 @@ MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
 Multiprocess aware   = Y
+Basic stats          = Y
+Stats per queue      = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0

69 lines not on this page · raw message

Signature · 1 line
2.47.2
#34 lore raw

PATCH v3 9/9 net/ioring: support multi-segment Rx and Tx

In reply to Stephen Hemminger

Use readv/writev to handle multi-segment transmit and receive. Account for virtio header that will be used for offload (later).

---
 drivers/net/ioring/rte_eth_ioring.c | 140 ++++++++++++++++++++--------
 1 file changed, 102 insertions(+), 38 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +102 -38
index 83446dc660..a803a9820b 100644
@@ -18,6 +18,7 @@
 #include <linux/if.h>
 #include <linux/if_arp.h>
 #include <linux/if_tun.h>
+#include <linux/virtio_net.h>
 
 #include <bus_vdev_driver.h>
 #include <ethdev_driver.h>
@@ -35,8 +36,11 @@
 #define IORING_MAX_QUEUES	128
 static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
 
-#define IORING_TX_OFFLOAD	RTE_ETH_TX_OFFLOAD_VLAN_INSERT
-#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_VLAN_STRIP
+#define IORING_TX_OFFLOAD	(RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \
+				 RTE_ETH_TX_OFFLOAD_MULTI_SEGS)
+
+#define IORING_RX_OFFLOAD	(RTE_ETH_RX_OFFLOAD_VLAN_STRIP | \
+				 RTE_ETH_RX_OFFLOAD_SCATTER)
 
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
@@ -166,7 +170,7 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist)
 		goto error;
 	}
 
-	int flags = IFF_TAP | IFF_MULTI_QUEUE | IFF_NO_PI;
+	int flags = IFF_TAP | IFF_MULTI_QUEUE | IFF_NO_PI | IFF_VNET_HDR;
 	if ((features & flags) != flags) {
 		PMD_LOG(ERR, "TUN features %#x missing support for %#x",
 			features, features & flags);
@@ -354,6 +358,8 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->max_rx_queues = IORING_MAX_QUEUES;
 	dev_info->max_tx_queues = IORING_MAX_QUEUES;
 	dev_info->min_rx_bufsize = 0;
+	dev_info->tx_queue_offload_capa = IORING_TX_OFFLOAD;
+	dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa;
 
 	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
 		.burst_size = IORING_DEFAULT_BURST,
@@ -487,13 +493,44 @@ eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
 		PMD_LOG(DEBUG, "io_uring no rx sqe");
 		rxq->rx_errors++;
 		rte_pktmbuf_free(mb);
-	} else {
-		void *base = rte_pktmbuf_mtod(mb, void *);
-		size_t len = mb->buf_len;
+		return;
+	}
 
-		io_uring_prep_read(sqe, fd, base, len, 0);
-		io_uring_sqe_set_data(sqe, mb);
+	RTE_VERIFY(mb->nb_segs < IOV_MAX);
+	struct iovec iovs[IOV_MAX];
+
+	for (uint16_t i = 0; i < mb->nb_segs; i++) {
+		iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *);
+		iovs[i].iov_len = rte_pktmbuf_tailroom(mb);
+		mb = mb->next;
 	}
+	io_uring_sqe_set_data(sqe, mb);
+	io_uring_prep_readv(sqe, fd, iovs, mb->nb_segs, 0);
+}
+
+static struct rte_mbuf *
+eth_ioring_rx_alloc(struct rx_queue *rxq)
+{
+	const struct rte_eth_dev *dev = &rte_eth_devices[rxq->port_id];
+	int buf_size = dev->data->mtu + sizeof(struct virtio_net_hdr);
+	struct rte_mbuf *m = NULL;
+	struct rte_mbuf **tail = &m;
+
+	do {
+		struct rte_mbuf *seg = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(seg == NULL)) {
+			rte_pktmbuf_free(m);
+			return NULL;
+		}
+		*tail = seg;
+		tail = &seg->next;
+		if (seg != m)
+			++m->nb_segs;
+
+		buf_size -= rte_pktmbuf_tailroom(seg);
+	} while (buf_size > 0);
+
+	return m;
 }
 
 static uint16_t
@@ -513,7 +550,8 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
 		num_cqe++;
 
-		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
+		struct virtio_net_hdr *hdr;
+		if (unlikely(len < (ssize_t)(sizeof(*hdr) + RTE_ETHER_HDR_LEN))) {
 			if (len < 0)
 				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
 			else
@@ -523,19 +561,31 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 			goto resubmit;
 		}
 
-		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
-		if (unlikely(nmb == 0)) {
-			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+		hdr = rte_pktmbuf_mtod(mb, struct virtio_net_hdr *);
+
+		struct rte_mbuf *nmb = eth_ioring_rx_alloc(rxq);
+		if (!nmb) {
 			++rxq->rx_nombuf;
 			goto resubmit;
 		}
 
-		if (rxq->offloads & RTE_ETH_RX_OFFLOAD_VLAN_STRIP)
-			rte_vlan_strip(mb);
+		len -= sizeof(*hdr);
+		mb->data_off += sizeof(*hdr);
 
 		mb->pkt_len = len;
-		mb->data_len = len;
 		mb->port = rxq->port_id;
+		struct rte_mbuf *seg = mb;
+		for (;;) {
+			seg->data_len = RTE_MIN(len, mb->buf_len);
+			seg = seg->next;
+			len -= seg->data_len;
+		} while (len > 0 && seg != NULL);
+
+		RTE_VERIFY(!(seg == NULL && len > 0));
+
+		if (rxq->offloads & RTE_ETH_RX_OFFLOAD_VLAN_STRIP)
+			rte_vlan_strip(mb);
+
 		__rte_mbuf_sanity_check(mb, 1);
 
 		num_bytes += len;
@@ -555,6 +605,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 	return num_rx;
 }
 
+
 static int
 eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
 		   unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,
@@ -587,20 +638,17 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_de
 		return -1;
 	}
 
-	struct rte_mbuf **mbufs = alloca(nb_rx_desc * sizeof(struct rte_mbuf *));
-	if (mbufs == NULL) {
-		PMD_LOG(ERR, "alloca for %u failed", nb_rx_desc);
-		return -1;
-	}
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
 
-	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
-		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
-		return -1;
-	}
+	for (uint16_t i = 0; i < nb_rx_desc; i++) {
+		struct rte_mbuf *mb = eth_ioring_rx_alloc(rxq);
+		if (mb == NULL) {
+			PMD_LOG(ERR, "Rx mbuf alloc buf failed");
+			return -1;
+		}
 
-	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
-	for (uint16_t i = 0; i < nb_rx_desc; i++)
-		eth_rx_submit(rxq, fd, mbufs[i]);
+		eth_rx_submit(rxq, fd, mb);
+	}
 
 	io_uring_submit(&rxq->io_ring);
 	return 0;
@@ -740,31 +788,47 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
 
-	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
+	if (likely(io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh))
 		eth_ioring_tx_cleanup(txq);
 
 	int fd = eth_queue_fd(txq->port_id, txq->queue_id);
 
 	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
 		struct rte_mbuf *mb = bufs[num_tx];
+		struct virtio_net_hdr *hdr;
 
 		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
 		if (sqe == NULL)
 			break;	/* submit ring is full */
 
-		io_uring_sqe_set_data(sqe, mb);
+		if (rte_mbuf_refcnt_read(mb) == 1 && RTE_MBUF_DIRECT(mb) &&
+		    rte_pktmbuf_headroom(mb) >= sizeof(*hdr)) {
+			hdr = (struct virtio_net_hdr *)rte_pktmbuf_prepend(mb, sizeof(*hdr));
+		} else {
+			struct rte_mbuf *mh = rte_pktmbuf_alloc(mb->pool);
+			if (unlikely(mh == NULL))
+				break;
+			hdr = (struct virtio_net_hdr *)rte_pktmbuf_append(mh, sizeof(*hdr));
 
-		if (rte_mbuf_refcnt_read(mb) == 1 &&
-		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
-			void *base = rte_pktmbuf_mtod(mb, void *);
-			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+			mh->next = mb;
+			mh->nb_segs = mb->nb_segs + 1;
+			mh->pkt_len += mb->pkt_len;
+			mh->ol_flags = mb->ol_flags & RTE_MBUF_F_TX_OFFLOAD_MASK;
+			mb = mh;
+		}
+		memset(hdr, 0, sizeof(*hdr));
 
-			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
-		} else {
-			PMD_LOG(ERR, "Can't do mbuf without space yet!");
-			++txq->tx_errors;
-			continue;
+		io_uring_sqe_set_data(sqe, mb);
+
+		struct iovec iovs[RTE_MBUF_MAX_NB_SEGS + 1];
+		unsigned int niov = mb->nb_segs;
+		for (unsigned int i = 0; i < niov; i++) {
+			iovs[i].iov_base = rte_pktmbuf_mtod(mb, char *);
+			iovs[i].iov_len = mb->data_len;
+			mb = mb->next;
 		}
+
+		io_uring_prep_writev(sqe, fd, iovs, niov, 0);
 	}
 	if (num_tx > 0)
 		io_uring_submit(&txq->io_ring);
Signature · 1 line
2.47.2
#35 lore raw

v4 cover new ioring PMD

In reply to Stephen Hemminger

This is a new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.

This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.

This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.

v4 - more testing and offload support

Stephen Hemminger (10):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement secondary process support net/ioring: implement receive and transmit net/ioring: implement statistics net/ioring: support multi-segment Rx and Tx net/ioring: support Tx checksum and segment offload net/ioring: add support for Rx offload

 MAINTAINERS                         |    6 +
 doc/guides/nics/features/ioring.ini |   18 +
 doc/guides/nics/index.rst           |    1 +
 doc/guides/nics/ioring.rst          |   60 ++
 drivers/net/ioring/meson.build      |   15 +
 drivers/net/ioring/rte_eth_ioring.c | 1288 +++++++++++++++++++++++++++
 drivers/net/meson.build             |    1 +
 7 files changed, 1389 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.47.2
#36 lore raw

PATCH v4 1/10 net/ioring: introduce new driver

Add basic driver initialization, documentation, and device creation and basic documentation.

---
 MAINTAINERS                         |   6 +
 doc/guides/nics/features/ioring.ini |   9 +
 doc/guides/nics/index.rst           |   1 +
 doc/guides/nics/ioring.rst          |  66 +++++++
 drivers/net/ioring/meson.build      |  15 ++
 drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++
 drivers/net/meson.build             |   1 +
 7 files changed, 360 insertions(+)
 create mode 100644 doc/guides/nics/features/ioring.ini
 create mode 100644 doc/guides/nics/ioring.rst
 create mode 100644 drivers/net/ioring/meson.build
 create mode 100644 drivers/net/ioring/rte_eth_ioring.c
MAINTAINERS +6 -0
index 82f6e2f917..78bf70c5a0 100644
@@ -853,6 +853,12 @@ F: drivers/net/intel/ipn3ke/
 F: doc/guides/nics/ipn3ke.rst
 F: doc/guides/nics/features/ipn3ke.ini
 
+Ioring - EXPERIMENTAL
+M: Stephen Hemminger <[email protected]>
+F: drivers/net/ioring/
+F: doc/guides/nics/ioring.rst
+F: doc/guides/nics/features/ioring.ini
+
 Marvell cnxk
 M: Nithin Dabilpuram <[email protected]>
 M: Kiran Kumar K <[email protected]>
doc/guides/nics/features/ioring.ini new file +9 -0
new file mode 100644
index 0000000000..c4c57caaa4
@@ -0,0 +1,9 @@
+;
+; Supported features of the 'ioring' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux		     = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index 10a2eca3b0..afb6bf289b 100644
@@ -41,6 +41,7 @@ Network Interface Controller Drivers
     igc
     intel_vf
     ionic
+    ioring
     ipn3ke
     ixgbe
     mana
doc/guides/nics/ioring.rst new file +66 -0

66 lines not on this page · raw message

drivers/net/ioring/meson.build new file +15 -0
new file mode 100644
index 0000000000..264554d069
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+dep = dependency('liburing', required:false)
+reason = 'missing dependency, "liburing"'
+build = dep.found()
+ext_deps += dep
+
+sources = files('rte_eth_ioring.c')
+require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0

262 lines not on this page · raw message

drivers/net/meson.build +1 -0
index 460eb69e5b..2e39136a5b 100644
@@ -34,6 +34,7 @@ drivers = [
         'intel/ixgbe',
         'intel/cpfl',  # depends on idpf, so must come after it
         'ionic',
+        'ioring',
         'mana',
         'memif',
         'mlx4',
Signature · 1 line
2.47.2
#37 lore raw

PATCH v4 2/10 net/ioring: implement link state

In reply to Stephen Hemminger

Add hooks to set kernel link up/down and report state.

---
 doc/guides/nics/features/ioring.ini |  1 +
 drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
index c4c57caaa4..d4bf70cb4f 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0

104 lines not on this page · raw message

Signature · 1 line
2.47.2
#38 lore raw

PATCH v4 3/10 net/ioring: implement control functions

In reply to Stephen Hemminger

These internal ops, just force changes to kernel visible net device.

---
 doc/guides/nics/features/ioring.ini |  3 ++
 drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
index d4bf70cb4f..199c7cd31c 100644
@@ -5,6 +5,9 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0

93 lines not on this page · raw message

Signature · 1 line
2.47.2
#39 lore raw

PATCH v4 4/10 net/ioring: implement management functions

In reply to Stephen Hemminger

Add start, stop, configure and info functions.

---
 drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
index fe3c72098c..b5b5ffdee3 100644
@@ -212,12 +212,16 @@ static int
 eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
+
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
 	struct ifreq ifr = { .ifr_mtu = mtu };
-	int ret;
 
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFMTU, &ifr);
+	int ret = ioctl(sock, SIOCSIFMTU, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFMTU) failed: %s", strerror(errno));
 		ret = -errno;
@@ -230,14 +234,17 @@ static int
 eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 {
 	struct pmd_internals *pmd = dev->data->dev_private;
-	struct ifreq ifr = { };
-	int ret;
 
+	int sock = socket(AF_INET, SOCK_DGRAM, 0);
+	if (sock < 0)
+		return -errno;
+
+	struct ifreq ifr = { };
 	strlcpy(ifr.ifr_name, pmd->ifname, IFNAMSIZ);
 	ifr.ifr_hwaddr.sa_family = ARPHRD_ETHER;
 	memcpy(ifr.ifr_hwaddr.sa_data, addr, sizeof(*addr));
 
-	ret = ioctl(pmd->ctl_sock, SIOCSIFHWADDR, &ifr);
+	int ret = ioctl(sock, SIOCSIFHWADDR, &ifr);
 	if (ret < 0) {
 		PMD_LOG(ERR, "ioctl(SIOCSIFHWADDR) failed: %s", strerror(errno));
 		ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 	return ret;
 }
 
+static int
+eth_dev_start(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
+	eth_dev_set_link_up(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_stop(struct rte_eth_dev *dev)
+{
+	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+	eth_dev_set_link_down(dev);
+
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+		dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED;
+	}
+
+	return 0;
+}
+
+static int
+eth_dev_configure(struct rte_eth_dev *dev)
+{
+	/* rx/tx must be paired */
+	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int
+eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	dev_info->if_index = if_nametoindex(pmd->ifname);
+	dev_info->max_mac_addrs = 1;
+	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+
+	return 0;
+}
+
 static int
 eth_dev_close(struct rte_eth_dev *dev)
 {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 		close(pmd->keep_fd);
 		pmd->keep_fd = -1;
 	}
-
 	return 0;
 }
 
 static const struct eth_dev_ops ops = {
+	.dev_start		= eth_dev_start,
+	.dev_stop		= eth_dev_stop,
+	.dev_configure		= eth_dev_configure,
+	.dev_infos_get		= eth_dev_info,
 	.dev_close		= eth_dev_close,
 	.link_update		= eth_link_update,
 	.dev_set_link_up	= eth_dev_set_link_up,
Signature · 1 line
2.47.2
#40 lore raw

PATCH v4 5/10 net/ioring: implement secondary process support

In reply to Stephen Hemminger

Add support for communicating fd's from primary to secondary.

---
 doc/guides/nics/features/ioring.ini |   1 +
 doc/guides/nics/ioring.rst          |   6 --
 drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++-
 3 files changed, 135 insertions(+), 8 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
index 199c7cd31c..da47062adb 100644
@@ -8,6 +8,7 @@ Link status          = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
+Multiprocess aware   = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
doc/guides/nics/ioring.rst +0 -6
index 7d37a6bb37..69102a5b38 100644
@@ -46,12 +46,6 @@ But this behavior can be overridden by the use of the persist flag, example::
   --vdev=net_ioring0,iface=io0,persist ...
 
 
-Multi-process sharing
----------------------
-
-The IORING device does not support secondary process (yet).
-
-
 Limitations
 -----------
 
drivers/net/ioring/rte_eth_ioring.c +134 -2

181 lines not on this page · raw message

Signature · 1 line
2.47.2
#41 lore raw

PATCH v4 6/10 net/ioring: implement receive and transmit

In reply to Stephen Hemminger

Use io_uring to read and write from TAP device.

---
 drivers/net/ioring/rte_eth_ioring.c | 366 +++++++++++++++++++++++++++-
 1 file changed, 365 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +365 -1
index f01db960a7..2f049e4c4f 100644
@@ -2,6 +2,7 @@
  * Copyright (c) Stephen Hemminger
  */
 
+#include <assert.h>
 #include <ctype.h>
 #include <errno.h>
 #include <fcntl.h>
@@ -9,8 +10,10 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <liburing.h>
 #include <sys/ioctl.h>
 #include <sys/socket.h>
+#include <sys/uio.h>
 #include <net/if.h>
 #include <linux/if.h>
 #include <linux/if_arp.h>
@@ -27,6 +30,12 @@
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
+#define IORING_DEFAULT_BURST	64
+#define IORING_NUM_BUFFERS	1024
+#define IORING_MAX_QUEUES	128
+
+static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
+
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -34,6 +43,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);
 #define RTE_LOGTYPE_IORING ioring_logtype
 #define PMD_LOG(level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__)
 
+#ifdef RTE_ETHDEV_DEBUG_RX
+#define PMD_RX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_RX_LOG(...) do { } while (0)
+#endif
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+#define PMD_TX_LOG(level, ...) \
+	RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__)
+#else
+#define PMD_TX_LOG(...) do { } while (0)
+#endif
+
 #define IORING_IFACE_ARG	"iface"
 #define IORING_PERSIST_ARG	"persist"
 
@@ -43,6 +66,30 @@ static const char * const valid_arguments[] = {
 	NULL
 };
 
+struct rx_queue {
+	struct rte_mempool *mb_pool;	/* rx buffer pool */
+	struct io_uring io_ring;	/* queue of posted read's */
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint64_t rx_packets;
+	uint64_t rx_bytes;
+	uint64_t rx_nombuf;
+	uint64_t rx_errors;
+};
+
+struct tx_queue {
+	struct io_uring io_ring;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t free_thresh;
+
+	uint64_t tx_packets;
+	uint64_t tx_bytes;
+	uint64_t tx_errors;
+};
+
 struct pmd_internals {
 	int keep_fd;			/* keep alive file descriptor */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
@@ -300,6 +347,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->if_index = if_nametoindex(pmd->ifname);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_rx_pktlen = RTE_ETHER_MAX_LEN;
+	dev_info->max_rx_queues = IORING_MAX_QUEUES;
+	dev_info->max_tx_queues = IORING_MAX_QUEUES;
+	dev_info->min_rx_bufsize = 0;
+
+	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
+		.burst_size = IORING_DEFAULT_BURST,
+		.ring_size = IORING_NUM_BUFFERS,
+		.nb_queues = 1,
+	};
 
 	return 0;
 }
@@ -311,6 +367,14 @@ eth_dev_close(struct rte_eth_dev *dev)
 
 	PMD_LOG(INFO, "Closing %s", pmd->ifname);
 
+	int *fds = dev->process_private;
+	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+		if (fds[i] == -1)
+			continue;
+		close(fds[i]);
+		fds[i] = -1;
+	}
+
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
@@ -324,6 +388,299 @@ eth_dev_close(struct rte_eth_dev *dev)
 	return 0;
 }
 
+/* Setup another fd to TAP device for the queue */
+static int
+eth_queue_setup(struct rte_eth_dev *dev, const char *name, uint16_t queue_id)
+{
+	int *fds = dev->process_private;
+
+	if (fds[queue_id] != -1)
+		return 0;	/* already setup */
+
+	struct ifreq ifr = { };
+	int tap_fd = tap_open(name, &ifr, 0);
+	if (tap_fd < 0) {
+		PMD_LOG(ERR, "tap_open failed");
+		return -1;
+	}
+
+	PMD_LOG(DEBUG, "opened %d for queue %u", tap_fd, queue_id);
+	fds[queue_id] = tap_fd;
+	return 0;
+}
+
+static int
+eth_queue_fd(uint16_t port_id, uint16_t queue_id)
+{
+	struct rte_eth_dev *dev = &rte_eth_devices[port_id];
+	int *fds = dev->process_private;
+
+	return fds[queue_id];
+}
+
+/* setup an submit queue to read mbuf */
+static inline void
+eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
+{
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+
+	if (unlikely(sqe == NULL)) {
+		PMD_LOG(DEBUG, "io_uring no rx sqe");
+		rxq->rx_errors++;
+		rte_pktmbuf_free(mb);
+		return;
+	}
+	io_uring_sqe_set_data(sqe, mb);
+
+	void *buf = rte_pktmbuf_mtod_offset(mb, void *, 0);
+	unsigned int nbytes = rte_pktmbuf_tailroom(mb);
+
+	io_uring_prep_read(sqe, fd, buf, nbytes, 0);
+}
+
+static uint16_t
+eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct rx_queue *rxq = queue;
+	struct io_uring_cqe *cqe;
+	unsigned int head, num_cqe = 0;
+	uint16_t num_rx = 0;
+	uint32_t num_bytes = 0;
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+
+	io_uring_for_each_cqe(&rxq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+		ssize_t len = cqe->res;
+
+		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
+		num_cqe++;
+
+		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
+			if (len < 0)
+				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
+			else
+				PMD_LOG(ERR, "io_uring_read missing hdr");
+
+			rxq->rx_errors++;
+			goto resubmit;
+		}
+
+		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(nmb == 0)) {
+			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+			++rxq->rx_nombuf;
+			goto resubmit;
+		}
+
+		mb->pkt_len = len;
+		mb->data_len = len;
+		mb->port = rxq->port_id;
+		__rte_mbuf_sanity_check(mb, 1);
+
+		num_bytes += len;
+		bufs[num_rx++] = mb;
+
+		mb = nmb;
+resubmit:
+		eth_rx_submit(rxq, fd, mb);
+
+		if (num_rx == nb_pkts)
+			break;
+	}
+	io_uring_cq_advance(&rxq->io_ring, num_cqe);
+
+	rxq->rx_packets += num_rx;
+	rxq->rx_bytes += num_bytes;
+	return num_rx;
+}
+
+static int
+eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc,
+		   unsigned int socket_id,
+		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   struct rte_mempool *mb_pool)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	PMD_LOG(DEBUG, "setup port %u queue %u rx_descriptors %u",
+		dev->data->port_id, queue_id, nb_rx_desc);
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct rx_queue *rxq = rte_zmalloc_socket(NULL, sizeof(*rxq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (rxq == NULL) {
+		PMD_LOG(ERR, "rxq alloc failed");
+		return -1;
+	}
+
+	rxq->mb_pool = mb_pool;
+	rxq->port_id = dev->data->port_id;
+	rxq->queue_id = queue_id;
+	dev->data->rx_queues[queue_id] = rxq;
+
+	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	struct rte_mbuf **mbufs = alloca(nb_rx_desc * sizeof(struct rte_mbuf *));
+	if (mbufs == NULL) {
+		PMD_LOG(ERR, "alloca for %u failed", nb_rx_desc);
+		return -1;
+	}
+
+	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
+		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
+		return -1;
+	}
+
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
+	for (uint16_t i = 0; i < nb_rx_desc; i++)
+		eth_rx_submit(rxq, fd, mbufs[i]);
+
+	io_uring_submit(&rxq->io_ring);
+	return 0;
+}
+
+static void
+eth_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct rx_queue *rxq = dev->data->rx_queues[queue_id];
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&rxq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&rxq->io_ring);
+}
+
+static int
+eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+		   uint16_t nb_tx_desc, unsigned int socket_id,
+		   const struct rte_eth_txconf *tx_conf)
+{
+	struct pmd_internals *pmd = dev->data->dev_private;
+
+	/* open shared tap fd maybe already setup */
+	if (eth_queue_setup(dev, pmd->ifname, queue_id) < 0)
+		return -1;
+
+	struct tx_queue *txq = rte_zmalloc_socket(NULL, sizeof(*txq),
+						  RTE_CACHE_LINE_SIZE, socket_id);
+	if (txq == NULL) {
+		PMD_LOG(ERR, "txq alloc failed");
+		return -1;
+	}
+
+	txq->port_id = dev->data->port_id;
+	txq->queue_id = queue_id;
+	txq->free_thresh = tx_conf->tx_free_thresh;
+	dev->data->tx_queues[queue_id] = txq;
+
+	if (io_uring_queue_init(nb_tx_desc, &txq->io_ring, 0) != 0) {
+		PMD_LOG(ERR, "io_uring_queue_init failed: %s", strerror(errno));
+		return -1;
+	}
+
+	return 0;
+}
+
+static void
+eth_ioring_tx_cleanup(struct tx_queue *txq)
+{
+	struct io_uring_cqe *cqe;
+	unsigned int head;
+	unsigned int tx_done = 0;
+	uint64_t tx_bytes = 0;
+
+	io_uring_for_each_cqe(&txq->io_ring, head, cqe) {
+		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
+
+		PMD_TX_LOG(DEBUG, " mbuf len %u result: %d", mb->pkt_len, cqe->res);
+		if (unlikely(cqe->res < 0)) {
+			++txq->tx_errors;
+		} else {
+			++tx_done;
+			tx_bytes += mb->pkt_len;
+		}
+
+		rte_pktmbuf_free(mb);
+	}
+	io_uring_cq_advance(&txq->io_ring, tx_done);
+
+	txq->tx_packets += tx_done;
+	txq->tx_bytes += tx_bytes;
+}
+
+static void
+eth_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
+{
+	struct tx_queue *txq = dev->data->tx_queues[queue_id];
+
+	eth_ioring_tx_cleanup(txq);
+
+	struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+	if (sqe == NULL) {
+		PMD_LOG(ERR, "io_uring_get_sqe failed: %s", strerror(errno));
+	} else {
+		io_uring_prep_cancel(sqe, NULL, IORING_ASYNC_CANCEL_ANY);
+		io_uring_submit_and_wait(&txq->io_ring, 1);
+	}
+
+	io_uring_queue_exit(&txq->io_ring);
+}
+
+static uint16_t
+eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
+	struct tx_queue *txq = queue;
+	uint16_t num_tx;
+
+	if (unlikely(nb_pkts == 0))
+		return 0;
+
+	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
+
+	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
+		eth_ioring_tx_cleanup(txq);
+
+	int fd = eth_queue_fd(txq->port_id, txq->queue_id);
+
+	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
+		struct rte_mbuf *mb = bufs[num_tx];
+
+		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
+		if (sqe == NULL)
+			break;	/* submit ring is full */
+
+		io_uring_sqe_set_data(sqe, mb);
+
+		if (rte_mbuf_refcnt_read(mb) == 1 &&
+		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
+			void *base = rte_pktmbuf_mtod(mb, void *);
+			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+
+			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
+		} else {
+			PMD_LOG(ERR, "Can't do mbuf without space yet!");
+			++txq->tx_errors;
+			continue;
+		}
+	}
+
+	if (likely(num_tx > 0))
+		io_uring_submit(&txq->io_ring);
+
+	return num_tx;
+}
+
 static const struct eth_dev_ops ops = {
 	.dev_start		= eth_dev_start,
 	.dev_stop		= eth_dev_stop,
@@ -339,9 +696,12 @@ static const struct eth_dev_ops ops = {
 	.promiscuous_disable	= eth_dev_promiscuous_disable,
 	.allmulticast_enable	= eth_dev_allmulticast_enable,
 	.allmulticast_disable	= eth_dev_allmulticast_disable,
+	.rx_queue_setup		= eth_rx_queue_setup,
+	.rx_queue_release	= eth_rx_queue_release,
+	.tx_queue_setup		= eth_tx_queue_setup,
+	.tx_queue_release	= eth_tx_queue_release,
 };
 
-
 static int
 ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 {
@@ -379,6 +739,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	}
 
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
+
+	dev->rx_pkt_burst = eth_ioring_rx;
+	dev->tx_pkt_burst = eth_ioring_tx;
+
 	return 0;
 
 error:
Signature · 1 line
2.47.2
#42 lore raw

PATCH v4 7/10 net/ioring: implement statistics

In reply to Stephen Hemminger

Add support for basic statistics

---
 doc/guides/nics/features/ioring.ini |  2 +
 drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
index da47062adb..c9a4582d0e 100644
@@ -9,6 +9,8 @@ MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
 Multiprocess aware   = Y
+Basic stats          = Y
+Stats per queue      = Y
 Linux		     = Y
 x86-64               = Y
 Usage doc            = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0

69 lines not on this page · raw message

Signature · 1 line
2.47.2
#43 lore raw

PATCH v4 8/10 net/ioring: support multi-segment Rx and Tx

In reply to Stephen Hemminger

Use readv/writev to handle multi-segment transmit and receive. Account for virtio header that will be used for offload (later).

---
 drivers/net/ioring/rte_eth_ioring.c | 203 ++++++++++++++++++++++------
 1 file changed, 160 insertions(+), 43 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +160 -43
index 18546f0137..633bfc21c2 100644
@@ -18,6 +18,7 @@
 #include <linux/if.h>
 #include <linux/if_arp.h>
 #include <linux/if_tun.h>
+#include <linux/virtio_net.h>
 
 #include <bus_vdev_driver.h>
 #include <ethdev_driver.h>
@@ -30,12 +31,18 @@
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
+static_assert(RTE_PKTMBUF_HEADROOM >= sizeof(struct virtio_net_hdr));
+
 #define IORING_DEFAULT_BURST	64
 #define IORING_NUM_BUFFERS	1024
 #define IORING_MAX_QUEUES	128
 
 static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd limit");
 
+#define IORING_TX_OFFLOAD	RTE_ETH_TX_OFFLOAD_MULTI_SEGS
+
+#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_SCATTER
+
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
 
@@ -162,7 +169,7 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist)
 		goto error;
 	}
 
-	int flags = IFF_TAP | IFF_MULTI_QUEUE | IFF_NO_PI;
+	int flags = IFF_TAP | IFF_MULTI_QUEUE | IFF_NO_PI | IFF_VNET_HDR;
 	if ((features & flags) != flags) {
 		PMD_LOG(ERR, "TUN features %#x missing support for %#x",
 			features, features & flags);
@@ -193,6 +200,13 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist)
 		goto error;
 	}
 
+
+	int hdr_size = sizeof(struct virtio_net_hdr);
+	if (ioctl(tap_fd, TUNSETVNETHDRSZ, &hdr_size) < 0) {
+		PMD_LOG(ERR, "ioctl(TUNSETVNETHDRSZ) %s", strerror(errno));
+		goto error;
+	}
+
 	return tap_fd;
 error:
 	close(tap_fd);
@@ -350,6 +364,8 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->max_rx_queues = IORING_MAX_QUEUES;
 	dev_info->max_tx_queues = IORING_MAX_QUEUES;
 	dev_info->min_rx_bufsize = 0;
+	dev_info->tx_queue_offload_capa = IORING_TX_OFFLOAD;
+	dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa;
 
 	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
 		.burst_size = IORING_DEFAULT_BURST,
@@ -478,7 +494,6 @@ static inline void
 eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
 {
 	struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring);
-
 	if (unlikely(sqe == NULL)) {
 		PMD_LOG(DEBUG, "io_uring no rx sqe");
 		rxq->rx_errors++;
@@ -487,10 +502,81 @@ eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)
 	}
 	io_uring_sqe_set_data(sqe, mb);
 
-	void *buf = rte_pktmbuf_mtod_offset(mb, void *, 0);
-	unsigned int nbytes = rte_pktmbuf_tailroom(mb);
+	RTE_ASSERT(rte_pktmbuf_headroom(mb) >= sizeof(struct virtio_net_hdr));
+	void *buf = rte_pktmbuf_mtod_offset(mb, void *, -sizeof(struct virtio_net_hdr));
+	unsigned int nbytes = sizeof(struct virtio_net_hdr) + rte_pktmbuf_tailroom(mb);
+
+	/* optimize for the case where packet fits in one mbuf */
+	if (mb->nb_segs == 1) {
+		io_uring_prep_read(sqe, fd, buf, nbytes, 0);
+	} else {
+		uint16_t nsegs = mb->nb_segs;
+		RTE_ASSERT(nsegs > 0 && nsegs < IOV_MAX);
+		struct iovec iovs[RTE_MBUF_MAX_NB_SEGS];
+
+		iovs[0].iov_base = buf;
+		iovs[0].iov_len = nbytes;
+
+		for (uint16_t i = 1; i < nsegs; i++) {
+			mb = mb->next;
+			iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *);
+			iovs[i].iov_len = rte_pktmbuf_tailroom(mb);
+		}
+		io_uring_prep_readv(sqe, fd, iovs, nsegs, 0);
+	}
+
+}
+
+
+/* Allocates one or more mbuf's to be used for reading packets */
+static struct rte_mbuf *
+eth_ioring_rx_alloc(struct rx_queue *rxq)
+{
+	const struct rte_eth_dev *dev = &rte_eth_devices[rxq->port_id];
+	int buf_size = dev->data->mtu;
+	struct rte_mbuf *m = NULL;
+	struct rte_mbuf **tail = &m;
+
+	do {
+		struct rte_mbuf *seg = rte_pktmbuf_alloc(rxq->mb_pool);
+		if (unlikely(seg == NULL)) {
+			rte_pktmbuf_free(m);
+			return NULL;
+		}
+		*tail = seg;
+		tail = &seg->next;
+		if (seg != m)
+			++m->nb_segs;
+
+		buf_size -= rte_pktmbuf_tailroom(seg);
+	} while (buf_size > 0);
+
+	__rte_mbuf_sanity_check(m, 1);
+	return m;
+}
+
+
+/* set length of received mbuf segments */
+static inline void
+eth_ioring_rx_adjust(struct rte_mbuf *mb, size_t len)
+{
+	struct rte_mbuf *seg;
+	unsigned int nsegs = 0;
+
+	for (seg = mb; seg != NULL && len > 0; seg = seg->next) {
+		uint16_t seg_len = RTE_MIN(len, rte_pktmbuf_tailroom(mb));
+
+		seg->data_len = seg_len;
+		len -= seg_len;
+		++nsegs;
+	}
 
-	io_uring_prep_read(sqe, fd, buf, nbytes, 0);
+	mb->nb_segs = nsegs;
+	if (len == 0 && seg != NULL) {
+		/* free any residual */
+		rte_pktmbuf_free(seg->next);
+		seg->next = NULL;
+	}
 }
 
 static uint16_t
@@ -505,37 +591,42 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 	io_uring_for_each_cqe(&rxq->io_ring, head, cqe) {
 		struct rte_mbuf *mb = (void *)(uintptr_t)cqe->user_data;
-		ssize_t len = cqe->res;
+		int32_t len = cqe->res;
 
-		PMD_RX_LOG(DEBUG, "cqe %u len %zd", num_cqe, len);
-		num_cqe++;
+		PMD_RX_LOG(DEBUG, "complete m=%p len=%d", mb, len);
 
-		if (unlikely(len < RTE_ETHER_HDR_LEN)) {
-			if (len < 0)
-				PMD_LOG(ERR, "io_uring_read: %s", strerror(-len));
-			else
-				PMD_LOG(ERR, "io_uring_read missing hdr");
+		num_cqe++;
 
+		struct virtio_net_hdr *hdr;
+		if (unlikely(len < (ssize_t)(sizeof(*hdr) + RTE_ETHER_HDR_LEN))) {
+			PMD_LOG(ERR, "io_uring_read result = %d", len);
 			rxq->rx_errors++;
 			goto resubmit;
 		}
 
-		struct rte_mbuf *nmb = rte_pktmbuf_alloc(rxq->mb_pool);
-		if (unlikely(nmb == 0)) {
-			PMD_LOG(DEBUG, "Rx mbuf alloc failed");
+		/* virtio header is before packet data */
+		hdr = rte_pktmbuf_mtod_offset(mb, struct virtio_net_hdr *, -sizeof(*hdr));
+		len -= sizeof(*hdr);
+
+		struct rte_mbuf *nmb = eth_ioring_rx_alloc(rxq);
+		if (!nmb) {
+			PMD_RX_LOG(NOTICE, "alloc failed");
 			++rxq->rx_nombuf;
 			goto resubmit;
 		}
 
-		mb->pkt_len = len;
-		mb->data_len = len;
 		mb->port = rxq->port_id;
-		__rte_mbuf_sanity_check(mb, 1);
+		mb->pkt_len = len;
+
+		if (mb->nb_segs == 1)
+			mb->data_len = len;
+		else
+			eth_ioring_rx_adjust(mb, len);
 
-		num_bytes += len;
+		num_bytes += mb->pkt_len;
 		bufs[num_rx++] = mb;
 
-		mb = nmb;
+		mb = nmb;	/* use the new buffer when resubmitting */
 resubmit:
 		eth_rx_submit(rxq, fd, mb);
 
@@ -581,20 +672,17 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_de
 		return -1;
 	}
 
-	struct rte_mbuf **mbufs = alloca(nb_rx_desc * sizeof(struct rte_mbuf *));
-	if (mbufs == NULL) {
-		PMD_LOG(ERR, "alloca for %u failed", nb_rx_desc);
-		return -1;
-	}
+	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
 
-	if (rte_pktmbuf_alloc_bulk(mb_pool, mbufs, nb_rx_desc) < 0) {
-		PMD_LOG(ERR, "Rx mbuf alloc %u bufs failed", nb_rx_desc);
-		return -1;
-	}
+	for (uint16_t i = 0; i < nb_rx_desc; i++) {
+		struct rte_mbuf *mb = eth_ioring_rx_alloc(rxq);
+		if (mb == NULL) {
+			PMD_LOG(ERR, "Rx mbuf alloc buf failed");
+			return -1;
+		}
 
-	int fd = eth_queue_fd(rxq->port_id, rxq->queue_id);
-	for (uint16_t i = 0; i < nb_rx_desc; i++)
-		eth_rx_submit(rxq, fd, mbufs[i]);
+		eth_rx_submit(rxq, fd, mb);
+	}
 
 	io_uring_submit(&rxq->io_ring);
 	return 0;
@@ -701,8 +789,6 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 	if (unlikely(nb_pkts == 0))
 		return 0;
 
-	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
-
 	if (io_uring_sq_space_left(&txq->io_ring) < txq->free_thresh)
 		eth_ioring_tx_cleanup(txq);
 
@@ -710,23 +796,54 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 	for (num_tx = 0; num_tx < nb_pkts; num_tx++) {
 		struct rte_mbuf *mb = bufs[num_tx];
+		struct virtio_net_hdr *hdr;
 
 		struct io_uring_sqe *sqe = io_uring_get_sqe(&txq->io_ring);
 		if (sqe == NULL)
 			break;	/* submit ring is full */
 
+		if (rte_mbuf_refcnt_read(mb) == 1 &&
+		    RTE_MBUF_DIRECT(mb) &&
+		    rte_pktmbuf_headroom(mb) >= sizeof(*hdr)) {
+			hdr = rte_pktmbuf_mtod_offset(mb, struct virtio_net_hdr *, sizeof(*hdr));
+		} else {
+			struct rte_mbuf *mh = rte_pktmbuf_alloc(mb->pool);
+			if (unlikely(mh == NULL)) {
+				++txq->tx_errors;
+				rte_pktmbuf_free(mb);
+				continue;
+			}
+
+			hdr = rte_pktmbuf_mtod_offset(mh, struct virtio_net_hdr *, sizeof(*hdr));
+			mh->next = mb;
+			mh->nb_segs = mb->nb_segs + 1;
+			mh->pkt_len = mb->pkt_len;
+			mh->ol_flags = mb->ol_flags & RTE_MBUF_F_TX_OFFLOAD_MASK;
+			mb = mh;
+		}
+
 		io_uring_sqe_set_data(sqe, mb);
 
-		if (rte_mbuf_refcnt_read(mb) == 1 &&
-		    RTE_MBUF_DIRECT(mb) && mb->nb_segs == 1) {
-			void *base = rte_pktmbuf_mtod(mb, void *);
-			io_uring_prep_write(sqe, fd, base, mb->pkt_len, 0);
+		PMD_TX_LOG(DEBUG, "write m=%p segs=%u", mb, mb->nb_segs);
+		void *buf = rte_pktmbuf_mtod_offset(mb, void *, -sizeof(*hdr));
+		unsigned int nbytes = sizeof(struct virtio_net_hdr) + mb->data_len;
 
-			PMD_TX_LOG(DEBUG, "tx mbuf: %p submit", mb);
+		if (mb->nb_segs == 1) {
+			io_uring_prep_write(sqe, fd, buf, nbytes, 0);
 		} else {
-			PMD_LOG(ERR, "Can't do mbuf without space yet!");
-			++txq->tx_errors;
-			continue;
+			struct iovec iovs[RTE_MBUF_MAX_NB_SEGS + 1];
+			unsigned int niov = mb->nb_segs;
+
+			iovs[0].iov_base = buf;
+			iovs[0].iov_len = nbytes;
+
+			for (unsigned int i = 1; i < niov; i++) {
+				mb = mb->next;
+				iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *);
+				iovs[i].iov_len = mb->data_len;
+			}
+
+			io_uring_prep_writev(sqe, fd, iovs, niov, 0);
 		}
 	}
 
Signature · 1 line
2.47.2
#44 lore raw

PATCH v4 9/10 net/ioring: support Tx checksum and segment offload

In reply to Stephen Hemminger

The code for transmit flag mapping can be copied from virtio driver. The TAP device does not support querying features of the virtio net header, so the driver assumes checksum offload is allowed.

---
 doc/guides/nics/features/ioring.ini |  2 ++
 drivers/net/ioring/rte_eth_ioring.c | 51 ++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)
doc/guides/nics/features/ioring.ini +2 -0
index c9a4582d0e..17497e1fd3 100644
@@ -10,6 +10,8 @@ Promiscuous mode     = Y
 Allmulticast mode    = Y
 Multiprocess aware   = Y
 Basic stats          = Y
+L3 checksum offload  = Y
+L4 checksum offload  = Y
 Stats per queue      = Y
 Linux		     = Y
 x86-64               = Y
drivers/net/ioring/rte_eth_ioring.c +50 -1

69 lines not on this page · raw message

Signature · 1 line
2.47.2
#45 lore raw

PATCH v4 10/10 net/ioring: add support for Rx offload

In reply to Stephen Hemminger

The TAP device supports receive offload.

---
 drivers/net/ioring/rte_eth_ioring.c | 98 ++++++++++++++++++++++++++++-
 1 file changed, 96 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +96 -2
index 704b887d36..7c60d49b23 100644
@@ -30,6 +30,7 @@
 #include <rte_ether.h>
 #include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_net.h>
 
 static_assert(RTE_PKTMBUF_HEADROOM >= sizeof(struct virtio_net_hdr));
 
@@ -44,7 +45,10 @@ static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd
 				 RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \
 				 RTE_ETH_TX_OFFLOAD_TCP_TSO)
 
-#define IORING_RX_OFFLOAD	RTE_ETH_RX_OFFLOAD_SCATTER
+#define IORING_RX_OFFLOAD	(RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_LRO | \
+				 RTE_ETH_RX_OFFLOAD_SCATTER)
 
 #define IORING_DEFAULT_IFNAME	"itap%d"
 #define IORING_MP_KEY		"ioring_mp_send_fds"
@@ -349,10 +353,31 @@ eth_dev_stop(struct rte_eth_dev *dev)
 static int
 eth_dev_configure(struct rte_eth_dev *dev)
 {
+	struct pmd_internals *pmd = dev->data->dev_private;
+
 	/* rx/tx must be paired */
 	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
 		return -EINVAL;
 
+	/*
+	 * Set offload flags visible on the kernel network interface.
+	 * This controls whether kernel will use checksum offload etc.
+	 * Note: kernel transmit is DPDK receive.
+	 */
+	const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode;
+	unsigned int offload = 0;
+	if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_CHECKSUM) {
+		offload |= TUN_F_CSUM;
+
+		if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO)
+			offload |= TUN_F_TSO4 | TUN_F_TSO6 | TUN_F_TSO_ECN;
+	}
+
+	if (ioctl(pmd->keep_fd, TUNSETOFFLOAD, offload) != 0) {
+		PMD_LOG(ERR, "ioctl(TUNSETOFFLOAD) failed: %s", strerror(errno));
+		return -1;
+	}
+
 	return 0;
 }
 
@@ -558,7 +583,6 @@ eth_ioring_rx_alloc(struct rx_queue *rxq)
 	return m;
 }
 
-
 /* set length of received mbuf segments */
 static inline void
 eth_ioring_rx_adjust(struct rte_mbuf *mb, size_t len)
@@ -582,6 +606,69 @@ eth_ioring_rx_adjust(struct rte_mbuf *mb, size_t len)
 	}
 }
 
+static int
+eth_ioring_rx_offload(struct rte_mbuf *m, const struct virtio_net_hdr *hdr)
+{
+	uint32_t ptype;
+	bool l4_supported = false;
+	struct rte_net_hdr_lens hdr_lens;
+
+	/* nothing to do */
+	if (hdr->flags == 0 && hdr->gso_type == VIRTIO_NET_HDR_GSO_NONE)
+		return 0;
+
+	m->ol_flags |= RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN;
+
+	ptype = rte_net_get_ptype(m, &hdr_lens, RTE_PTYPE_ALL_MASK);
+	m->packet_type = ptype;
+	if ((ptype & RTE_PTYPE_L4_MASK) == RTE_PTYPE_L4_TCP ||
+	    (ptype & RTE_PTYPE_L4_MASK) == RTE_PTYPE_L4_UDP ||
+	    (ptype & RTE_PTYPE_L4_MASK) == RTE_PTYPE_L4_SCTP)
+		l4_supported = true;
+
+	if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
+		uint32_t hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
+		if (hdr->csum_start <= hdrlen && l4_supported) {
+			m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
+		} else {
+			/* Unknown proto or tunnel, do sw cksum. */
+			uint16_t csum = 0, off;
+
+			if (rte_raw_cksum_mbuf(m, hdr->csum_start,
+					       rte_pktmbuf_pkt_len(m) - hdr->csum_start,
+					       &csum) < 0)
+				return -EINVAL;
+			if (likely(csum != 0xffff))
+				csum = ~csum;
+			off = hdr->csum_offset + hdr->csum_start;
+			if (rte_pktmbuf_data_len(m) >= off + 1)
+				*rte_pktmbuf_mtod_offset(m, uint16_t *, off) = csum;
+		}
+	} else if ((hdr->flags & VIRTIO_NET_HDR_F_DATA_VALID) && l4_supported) {
+		m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_GOOD;
+	}
+
+	/* GSO request, save required information in mbuf */
+	if (hdr->gso_type != VIRTIO_NET_HDR_GSO_NONE) {
+		/* Check unsupported modes */
+		if ((hdr->gso_type & VIRTIO_NET_HDR_GSO_ECN) || hdr->gso_size == 0)
+			return -EINVAL;
+
+		/* Update mss lengths in mbuf */
+		m->tso_segsz = hdr->gso_size;
+		switch (hdr->gso_type & ~VIRTIO_NET_HDR_GSO_ECN) {
+		case VIRTIO_NET_HDR_GSO_TCPV4:
+		case VIRTIO_NET_HDR_GSO_TCPV6:
+			m->ol_flags |= RTE_MBUF_F_RX_LRO | RTE_MBUF_F_RX_L4_CKSUM_NONE;
+			break;
+		default:
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 static uint16_t
 eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 {
@@ -626,6 +713,13 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 		else
 			eth_ioring_rx_adjust(mb, len);
 
+		if (unlikely(eth_ioring_rx_offload(mb, hdr) < 0)) {
+			PMD_RX_LOG(ERR, "invalid rx offload");
+			++rxq->rx_errors;
+			goto resubmit;
+		}
+
+		__rte_mbuf_sanity_check(mb, 1);
 		num_bytes += mb->pkt_len;
 		bufs[num_rx++] = mb;
 
Signature · 1 line
2.47.2
#46 lore raw

v5 cover net/rtap: add io_uring based TAP driver

In reply to Stephen Hemminger

This series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.

Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.

Motivation
----------

This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.

The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.

Why not extend net_tap?
-----------------------

The existing net_tap driver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity: rte_flow support emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.

net_rtap takes the opposite approach — use the kernel efficiently and let it do what it does well. There is no rte_flow support; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.

Given these fundamentally different design goals, a clean implementation was more practical than refactoring net_tap.

Acknowledgement
---------------

Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.

Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)
- liburing >= 2.0

Known working distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).

v5 - revised, renamed and expanded from the v4 ioring PMD
- more complete testing and dependency handling

Stephen Hemminger (10):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add link state change interrupt net/rtap: add multi-process support
net/rtap: add Rx interrupt support
test: add unit tests for rtap PMD

 MAINTAINERS                            |    7 +
 app/test/meson.build                   |    1 +
 app/test/test_pmd_rtap.c               | 2044 ++++++++++++++++++++++++
 doc/guides/nics/features/rtap.ini      |   25 +
 doc/guides/nics/index.rst              |    1 +
 doc/guides/nics/rtap.rst               |  101 ++
 doc/guides/rel_notes/release_26_03.rst |    6 +
 drivers/net/meson.build                |    1 +
 drivers/net/rtap/meson.build           |   28 +
 drivers/net/rtap/rtap.h                |  100 ++
 drivers/net/rtap/rtap_ethdev.c         |  908 +++++++++++
 drivers/net/rtap/rtap_intr.c           |  267 ++++
 drivers/net/rtap/rtap_rxtx.c           |  784 +++++++++
 13 files changed, 4273 insertions(+)
 create mode 100644 app/test/test_pmd_rtap.c
 create mode 100644 doc/guides/nics/features/rtap.ini
 create mode 100644 doc/guides/nics/rtap.rst
 create mode 100644 drivers/net/rtap/meson.build
 create mode 100644 drivers/net/rtap/rtap.h
 create mode 100644 drivers/net/rtap/rtap_ethdev.c
 create mode 100644 drivers/net/rtap/rtap_intr.c
 create mode 100644 drivers/net/rtap/rtap_rxtx.c
Signature · 1 line
2.51.0
#47 lore raw

PATCH v5 1/10 net/rtap: add driver skeleton and documentation

Add the initial skeleton for the rtap poll mode driver, a virtual ethernet device that uses Linux io_uring for packet I/O with kernel TAP devices.

This patch includes:
- MAINTAINERS entry
- Driver documentation (doc/guides/nics/rtap.rst)
- Feature matrix (doc/guides/nics/features/rtap.ini)
- Release notes update
- Meson build integration with liburing dependency
- Header file with shared data structures and declarations
- Stub probe/remove handlers that register the vdev driver
- Empty dev_ops with only dev_close implemented

The driver registers as net_rtap and is Linux-only.
It requires the liburing library version 2.0 or later. Earlier versions have known security and build issues. The library is available in all currently supported distributions (Debian 12+, Ubuntu 22.04+, RHEL 9+, Fedora 35+)

---
 MAINTAINERS                            |   7 +
 doc/guides/nics/features/rtap.ini      |  13 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/nics/rtap.rst               | 101 +++++++++++++++
 doc/guides/rel_notes/release_26_03.rst |   6 +
 drivers/net/meson.build                |   1 +
 drivers/net/rtap/meson.build           |  26 ++++
 drivers/net/rtap/rtap.h                |  69 ++++++++++
 drivers/net/rtap/rtap_ethdev.c         | 172 +++++++++++++++++++++++++
 9 files changed, 396 insertions(+)
 create mode 100644 doc/guides/nics/features/rtap.ini
 create mode 100644 doc/guides/nics/rtap.rst
 create mode 100644 drivers/net/rtap/meson.build
 create mode 100644 drivers/net/rtap/rtap.h
 create mode 100644 drivers/net/rtap/rtap_ethdev.c
MAINTAINERS +7 -0
index 5683b87e4a..3d0877fdc7 100644
@@ -1135,6 +1135,13 @@ F: doc/guides/nics/pcap_ring.rst
 F: app/test/test_pmd_ring.c
 F: app/test/test_pmd_ring_perf.c
 
+Rtap PMD - EXPERIMENTAL
+M: Stephen Hemminger <[email protected]>
+F: drivers/net/rtap/
+F: app/test/test_pmd_rtap.c
+F: doc/guides/nics/rtap.rst
+F: doc/guides/nics/features/rtap.ini
+
 Null Networking PMD
 M: Tetsuya Mukawa <[email protected]>
 F: drivers/net/null/
doc/guides/nics/features/rtap.ini new file +13 -0
new file mode 100644
index 0000000000..ed7c638029
@@ -0,0 +1,13 @@
+;
+; Supported features of the 'rtap' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+ARMv7                = Y
+ARMv8                = Y
+Power8               = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index b00ed998c5..274575fe70 100644
@@ -65,6 +65,7 @@ Network Interface Controller Drivers
     qede
     r8169
     rnp
+    rtap
     sfc_efx
     softnic
     tap
doc/guides/nics/rtap.rst new file +101 -0

101 lines not on this page · raw message

doc/guides/rel_notes/release_26_03.rst +6 -0
index 031eaa657e..db5c61a15c 100644
@@ -63,6 +63,12 @@ New Features
 
   * Added support for pre and post VF reset callbacks.
 
+* **Added rtap virtual ethernet driver.**
+
+  Added a new experimental virtual device driver that uses Linux io_uring
+  for packet injection into the kernel network stack.
+  It requires Linux kernel 5.1 or later and the liburing library.
+
 
 Removed Items
 -------------
drivers/net/meson.build +1 -0
index c7dae4ad27..ef1ee68385 100644
@@ -56,6 +56,7 @@ drivers = [
         'r8169',
         'ring',
         'rnp',
+        'rtap',
         'sfc',
         'softnic',
         'tap',
drivers/net/rtap/meson.build new file +26 -0
new file mode 100644
index 0000000000..7bd7806ef3
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+liburing = dependency('liburing', version: '>= 2.0', required: false)
+if not liburing.found()
+    build = false
+    reason = 'missing dependency, "liburing"'
+endif
+
+if build and not cc.has_header_symbol('linux/io_uring.h', 'IORING_ASYNC_CANCEL_ALL')
+    build = false
+    reason = 'kernel headers missing IORING_ASYNC_CANCEL_ALL (need kernel >= 5.19 headers)'
+endif
+
+sources = files(
+        'rtap_ethdev.c',
+)
+
+ext_deps += liburing
+
+require_iova_in_mbuf = false
drivers/net/rtap/rtap.h new file +69 -0

69 lines not on this page · raw message

drivers/net/rtap/rtap_ethdev.c new file +172 -0

172 lines not on this page · raw message

Signature · 1 line
2.51.0
#48 lore raw

PATCH v5 2/10 net/rtap: add TAP device creation and queue management

In reply to Stephen Hemminger

Add TAP device creation using the Linux TUN/TAP interface with IFF_MULTI_QUEUE, IFF_NO_PI, and IFF_VNET_HDR flags. Enable NAPI mode when the kernel supports it.

The driver maintains a keep-alive file descriptor to the TAP device and opens additional per-queue file descriptors for data path I/O. This mirrors the multi-queue TAP architecture where each queue pair (rx + tx) shares a single TAP fd.

Add the rtap_create() function that:
- Opens the TAP device with configurable interface name
- Configures the virtio-net header size
- Reads the kernel-assigned MAC address
- Supports the 'persist' option to keep the interface after exit
- Detaches the keep-alive queue from data traffic

Add rtap_queue_open() and rtap_queue_close() for per-queue fd management used during queue setup and teardown.

---
 drivers/net/rtap/rtap.h        |   4 +
 drivers/net/rtap/rtap_ethdev.c | 216 ++++++++++++++++++++++++++++++++-
 2 files changed, 219 insertions(+), 1 deletion(-)
drivers/net/rtap/rtap.h +4 -0
index 507ab000f3..39a3188a7b 100644
@@ -66,4 +66,8 @@ struct rtap_pmd {
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 };
 
+/* rtap_ethdev.c */
+int rtap_queue_open(struct rte_eth_dev *dev, uint16_t queue_id);
+void rtap_queue_close(struct rte_eth_dev *dev, uint16_t queue_id);
+
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +215 -1

245 lines not on this page · raw message

Signature · 1 line
2.51.0
#49 lore raw

PATCH v5 3/10 net/rtap: add Rx/Tx with scatter/gather support

In reply to Stephen Hemminger

Implement packet receive and transmit using io_uring asynchronous I/O, with full support for both single-segment and multi-segment mbufs.

Rx path:
- rtap_rx_alloc() chains multiple mbufs when the MTU exceeds a single mbuf's tailroom capacity
- Pre-post read/readv requests to the io_uring submission queue, each backed by a pre-allocated (possibly chained) mbuf
- On rx_burst, harvest completed CQEs and replace each consumed mbuf with a freshly allocated one
- rtap_rx_adjust() distributes received data across segments and frees unused trailing segments
- Parse the prepended virtio-net header (offload fields are ignored until the offload patch)

Tx path:
- For single-segment mbufs, use io_uring write and batch submits
- For multi-segment mbufs, use writev via io_uring with immediate submit (iovec is stack-allocated)
- When the mbuf headroom is not writable (shared or indirect), chain a new header mbuf for the virtio-net header
- Prepend a zeroed virtio-net header (offload population deferred)
- Clean completed tx CQEs to free transmitted mbufs

Add io_uring cancel-all logic using IORING_ASYNC_CANCEL_ALL for clean queue teardown, draining all pending CQEs and freeing mbufs.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/meson.build      |   1 +
 drivers/net/rtap/rtap.h           |  13 +
 drivers/net/rtap/rtap_ethdev.c    |   7 +
 drivers/net/rtap/rtap_rxtx.c      | 755 ++++++++++++++++++++++++++++++
 5 files changed, 777 insertions(+)
 create mode 100644 drivers/net/rtap/rtap_rxtx.c
doc/guides/nics/features/rtap.ini +1 -0
index ed7c638029..c064e1e0b9 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Scattered Rx         = P
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/meson.build +1 -0
index 7bd7806ef3..8e2b15f382 100644
@@ -19,6 +19,7 @@ endif
 
 sources = files(
         'rtap_ethdev.c',
+        'rtap_rxtx.c',
 )
 
 ext_deps += liburing
drivers/net/rtap/rtap.h +13 -0
index 39a3188a7b..a0bbb1a8a0 100644
@@ -70,4 +70,17 @@ struct rtap_pmd {
 int rtap_queue_open(struct rte_eth_dev *dev, uint16_t queue_id);
 void rtap_queue_close(struct rte_eth_dev *dev, uint16_t queue_id);
 
+/* rtap_rxtx.c */
+uint16_t rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts);
+uint16_t rtap_tx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts);
+int rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+			uint16_t nb_rx_desc, unsigned int socket_id,
+			const struct rte_eth_rxconf *rx_conf,
+			struct rte_mempool *mb_pool);
+void rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+			uint16_t nb_tx_desc, unsigned int socket_id,
+			const struct rte_eth_txconf *tx_conf);
+void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
+
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +7 -0
index 4e7847ff8d..a65a8b77ad 100644
@@ -232,6 +232,10 @@ static const struct eth_dev_ops rtap_ops = {
 	.dev_stop		= rtap_dev_stop,
 	.dev_configure		= rtap_dev_configure,
 	.dev_close		= rtap_dev_close,
+	.rx_queue_setup		= rtap_rx_queue_setup,
+	.rx_queue_release	= rtap_rx_queue_release,
+	.tx_queue_setup		= rtap_tx_queue_setup,
+	.tx_queue_release	= rtap_tx_queue_release,
 };
 
 static int
@@ -272,6 +276,9 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 
 	PMD_LOG(DEBUG, "%s setup", ifr.ifr_name);
 
+	dev->rx_pkt_burst = rtap_rx_burst;
+	dev->tx_pkt_burst = rtap_tx_burst;
+
 	return 0;
 
 error:
drivers/net/rtap/rtap_rxtx.c new file +755 -0

755 lines not on this page · raw message

Signature · 1 line
2.51.0
#50 lore raw

PATCH v5 4/10 net/rtap: add statistics and device info

In reply to Stephen Hemminger

Implement basic and per-queue statistics collection and the device info query.

Stats tracked per rx queue: packets, bytes, errors, and missed. Stats tracked per tx queue: packets, bytes, errors.

Device info reports:
- Interface index from the kernel TAP device
- Maximum queue counts
- Default burst size and ring size

Enable AUTOFILL_QUEUE_XSTATS for automatic per-queue extended statistics.

---
 doc/guides/nics/features/rtap.ini |   2 +
 drivers/net/rtap/rtap.h           |   2 +
 drivers/net/rtap/rtap_ethdev.c    | 139 ++++++++++++++++++++++++++++++
 3 files changed, 143 insertions(+)
doc/guides/nics/features/rtap.ini +2 -0
index c064e1e0b9..9bef9e341d 100644
@@ -5,6 +5,8 @@
 ;
 [Features]
 Scattered Rx         = P
+Basic stats          = Y
+Stats per queue      = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap.h +2 -0
index a0bbb1a8a0..a93d7dae17 100644
@@ -64,6 +64,8 @@ struct rtap_pmd {
 	int keep_fd;			/* keep alive file descriptor */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
+
+	uint64_t rx_drop_base;		/* value of rx_dropped when reset */
 };
 
 /* rtap_ethdev.c */
drivers/net/rtap/rtap_ethdev.c +139 -0

170 lines not on this page · raw message

Signature · 1 line
2.51.0
#51 lore raw

PATCH v5 5/10 net/rtap: add link and device management operations

In reply to Stephen Hemminger

Add device management operations for the TAP interface:
- Link up/down control via SIOCSIFFLAGS
- Link status query via SIOCGIFFLAGS
- Promiscuous mode enable/disable
- All-multicast mode enable/disable
- MTU configuration via SIOCSIFMTU
- MAC address configuration via SIOCSIFHWADDR

These operations manipulate the kernel TAP interface flags and parameters through ioctl calls.

---
 doc/guides/nics/features/rtap.ini |   4 +
 drivers/net/rtap/rtap.h           |   1 +
 drivers/net/rtap/rtap_ethdev.c    | 172 ++++++++++++++++++++++++++++++
 3 files changed, 177 insertions(+)
doc/guides/nics/features/rtap.ini +4 -0
index 9bef9e341d..ce0804d795 100644
@@ -4,6 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Scattered Rx         = P
 Basic stats          = Y
 Stats per queue      = Y
drivers/net/rtap/rtap.h +1 -0
index a93d7dae17..99f413f001 100644
@@ -71,6 +71,7 @@ struct rtap_pmd {
 /* rtap_ethdev.c */
 int rtap_queue_open(struct rte_eth_dev *dev, uint16_t queue_id);
 void rtap_queue_close(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_link_update(struct rte_eth_dev *dev, int wait_to_complete);
 
 /* rtap_rxtx.c */
 uint16_t rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts);
drivers/net/rtap/rtap_ethdev.c +172 -0

193 lines not on this page · raw message

Signature · 1 line
2.51.0
#52 lore raw

PATCH v5 6/10 net/rtap: add checksum and TSO offload support

In reply to Stephen Hemminger

Add transmit and receive offload support using the virtio-net header that is exchanged with the kernel TAP device.

Tx offloads:
- UDP and TCP checksum offload: set VIRTIO_NET_HDR_F_NEEDS_CSUM with appropriate csum_start and csum_offset fields
- TCP segmentation offload (TSO): set gso_type and gso_size

Rx offloads:
- Parse virtio-net header flags to determine checksum status
- Handle NEEDS_CSUM by either marking L4_CKSUM_NONE for known protocols or computing the checksum in software for tunnels
- Handle DATA_VALID flag to mark L4_CKSUM_GOOD
- Parse GSO headers to populate LRO metadata

Configure kernel-side offloads via TUNSETOFFLOAD in dev_configure based on the requested rx offload flags.

Report offload capabilities in dev_info.

---
 doc/guides/nics/features/rtap.ini |  2 ++
 drivers/net/rtap/rtap_ethdev.c    | 30 +++++++++++++++++++++++++++++-
 2 files changed, 31 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +2 -0
index ce0804d795..b8eaa805fe 100644
@@ -11,6 +11,8 @@ Allmulticast mode    = Y
 Scattered Rx         = P
 Basic stats          = Y
 Stats per queue      = Y
+TSO                  = Y
+L4 checksum offload  = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap_ethdev.c +29 -1
index 591fe91eac..277a280772 100644
@@ -31,6 +31,16 @@
 
 #define RTAP_DEFAULT_IFNAME	"rtap%d"
 
+#define RTAP_TX_OFFLOAD		(RTE_ETH_TX_OFFLOAD_MULTI_SEGS | \
+				 RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \
+				 RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \
+				 RTE_ETH_TX_OFFLOAD_TCP_TSO)
+
+#define RTAP_RX_OFFLOAD		(RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_LRO | \
+				 RTE_ETH_RX_OFFLOAD_SCATTER)
+
 #define RTAP_DEFAULT_BURST	64
 #define RTAP_NUM_BUFFERS	1024
 #define RTAP_MAX_QUEUES		128
@@ -317,7 +327,21 @@ rtap_dev_configure(struct rte_eth_dev *dev)
 	if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)
 		return -EINVAL;
 
-	if (ioctl(pmd->keep_fd, TUNSETOFFLOAD, 0) != 0) {
+	/*
+	 * Set offload flags visible on the kernel network interface.
+	 * This controls whether kernel will use checksum offload etc.
+	 * Note: kernel transmit is DPDK receive.
+	 */
+	const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode;
+	unsigned int offload = 0;
+	if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_CHECKSUM) {
+		offload |= TUN_F_CSUM;
+
+		if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO)
+			offload |= TUN_F_TSO4 | TUN_F_TSO6 | TUN_F_TSO_ECN;
+	}
+
+	if (ioctl(pmd->keep_fd, TUNSETOFFLOAD, offload) != 0) {
 		PMD_LOG(ERR, "ioctl(TUNSETOFFLOAD) failed: %s", strerror(errno));
 		return -1;
 	}
@@ -336,6 +360,10 @@ rtap_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_rx_bufsize = RTAP_MIN_RX_BUFSIZE;
 	dev_info->max_rx_queues = RTAP_MAX_QUEUES;
 	dev_info->max_tx_queues = RTAP_MAX_QUEUES;
+	dev_info->rx_queue_offload_capa = RTAP_RX_OFFLOAD;
+	dev_info->rx_offload_capa = dev_info->rx_queue_offload_capa;
+	dev_info->tx_queue_offload_capa = RTAP_TX_OFFLOAD;
+	dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa;
 
 	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
 		.burst_size = RTAP_DEFAULT_BURST,
Signature · 1 line
2.51.0
#53 lore raw

PATCH v5 7/10 net/rtap: add link state change interrupt

In reply to Stephen Hemminger

Add link state change (LSC) notification support using a netlink socket subscribed to RTMGRP_LINK events.

When LSC is enabled in the device configuration, the driver:
- Creates a NETLINK_ROUTE socket filtering for RTM_NEWLINK and RTM_DELLINK messages
- Registers the socket fd with the EAL interrupt handler
- On interrupt, drains all pending netlink messages and updates the link status for matching interface index changes

The interrupt is enabled during dev_start and disabled during dev_stop, with proper cleanup of the netlink socket and EAL callback registration.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/meson.build      |   1 +
 drivers/net/rtap/rtap.h           |   5 +
 drivers/net/rtap/rtap_ethdev.c    |  36 +++++++-
 drivers/net/rtap/rtap_intr.c      | 147 ++++++++++++++++++++++++++++++
 5 files changed, 187 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/rtap/rtap_intr.c
doc/guides/nics/features/rtap.ini +1 -0
index b8eaa805fe..36a14e9696 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+Link status event    = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
drivers/net/rtap/meson.build +1 -0
index 8e2b15f382..86d400323c 100644
@@ -19,6 +19,7 @@ endif
 
 sources = files(
         'rtap_ethdev.c',
+        'rtap_intr.c',
         'rtap_rxtx.c',
 )
 
drivers/net/rtap/rtap.h +5 -0
index 99f413f001..f73b5e317d 100644
@@ -13,6 +13,7 @@
 
 #include <ethdev_driver.h>
 #include <rte_ether.h>
+#include <rte_interrupts.h>
 #include <rte_log.h>
 
 
@@ -62,6 +63,7 @@ struct rtap_tx_queue {
 
 struct rtap_pmd {
 	int keep_fd;			/* keep alive file descriptor */
+	struct rte_intr_handle *intr_handle; /* LSC interrupt handle */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 
@@ -86,4 +88,7 @@ int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 			const struct rte_eth_txconf *tx_conf);
 void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
 
+/* rtap_intr.c */
+int rtap_lsc_set(struct rte_eth_dev *dev, int set);
+
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +33 -3
index 277a280772..8c22021655 100644
@@ -24,6 +24,7 @@
 #include <rte_eal.h>
 #include <rte_ethdev.h>
 #include <rte_ether.h>
+#include <rte_interrupts.h>
 #include <rte_kvargs.h>
 #include <rte_log.h>
 
@@ -141,11 +142,16 @@ rtap_change_flags(struct rte_eth_dev *dev, uint32_t flags, uint32_t mask)
 	ifr.ifr_flags |= flags;
 
 	ret = ioctl(sock, SIOCSIFFLAGS, &ifr);
-	if (ret < 0)
+	if (ret < 0) {
 		PMD_LOG_ERRNO(ERR, "Unable to set flags for %s", ifr.ifr_name);
+		goto error;
+	}
+	close(sock);
+	return 0;
 error:
+	ret = -errno;
 	close(sock);
-	return (ret < 0) ? -errno : 0;
+	return ret;
 }
 
 static int
@@ -277,11 +283,18 @@ rtap_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 static int
 rtap_dev_start(struct rte_eth_dev *dev)
 {
-	int ret = rtap_set_link_up(dev);
+	int ret;
 
+	ret = rtap_lsc_set(dev, 1);
 	if (ret != 0)
 		return ret;
 
+	ret = rtap_set_link_up(dev);
+	if (ret != 0) {
+		rtap_lsc_set(dev, 0);
+		return ret;
+	}
+
 	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
 	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
 		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
@@ -298,6 +311,7 @@ rtap_dev_stop(struct rte_eth_dev *dev)
 
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
 
+	rtap_lsc_set(dev, 0);
 	rtap_set_link_down(dev);
 
 	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
@@ -508,6 +522,9 @@ rtap_dev_close(struct rte_eth_dev *dev)
 			close(pmd->keep_fd);
 			pmd->keep_fd = -1;
 		}
+
+		rte_intr_instance_free(pmd->intr_handle);
+		pmd->intr_handle = NULL;
 	}
 
 	free(dev->process_private);
@@ -585,6 +602,17 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	pmd->keep_fd = -1;
 	pmd->rx_drop_base = 0;
 
+	/* Allocate interrupt instance for link state change events */
+	pmd->intr_handle = rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_SHARED);
+	if (pmd->intr_handle == NULL) {
+		PMD_LOG(ERR, "Failed to allocate intr handle");
+		goto error;
+	}
+	rte_intr_type_set(pmd->intr_handle, RTE_INTR_HANDLE_EXT);
+	rte_intr_fd_set(pmd->intr_handle, -1);
+	dev->intr_handle = pmd->intr_handle;
+	data->dev_flags |= RTE_ETH_DEV_INTR_LSC;
+
 	dev->dev_ops = &rtap_ops;
 
 	/* Get the initial fd used to keep the tap device around */
@@ -623,6 +651,8 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 error:
 	if (pmd->keep_fd != -1)
 		close(pmd->keep_fd);
+	rte_intr_instance_free(pmd->intr_handle);
+	pmd->intr_handle = NULL;
 	return -1;
 }
 
drivers/net/rtap/rtap_intr.c new file +147 -0

147 lines not on this page · raw message

Signature · 1 line
2.51.0
#54 lore raw

PATCH v5 8/10 net/rtap: add multi-process support

In reply to Stephen Hemminger

Add support for DPDK multi-process operation. Secondary processes need access to the per-queue TAP file descriptors owned by the primary process.

Implement fd sharing using the DPDK multi-process IPC mechanism:
- Primary registers an MP action handler that responds to fd requests by sending queue fds via rte_mp_reply()
- Secondary process attaches to the existing device and requests fds from primary via rte_mp_request_sync()

The MP action is registered on the first device probe and unregistered when the last device is closed.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/rtap_ethdev.c    | 130 ++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
doc/guides/nics/features/rtap.ini +1 -0
index 36a14e9696..fe0c88a8fc 100644
@@ -14,6 +14,7 @@ Basic stats          = Y
 Stats per queue      = Y
 TSO                  = Y
 L4 checksum offload  = Y
+Multiprocess aware   = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap_ethdev.c +130 -0

166 lines not on this page · raw message

Signature · 1 line
2.51.0
#55 lore raw

PATCH v5 9/10 net/rtap: add Rx interrupt support

In reply to Stephen Hemminger

Add support for the DPDK Rx interrupt mechanism, enabling power-aware applications (e.g. l3fwd-power) to sleep until packets arrive rather than busy-polling.

Each Rx queue creates an eventfd during queue setup and registers it with its io_uring instance via io_uring_register_eventfd(). When the kernel posts a CQE (completing a read, i.e. a packet arrived), it signals the eventfd. These per-queue eventfds are wired into a VDEV interrupt handle during dev_start when the application has set intr_conf.rxq.

The enable op drains the eventfd counter to re-arm notification; disable is a no-op since the application simply stops polling. The eventfd is always created unconditionally so it is available if the application enables Rx interrupts later.

The Rx interrupt handle is kept separate from the existing LSC netlink interrupt handle to avoid coupling the two mechanisms.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/rtap.h           |   6 ++
 drivers/net/rtap/rtap_ethdev.c    |  16 ++++
 drivers/net/rtap/rtap_intr.c      | 120 ++++++++++++++++++++++++++++++
 drivers/net/rtap/rtap_rxtx.c      |  31 +++++++-
 5 files changed, 173 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +1 -0
index fe0c88a8fc..48fe3f1b33 100644
@@ -6,6 +6,7 @@
 [Features]
 Link status          = Y
 Link status event    = Y
+Rx interrupt         = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
drivers/net/rtap/rtap.h +6 -0
index f73b5e317d..f37cac87ad 100644
@@ -42,6 +42,7 @@ extern int rtap_logtype;
 struct rtap_rx_queue {
 	struct rte_mempool *mb_pool;	/* rx buffer pool */
 	struct io_uring io_ring;	/* queue of posted read's */
+	int intr_fd;			/* eventfd for Rx interrupt */
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -64,6 +65,7 @@ struct rtap_tx_queue {
 struct rtap_pmd {
 	int keep_fd;			/* keep alive file descriptor */
 	struct rte_intr_handle *intr_handle; /* LSC interrupt handle */
+	struct rte_intr_handle *rx_intr_handle; /* Rx queue interrupt handle */
 	char ifname[IFNAMSIZ];		/* name assigned by kernel */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 
@@ -90,5 +92,9 @@ void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
 
 /* rtap_intr.c */
 int rtap_lsc_set(struct rte_eth_dev *dev, int set);
+int rtap_rx_intr_vec_install(struct rte_eth_dev *dev);
+void rtap_rx_intr_vec_uninstall(struct rte_eth_dev *dev);
+int rtap_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id);
 
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +16 -0
index 9b8ad1f452..8a9fb5be85 100644
@@ -293,8 +293,18 @@ rtap_dev_start(struct rte_eth_dev *dev)
 	if (ret != 0)
 		return ret;
 
+	/* Install Rx interrupt vector if requested by application */
+	if (dev->data->dev_conf.intr_conf.rxq) {
+		ret = rtap_rx_intr_vec_install(dev);
+		if (ret != 0) {
+			rtap_lsc_set(dev, 0);
+			return ret;
+		}
+	}
+
 	ret = rtap_set_link_up(dev);
 	if (ret != 0) {
+		rtap_rx_intr_vec_uninstall(dev);
 		rtap_lsc_set(dev, 0);
 		return ret;
 	}
@@ -315,6 +325,7 @@ rtap_dev_stop(struct rte_eth_dev *dev)
 
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
 
+	rtap_rx_intr_vec_uninstall(dev);
 	rtap_lsc_set(dev, 0);
 	rtap_set_link_down(dev);
 
@@ -527,6 +538,9 @@ rtap_dev_close(struct rte_eth_dev *dev)
 			pmd->keep_fd = -1;
 		}
 
+		rte_intr_instance_free(pmd->rx_intr_handle);
+		pmd->rx_intr_handle = NULL;
+
 		rte_intr_instance_free(pmd->intr_handle);
 		pmd->intr_handle = NULL;
 	}
@@ -597,6 +611,8 @@ static const struct eth_dev_ops rtap_ops = {
 	.rx_queue_release	= rtap_rx_queue_release,
 	.tx_queue_setup		= rtap_tx_queue_setup,
 	.tx_queue_release	= rtap_tx_queue_release,
+	.rx_queue_intr_enable	= rtap_rx_queue_intr_enable,
+	.rx_queue_intr_disable	= rtap_rx_queue_intr_disable,
 };
 
 static int
drivers/net/rtap/rtap_intr.c +120 -0

123 lines not on this page · raw message

drivers/net/rtap/rtap_rxtx.c +30 -1
index c972ab4ca0..87d181eded 100644
@@ -8,6 +8,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <sys/eventfd.h>
 #include <liburing.h>
 #include <sys/uio.h>
 #include <linux/virtio_net.h>
@@ -369,6 +370,7 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 	rxq->mb_pool = mb_pool;
 	rxq->port_id = dev->data->port_id;
 	rxq->queue_id = queue_id;
+	rxq->intr_fd = -1;
 	dev->data->rx_queues[queue_id] = rxq;
 
 	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
@@ -376,10 +378,26 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 		goto error_rxq_free;
 	}
 
+	/*
+	 * Create an eventfd for Rx interrupt notification.
+	 * io_uring will signal this fd whenever a CQE is posted,
+	 * enabling power-aware applications to sleep until packets arrive.
+	 */
+	rxq->intr_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+	if (rxq->intr_fd < 0) {
+		PMD_LOG(ERR, "eventfd failed: %s", strerror(errno));
+		goto error_iouring_exit;
+	}
+
+	if (io_uring_register_eventfd(&rxq->io_ring, rxq->intr_fd) < 0) {
+		PMD_LOG(ERR, "io_uring_register_eventfd failed: %s", strerror(errno));
+		goto error_eventfd_close;
+	}
+
 	mbufs = calloc(nb_rx_desc, sizeof(struct rte_mbuf *));
 	if (mbufs == NULL) {
 		PMD_LOG(ERR, "Rx mbuf pointer alloc failed");
-		goto error_iouring_exit;
+		goto error_eventfd_close;
 	}
 
 	/* open shared tap fd maybe already setup */
@@ -429,6 +447,11 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 	}
 	rtap_queue_close(dev, queue_id);
 	free(mbufs);
+error_eventfd_close:
+	if (rxq->intr_fd >= 0) {
+		close(rxq->intr_fd);
+		rxq->intr_fd = -1;
+	}
 error_iouring_exit:
 	io_uring_queue_exit(&rxq->io_ring);
 error_rxq_free:
@@ -503,6 +526,12 @@ rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
 	if (rxq == NULL)
 		return;
 
+	if (rxq->intr_fd >= 0) {
+		io_uring_unregister_eventfd(&rxq->io_ring);
+		close(rxq->intr_fd);
+		rxq->intr_fd = -1;
+	}
+
 	rtap_cancel_all(&rxq->io_ring);
 	io_uring_queue_exit(&rxq->io_ring);
 
Signature · 1 line
2.51.0
#56 lore raw

PATCH v5 10/10 test: add unit tests for rtap PMD

In reply to Stephen Hemminger

Add a comprehensive test suite for the rtap poll mode driver covering:

- Device creation and configuration
- Device info query
- Link status and link up/down control
- Statistics collection and reset
- Promiscuous and all-multicast mode toggling
- MAC address and MTU configuration
- Multi-queue setup, teardown, and queue count reduction
- Mismatched rx/tx queue count rejection
- Queue reconfiguration (stop, reconfigure, restart)
- Rx path: inject packet via AF_PACKET, receive via DPDK
- Tx path: transmit via DPDK, capture via AF_PACKET
- Multi-segment Tx with chained mbufs
- Multi-segment Rx with small-buffer mempool
- Offload capability reporting and configuration
- Tx UDP checksum offload end-to-end verification
- File descriptor leak detection after device close
- Command-line created device testing

Tests require CAP_NET_ADMIN or root privileges

---
 app/test/meson.build     |    1 +
 app/test/test_pmd_rtap.c | 2044 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 2045 insertions(+)
 create mode 100644 app/test/test_pmd_rtap.c
app/test/meson.build +1 -0
index 48874037eb..5855077066 100644
@@ -143,6 +143,7 @@ source_file_deps = {
     'test_pie.c': ['sched'],
     'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps,
     'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'],
+    'test_pmd_rtap.c': ['net_rtap', 'ethdev', 'bus_vdev'],
     'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'],
     'test_pmu.c': ['pmu'],
     'test_power.c': ['power', 'power_acpi', 'power_kvm_vm', 'power_intel_pstate',
app/test/test_pmd_rtap.c new file +2044 -0

2,044 lines not on this page · raw message

Signature · 1 line
2.51.0
#57 lore raw

RE: [PATCH v5 00/10] net/rtap: add io_uring based TAP driver

In reply to Stephen Hemminger
Quoted 6 lines

This series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.

Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.

Robin Jarry recently added Network Namespace support to the TAP driver [1]. Does the rtap driver support Network Namespaces?

[1]: https://inbox.dpdk.org/dev/[email protected]/

Quoted 2 lines

Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)

The DPDK 25.11 minimum requirement [2] is Kernel 5.4.
Since this driver requires a higher version, it should be prominently mentioned in relevant documentation.

In December 2026, shortly after the release of DPDK 26.11, the oldest supported Kernel LTS version will be 6.1. For now, it is 5.10.

[2]: https://doc.dpdk.org/guides-25.11/linux_gsg/sys_reqs.html#system-software

Quoted 5 lines

- liburing >= 2.0

Known working distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).

#58 lore raw

v6 cover net/rtap: add io_uring based TAP driver

In reply to Stephen Hemminger

This series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.

Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.

Motivation
----------

This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.

The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.

Why not extend net_tap?
-----------------------

The existing net_tap driver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity: rte_flow support emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.

net_rtap takes the opposite approach -- use the kernel efficiently and let it do what it does well. There is no rte_flow support; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.

Given these fundamentally different design goals, a clean implementation was more practical than refactoring net_tap.

Acknowledgement
---------------

Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.

Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)
- liburing >= 2.0

Should work on distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).

v6:
- fix lots of bugs found doing automated review
- convert to use ifindex and netlink to avoid rename issues
- implement xstats

Stephen Hemminger (11):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add multi-process support
net/rtap: add link state change interrupt net/rtap: add Rx interrupt support
net/rtap: add extended statistics support
test: add unit tests for rtap PMD

 MAINTAINERS                            |    7 +
 app/test/meson.build                   |    1 +
 app/test/test_pmd_rtap.c               | 2620 ++++++++++++++++++++++++
 doc/guides/nics/features/rtap.ini      |   26 +
 doc/guides/nics/index.rst              |    1 +
 doc/guides/nics/rtap.rst               |  101 +
 doc/guides/rel_notes/release_26_03.rst |    7 +
 drivers/net/meson.build                |    1 +
 drivers/net/rtap/meson.build           |   30 +
 drivers/net/rtap/rtap.h                |  152 ++
 drivers/net/rtap/rtap_ethdev.c         |  864 ++++++++
 drivers/net/rtap/rtap_intr.c           |  207 ++
 drivers/net/rtap/rtap_netlink.c        |  445 ++++
 drivers/net/rtap/rtap_rxtx.c           |  803 ++++++++
 drivers/net/rtap/rtap_xstats.c         |  293 +++
 15 files changed, 5558 insertions(+)
 create mode 100644 app/test/test_pmd_rtap.c
 create mode 100644 doc/guides/nics/features/rtap.ini
 create mode 100644 doc/guides/nics/rtap.rst
 create mode 100644 drivers/net/rtap/meson.build
 create mode 100644 drivers/net/rtap/rtap.h
 create mode 100644 drivers/net/rtap/rtap_ethdev.c
 create mode 100644 drivers/net/rtap/rtap_intr.c
 create mode 100644 drivers/net/rtap/rtap_netlink.c
 create mode 100644 drivers/net/rtap/rtap_rxtx.c
 create mode 100644 drivers/net/rtap/rtap_xstats.c
Signature · 1 line
2.51.0
#59 lore raw

PATCH v6 1/11 net/rtap: add driver skeleton and documentation

Add the initial skeleton for the rtap poll mode driver, a virtual ethernet device that uses Linux io_uring for packet I/O with kernel TAP devices.

This patch includes:
- MAINTAINERS entry
- Driver documentation (doc/guides/nics/rtap.rst)
- Feature matrix (doc/guides/nics/features/rtap.ini)
- Release notes update
- Meson build integration with liburing dependency
- Header file with shared data structures and declarations
- Stub probe/remove handlers that register the vdev driver
- Empty dev_ops with only dev_close implemented

The driver registers as net_rtap and is Linux-only.

Requires the liburing library version 2.0 or later. Earlier versions have known security and build issues.

---
 MAINTAINERS                            |   7 +
 doc/guides/nics/features/rtap.ini      |  13 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/nics/rtap.rst               | 101 ++++++++++++++
 doc/guides/rel_notes/release_26_03.rst |   7 +
 drivers/net/meson.build                |   1 +
 drivers/net/rtap/meson.build           |  26 ++++
 drivers/net/rtap/rtap.h                |  81 +++++++++++
 drivers/net/rtap/rtap_ethdev.c         | 177 +++++++++++++++++++++++++
 9 files changed, 414 insertions(+)
 create mode 100644 doc/guides/nics/features/rtap.ini
 create mode 100644 doc/guides/nics/rtap.rst
 create mode 100644 drivers/net/rtap/meson.build
 create mode 100644 drivers/net/rtap/rtap.h
 create mode 100644 drivers/net/rtap/rtap_ethdev.c
MAINTAINERS +7 -0
index 25fb109ef4..45721c9d03 100644
@@ -1135,6 +1135,13 @@ F: doc/guides/nics/ring.rst
 F: app/test/test_pmd_ring.c
 F: app/test/test_pmd_ring_perf.c
 
+Rtap PMD - EXPERIMENTAL
+M: Stephen Hemminger <[email protected]>
+F: drivers/net/rtap/
+F: app/test/test_pmd_rtap.c
+F: doc/guides/nics/rtap.rst
+F: doc/guides/nics/features/rtap.ini
+
 Null Networking PMD
 M: Tetsuya Mukawa <[email protected]>
 F: drivers/net/null/
doc/guides/nics/features/rtap.ini new file +13 -0
new file mode 100644
index 0000000000..ed7c638029
@@ -0,0 +1,13 @@
+;
+; Supported features of the 'rtap' driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+ARMv7                = Y
+ARMv8                = Y
+Power8               = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/index.rst +1 -0
index cb818284fe..24746596b7 100644
@@ -66,6 +66,7 @@ Network Interface Controller Drivers
     r8169
     ring
     rnp
+    rtap
     sfc_efx
     softnic
     tap
doc/guides/nics/rtap.rst new file +101 -0

101 lines not on this page · raw message

doc/guides/rel_notes/release_26_03.rst +7 -0
index afdf1af06c..40320b0101 100644
@@ -87,6 +87,13 @@ New Features
   * Added support for AES-XTS cipher algorithm.
   * Added support for SHAKE-128 and SHAKE-256 authentication algorithms.
 
+* **Added rtap virtual ethernet driver.**
+
+  Added a new experimental virtual device driver that uses Linux io_uring
+  for packet injection into the kernel network stack.
+  It requires Linux kernel 5.19 or later for IORING_ASYNC_CANCEL
+  and liburing 2.0 or later.
+
 
 Removed Items
 -------------
drivers/net/meson.build +1 -0
index c7dae4ad27..ef1ee68385 100644
@@ -56,6 +56,7 @@ drivers = [
         'r8169',
         'ring',
         'rnp',
+        'rtap',
         'sfc',
         'softnic',
         'tap',
drivers/net/rtap/meson.build new file +26 -0
new file mode 100644
index 0000000000..7bd7806ef3
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Stephen Hemminger
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+endif
+
+liburing = dependency('liburing', version: '>= 2.0', required: false)
+if not liburing.found()
+    build = false
+    reason = 'missing dependency, "liburing"'
+endif
+
+if build and not cc.has_header_symbol('linux/io_uring.h', 'IORING_ASYNC_CANCEL_ALL')
+    build = false
+    reason = 'kernel headers missing IORING_ASYNC_CANCEL_ALL (need kernel >= 5.19 headers)'
+endif
+
+sources = files(
+        'rtap_ethdev.c',
+)
+
+ext_deps += liburing
+
+require_iova_in_mbuf = false
drivers/net/rtap/rtap.h new file +81 -0

81 lines not on this page · raw message

drivers/net/rtap/rtap_ethdev.c new file +177 -0

177 lines not on this page · raw message

Signature · 1 line
2.51.0
#60 lore raw

PATCH v6 2/11 net/rtap: add TAP device creation and queue management

In reply to Stephen Hemminger

Add TAP device creation using /dev/net/tun with IFF_MULTI_QUEUE, IFF_NO_PI, IFF_VNET_HDR, and optional IFF_NAPI flags.

The driver maintains a keep-alive fd and opens additional per-queue fds for I/O. Each queue pair (rx+tx) shares a single TAP fd.

Key operations:
- rtap_create(): Opens TAP, gets stable ifindex, opens netlink socket, retrieves MAC via netlink, detaches keep-alive queue
- rtap_queue_open/close(): Per-queue fd management (converts ifindex to name for TUNSETIFF ioctl)
- rtap_dev_configure(): Validates paired queues, clears offloads
- rtap_dev_start/stop(): Manages link status and queue states

The driver uses netlink (RTM_GETLINK/RTM_NEWLINK) and ifindex for interface control rather than ioctl() and interface names. This avoids issues with interface renames and namespace moves.

---
 drivers/net/rtap/meson.build    |   1 +
 drivers/net/rtap/rtap.h         |   4 +
 drivers/net/rtap/rtap_ethdev.c  | 241 ++++++++++++++++-
 drivers/net/rtap/rtap_netlink.c | 445 ++++++++++++++++++++++++++++++++
 4 files changed, 689 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/rtap/rtap_netlink.c
drivers/net/rtap/meson.build +1 -0
index 7bd7806ef3..1a24ea0555 100644
@@ -19,6 +19,7 @@ endif
 
 sources = files(
         'rtap_ethdev.c',
+        'rtap_netlink.c',
 )
 
 ext_deps += liburing
drivers/net/rtap/rtap.h +4 -0
index 9004953e04..a2d1149cac 100644
@@ -64,6 +64,10 @@ struct rtap_pmd {
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 };
 
+/* rtap_ethdev.c */
+int rtap_queue_open(struct rte_eth_dev *dev, uint16_t queue_id);
+void rtap_queue_close(struct rte_eth_dev *dev, uint16_t queue_id);
+
 /* rtap_netlink.c */
 int rtap_nl_open(unsigned int groups);
 struct rte_eth_dev;
drivers/net/rtap/rtap_ethdev.c +239 -2

276 lines not on this page · raw message

drivers/net/rtap/rtap_netlink.c new file +445 -0

445 lines not on this page · raw message

Signature · 1 line
2.51.0
#61 lore raw

PATCH v6 3/11 net/rtap: add Rx/Tx with scatter/gather support

In reply to Stephen Hemminger

Implement packet receive and transmit using io_uring asynchronous I/O, with full support for both single-segment and multi-segment mbufs.

Rx path:
- rtap_rx_alloc() chains multiple mbufs when the MTU exceeds a single mbuf's tailroom capacity
- Pre-post read/readv requests to the io_uring submission queue, each backed by a pre-allocated (possibly chained) mbuf
- On rx_burst, harvest completed CQEs and replace each consumed mbuf with a freshly allocated one
- rtap_rx_adjust() distributes received data across segments and frees unused trailing segments
- Parse the prepended virtio-net header

Tx path:
- For single-segment mbufs, use io_uring write and batch submits
- For multi-segment mbufs, use writev via io_uring with immediate submit (iovec is stack-allocated)
- When the mbuf headroom is not writable (shared or indirect), chain a new header mbuf for the virtio-net header
- Prepend a zeroed virtio-net header
- Clean completed Tx CQEs to free transmitted mbufs

Add io_uring cancel-all logic using IORING_ASYNC_CANCEL_ALL for clean queue teardown, draining all pending CQEs and freeing mbufs.

Full offload configuration is not enabled until later.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/meson.build      |   1 +
 drivers/net/rtap/rtap.h           |  13 +
 drivers/net/rtap/rtap_ethdev.c    |   7 +
 drivers/net/rtap/rtap_rxtx.c      | 771 ++++++++++++++++++++++++++++++
 5 files changed, 793 insertions(+)
 create mode 100644 drivers/net/rtap/rtap_rxtx.c
doc/guides/nics/features/rtap.ini +1 -0
index ed7c638029..c064e1e0b9 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Scattered Rx         = P
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/meson.build +1 -0
index 1a24ea0555..ed0bcc1313 100644
@@ -20,6 +20,7 @@ endif
 sources = files(
         'rtap_ethdev.c',
         'rtap_netlink.c',
+        'rtap_rxtx.c',
 )
 
 ext_deps += liburing
drivers/net/rtap/rtap.h +13 -0
index a2d1149cac..823b8c59f7 100644
@@ -82,4 +82,17 @@ int rtap_nl_get_mac(int nlsk_fd, int if_index, struct rte_ether_addr *addr);
 struct rtnl_link_stats64;
 int rtap_nl_get_stats(int if_index, struct rtnl_link_stats64 *stats);
 
+/* rtap_rxtx.c */
+uint16_t rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts);
+uint16_t rtap_tx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts);
+int rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+			uint16_t nb_rx_desc, unsigned int socket_id,
+			const struct rte_eth_rxconf *rx_conf,
+			struct rte_mempool *mb_pool);
+void rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
+			uint16_t nb_tx_desc, unsigned int socket_id,
+			const struct rte_eth_txconf *tx_conf);
+void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
+
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +7 -0
index 0eab0a48fa..463e24a6e1 100644
@@ -248,6 +248,10 @@ static const struct eth_dev_ops rtap_ops = {
 	.dev_stop		= rtap_dev_stop,
 	.dev_configure		= rtap_dev_configure,
 	.dev_close		= rtap_dev_close,
+	.rx_queue_setup		= rtap_rx_queue_setup,
+	.rx_queue_release	= rtap_rx_queue_release,
+	.tx_queue_setup		= rtap_tx_queue_setup,
+	.tx_queue_release	= rtap_tx_queue_release,
 };
 
 static int
@@ -298,6 +302,9 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 
 	PMD_LOG(DEBUG, "ifindex %d setup", pmd->if_index);
 
+	dev->rx_pkt_burst = rtap_rx_burst;
+	dev->tx_pkt_burst = rtap_tx_burst;
+
 	return 0;
 
 error:
drivers/net/rtap/rtap_rxtx.c new file +771 -0

771 lines not on this page · raw message

Signature · 1 line
2.51.0
#62 lore raw

PATCH v6 4/11 net/rtap: add statistics and device info

In reply to Stephen Hemminger

Implement basic and per-queue statistics collection. Usual statistics per queue: packets, bytes, errors.

The imissed counter reports kernel rx_dropped (packets dropped before reaching the interface) retrieved via netlink IFLA_STATS64. The rx_drop_base field captures the baseline value at stats_reset.

Device info reports ifindex, max queue counts, and default burst/ring sizes.

---
 doc/guides/nics/features/rtap.ini |   2 +
 drivers/net/rtap/rtap.h           |   2 +
 drivers/net/rtap/rtap_ethdev.c    | 120 ++++++++++++++++++++++++++++++
 3 files changed, 124 insertions(+)
doc/guides/nics/features/rtap.ini +2 -0
index c064e1e0b9..9bef9e341d 100644
@@ -5,6 +5,8 @@
 ;
 [Features]
 Scattered Rx         = P
+Basic stats          = Y
+Stats per queue      = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap.h +2 -0
index 823b8c59f7..abfde20e60 100644
@@ -62,6 +62,8 @@ struct rtap_pmd {
 	int if_index;			/* interface index */
 	int nlsk_fd;			/* netlink control socket */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
+
+	uint64_t rx_drop_base;		/* value of rx_dropped when reset */
 };
 
 /* rtap_ethdev.c */
drivers/net/rtap/rtap_ethdev.c +120 -0

157 lines not on this page · raw message

Signature · 1 line
2.51.0
#63 lore raw

PATCH v6 5/11 net/rtap: add link and device management operations

In reply to Stephen Hemminger

Add device management operations via netlink:
- Link up/down control
- Link status query (checks IFF_UP and IFF_RUNNING)
- Promiscuous mode enable/disable
- All-multicast mode enable/disable
- MTU configuration
- MAC address configuration

All operations use the persistent netlink socket (pmd->nlsk_fd) and ifindex for stable interface identification across renames and namespace moves.

---
 doc/guides/nics/features/rtap.ini |   4 ++
 drivers/net/rtap/rtap.h           |   1 +
 drivers/net/rtap/rtap_ethdev.c    | 102 ++++++++++++++++++++++++++++++
 3 files changed, 107 insertions(+)
doc/guides/nics/features/rtap.ini +4 -0
index 9bef9e341d..ce0804d795 100644
@@ -4,6 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
+MTU update           = Y
+Promiscuous mode     = Y
+Allmulticast mode    = Y
 Scattered Rx         = P
 Basic stats          = Y
 Stats per queue      = Y
drivers/net/rtap/rtap.h +1 -0
index abfde20e60..aeda7f3f25 100644
@@ -69,6 +69,7 @@ struct rtap_pmd {
 /* rtap_ethdev.c */
 int rtap_queue_open(struct rte_eth_dev *dev, uint16_t queue_id);
 void rtap_queue_close(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_link_update(struct rte_eth_dev *dev, int wait_to_complete);
 
 /* rtap_netlink.c */
 int rtap_nl_open(unsigned int groups);
drivers/net/rtap/rtap_ethdev.c +102 -0

123 lines not on this page · raw message

Signature · 1 line
2.51.0
#64 lore raw

PATCH v6 6/11 net/rtap: add checksum and TSO offload support

In reply to Stephen Hemminger

Add transmit and receive offload support using the virtio-net header that is exchanged with the kernel TAP device.

Tx offloads:
- UDP and TCP checksum offload: set VIRTIO_NET_HDR_F_NEEDS_CSUM with appropriate csum_start and csum_offset fields
- TCP segmentation offload (TSO): set gso_type and gso_size

Rx offloads:
- Enable receive offloads based on configuration

Report offload capabilities in dev_info.

---
 doc/guides/nics/features/rtap.ini |  2 ++
 drivers/net/rtap/rtap_ethdev.c    | 30 +++++++++++++++++++++++++++++-
 2 files changed, 31 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +2 -0
index ce0804d795..b8eaa805fe 100644
@@ -11,6 +11,8 @@ Allmulticast mode    = Y
 Scattered Rx         = P
 Basic stats          = Y
 Stats per queue      = Y
+TSO                  = Y
+L4 checksum offload  = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap_ethdev.c +29 -1
index b6c6421e3d..c4125e04b3 100644
@@ -31,6 +31,16 @@
 
 #define RTAP_DEFAULT_IFNAME	"rtap%d"
 
+#define RTAP_TX_OFFLOAD		(RTE_ETH_TX_OFFLOAD_MULTI_SEGS | \
+				 RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \
+				 RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \
+				 RTE_ETH_TX_OFFLOAD_TCP_TSO)
+
+#define RTAP_RX_OFFLOAD		(RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \
+				 RTE_ETH_RX_OFFLOAD_TCP_LRO | \
+				 RTE_ETH_RX_OFFLOAD_SCATTER)
+
 #define RTAP_DEFAULT_BURST	64
 #define RTAP_NUM_BUFFERS	1024
 #define RTAP_MAX_QUEUES		128
@@ -250,7 +260,21 @@ rtap_dev_configure(struct rte_eth_dev *dev)
 		return -EINVAL;
 	}
 
-	if (ioctl(pmd->keep_fd, TUNSETOFFLOAD, 0) != 0) {
+	/*
+	 * Set offload flags visible on the kernel network interface.
+	 * This controls whether kernel will use checksum offload etc.
+	 * Note: kernel transmit is DPDK receive.
+	 */
+	const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode;
+	unsigned int offload = 0;
+	if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_CHECKSUM) {
+		offload |= TUN_F_CSUM;
+
+		if (rx_mode->offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO)
+			offload |= TUN_F_TSO4 | TUN_F_TSO6 | TUN_F_TSO_ECN;
+	}
+
+	if (ioctl(pmd->keep_fd, TUNSETOFFLOAD, offload) != 0) {
 		int ret = -errno;
 
 		PMD_LOG(ERR, "ioctl(TUNSETOFFLOAD) failed: %s", strerror(errno));
@@ -271,6 +295,10 @@ rtap_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_rx_bufsize = RTAP_MIN_RX_BUFSIZE;
 	dev_info->max_rx_queues = RTAP_MAX_QUEUES;
 	dev_info->max_tx_queues = RTAP_MAX_QUEUES;
+	dev_info->rx_queue_offload_capa = RTAP_RX_OFFLOAD;
+	dev_info->rx_offload_capa = dev_info->rx_queue_offload_capa;
+	dev_info->tx_queue_offload_capa = RTAP_TX_OFFLOAD;
+	dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa;
 
 	dev_info->default_rxportconf = (struct rte_eth_dev_portconf) {
 		.burst_size = RTAP_DEFAULT_BURST,
Signature · 1 line
2.51.0
#65 lore raw

PATCH v6 7/11 net/rtap: add multi-process support

In reply to Stephen Hemminger

Add support for DPDK multi-process operation. Secondary processes need access to the per-queue TAP file descriptors owned by the primary process.

Implement fd sharing using the DPDK multi-process IPC mechanism:
- Primary registers an MP action handler that responds to fd requests by sending queue fds via rte_mp_reply()
- Secondary process attaches to the existing device and requests fds from primary via rte_mp_request_sync()

The MP action is registered on the first device probe and unregistered when the last device is closed.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/rtap_ethdev.c    | 140 ++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
doc/guides/nics/features/rtap.ini +1 -0
index b8eaa805fe..cfbd29ef08 100644
@@ -13,6 +13,7 @@ Basic stats          = Y
 Stats per queue      = Y
 TSO                  = Y
 L4 checksum offload  = Y
+Multiprocess aware   = Y
 Linux                = Y
 ARMv7                = Y
 ARMv8                = Y
drivers/net/rtap/rtap_ethdev.c +140 -0

192 lines not on this page · raw message

Signature · 1 line
2.51.0
#66 lore raw

PATCH v6 8/11 net/rtap: add link state change interrupt

In reply to Stephen Hemminger

Add link state change (LSC) notification support using a netlink socket subscribed to RTMGRP_LINK events.

When enabled, the driver registers a netlink socket with the EAL interrupt handler. On receiving RTM_NEWLINK/RTM_DELLINK messages matching the interface ifindex, the driver updates link status and triggers RTE_ETH_EVENT_INTR_LSC callbacks.

LSC is enabled in dev_start and disabled in dev_stop.

---
 doc/guides/nics/features/rtap.ini |  1 +
 drivers/net/rtap/meson.build      |  1 +
 drivers/net/rtap/rtap.h           |  4 ++
 drivers/net/rtap/rtap_ethdev.c    | 26 ++++++++-
 drivers/net/rtap/rtap_intr.c      | 87 +++++++++++++++++++++++++++++++
 5 files changed, 118 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/rtap/rtap_intr.c
doc/guides/nics/features/rtap.ini +1 -0
index cfbd29ef08..fe0c88a8fc 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+Link status event    = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
drivers/net/rtap/meson.build +1 -0
index ed0bcc1313..58943e035a 100644
@@ -19,6 +19,7 @@ endif
 
 sources = files(
         'rtap_ethdev.c',
+        'rtap_intr.c',
         'rtap_netlink.c',
         'rtap_rxtx.c',
 )
drivers/net/rtap/rtap.h +4 -0
index aeda7f3f25..2cc55c9667 100644
@@ -61,6 +61,7 @@ struct rtap_pmd {
 	int keep_fd;			/* keep alive file descriptor */
 	int if_index;			/* interface index */
 	int nlsk_fd;			/* netlink control socket */
+	struct rte_intr_handle *intr_handle; /* LSC interrupt handle */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 
 	uint64_t rx_drop_base;		/* value of rx_dropped when reset */
@@ -98,4 +99,7 @@ int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 			const struct rte_eth_txconf *tx_conf);
 void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
 
+/* rtap_intr.c */
+int rtap_lsc_set(struct rte_eth_dev *dev, int set);
+
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +25 -1
index 705d98323e..9c23ca2f16 100644
@@ -214,11 +214,18 @@ rtap_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
 static int
 rtap_dev_start(struct rte_eth_dev *dev)
 {
-	int ret = rtap_set_link_up(dev);
+	int ret;
 
+	ret = rtap_lsc_set(dev, 1);
 	if (ret != 0)
 		return ret;
 
+	ret = rtap_set_link_up(dev);
+	if (ret != 0) {
+		rtap_lsc_set(dev, 0);
+		return ret;
+	}
+
 	dev->data->dev_link.link_status = RTE_ETH_LINK_UP;
 	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
 		dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STARTED;
@@ -235,6 +242,7 @@ rtap_dev_stop(struct rte_eth_dev *dev)
 
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
 
+	rtap_lsc_set(dev, 0);
 	rtap_set_link_down(dev);
 
 	for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
@@ -435,6 +443,9 @@ rtap_dev_close(struct rte_eth_dev *dev)
 			close(pmd->nlsk_fd);
 			pmd->nlsk_fd = -1;
 		}
+
+		rte_intr_instance_free(pmd->intr_handle);
+		pmd->intr_handle = NULL;
 	}
 
 	free(dev->process_private);
@@ -522,6 +533,17 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 	pmd->nlsk_fd = -1;
 	pmd->rx_drop_base = 0;
 
+	/* Allocate interrupt instance for link state change events */
+	pmd->intr_handle = rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_SHARED);
+	if (pmd->intr_handle == NULL) {
+		PMD_LOG(ERR, "Failed to allocate intr handle");
+		goto error;
+	}
+	rte_intr_type_set(pmd->intr_handle, RTE_INTR_HANDLE_EXT);
+	rte_intr_fd_set(pmd->intr_handle, -1);
+	dev->intr_handle = pmd->intr_handle;
+	data->dev_flags |= RTE_ETH_DEV_INTR_LSC;
+
 	dev->dev_ops = &rtap_ops;
 
 	/* Get the initial fd used to keep the tap device around */
@@ -571,6 +593,8 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)
 		close(pmd->nlsk_fd);
 	if (pmd->keep_fd != -1)
 		close(pmd->keep_fd);
+	rte_intr_instance_free(pmd->intr_handle);
+	pmd->intr_handle = NULL;
 	return -1;
 }
 
drivers/net/rtap/rtap_intr.c new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
2.51.0
#67 lore raw

PATCH v6 9/11 net/rtap: add Rx interrupt support

In reply to Stephen Hemminger

Add support for the DPDK Rx interrupt mechanism, enabling power-aware applications (e.g. l3fwd-power) to sleep until packets arrive rather than busy-polling.

Each Rx queue creates an eventfd during queue setup and registers it with its io_uring instance via io_uring_register_eventfd(). When the kernel posts a CQE (completing a read, i.e. a packet arrived), it signals the eventfd. These per-queue eventfds are wired into a VDEV interrupt handle during dev_start when the application has set intr_conf.rxq.

The enable op drains the eventfd counter to re-arm notification; disable is a no-op since the application simply stops polling. The eventfd is always created unconditionally so it is available if the application enables Rx interrupts later.

The Rx interrupt handle is kept separate from the existing LSC netlink interrupt handle to avoid coupling the two mechanisms.

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/rtap.h           |   6 ++
 drivers/net/rtap/rtap_ethdev.c    |  26 +++++++
 drivers/net/rtap/rtap_intr.c      | 120 ++++++++++++++++++++++++++++++
 drivers/net/rtap/rtap_rxtx.c      |  31 +++++++-
 5 files changed, 183 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +1 -0
index fe0c88a8fc..48fe3f1b33 100644
@@ -6,6 +6,7 @@
 [Features]
 Link status          = Y
 Link status event    = Y
+Rx interrupt         = Y
 MTU update           = Y
 Promiscuous mode     = Y
 Allmulticast mode    = Y
drivers/net/rtap/rtap.h +6 -0
index 2cc55c9667..2c17117a80 100644
@@ -38,6 +38,7 @@ extern int rtap_logtype;
 struct rtap_rx_queue {
 	struct rte_mempool *mb_pool;	/* rx buffer pool */
 	struct io_uring io_ring;	/* queue of posted read's */
+	int intr_fd;			/* eventfd for Rx interrupt */
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -62,6 +63,7 @@ struct rtap_pmd {
 	int if_index;			/* interface index */
 	int nlsk_fd;			/* netlink control socket */
 	struct rte_intr_handle *intr_handle; /* LSC interrupt handle */
+	struct rte_intr_handle *rx_intr_handle; /* Rx queue interrupt handle */
 	struct rte_ether_addr eth_addr; /* address assigned by kernel */
 
 	uint64_t rx_drop_base;		/* value of rx_dropped when reset */
@@ -101,5 +103,9 @@ void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);
 
 /* rtap_intr.c */
 int rtap_lsc_set(struct rte_eth_dev *dev, int set);
+int rtap_rx_intr_vec_install(struct rte_eth_dev *dev);
+void rtap_rx_intr_vec_uninstall(struct rte_eth_dev *dev);
+int rtap_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id);
+int rtap_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id);
 
 #endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +26 -0
index 9c23ca2f16..2cbb66b675 100644
@@ -220,8 +220,18 @@ rtap_dev_start(struct rte_eth_dev *dev)
 	if (ret != 0)
 		return ret;
 
+	/* Install Rx interrupt vector if requested by application */
+	if (dev->data->dev_conf.intr_conf.rxq) {
+		ret = rtap_rx_intr_vec_install(dev);
+		if (ret != 0) {
+			rtap_lsc_set(dev, 0);
+			return ret;
+		}
+	}
+
 	ret = rtap_set_link_up(dev);
 	if (ret != 0) {
+		rtap_rx_intr_vec_uninstall(dev);
 		rtap_lsc_set(dev, 0);
 		return ret;
 	}
@@ -242,6 +252,7 @@ rtap_dev_stop(struct rte_eth_dev *dev)
 
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
 
+	rtap_rx_intr_vec_uninstall(dev);
 	rtap_lsc_set(dev, 0);
 	rtap_set_link_down(dev);
 
@@ -275,6 +286,16 @@ rtap_dev_configure(struct rte_eth_dev *dev)
 		return -EINVAL;
 	}
 
+	/*
+	 * LSC and Rx queue interrupts both need dev->intr_handle,
+	 * so they cannot be enabled simultaneously.
+	 */
+	if (dev->data->dev_conf.intr_conf.lsc &&
+	    dev->data->dev_conf.intr_conf.rxq) {
+		PMD_LOG(ERR, "LSC and Rx queue interrupts are mutually exclusive");
+		return -ENOTSUP;
+	}
+
 	/*
 	 * Set offload flags visible on the kernel network interface.
 	 * This controls whether kernel will use checksum offload etc.
@@ -444,6 +465,9 @@ rtap_dev_close(struct rte_eth_dev *dev)
 			pmd->nlsk_fd = -1;
 		}
 
+		rte_intr_instance_free(pmd->rx_intr_handle);
+		pmd->rx_intr_handle = NULL;
+
 		rte_intr_instance_free(pmd->intr_handle);
 		pmd->intr_handle = NULL;
 	}
@@ -521,6 +545,8 @@ static const struct eth_dev_ops rtap_ops = {
 	.rx_queue_release	= rtap_rx_queue_release,
 	.tx_queue_setup		= rtap_tx_queue_setup,
 	.tx_queue_release	= rtap_tx_queue_release,
+	.rx_queue_intr_enable	= rtap_rx_queue_intr_enable,
+	.rx_queue_intr_disable	= rtap_rx_queue_intr_disable,
 };
 
 static int
drivers/net/rtap/rtap_intr.c +120 -0

123 lines not on this page · raw message

drivers/net/rtap/rtap_rxtx.c +30 -1
index 7826169751..cd9b4f0bac 100644
@@ -10,6 +10,7 @@
 #include <stddef.h>
 #include <string.h>
 #include <liburing.h>
+#include <sys/eventfd.h>
 #include <sys/types.h>
 #include <sys/uio.h>
 #include <linux/virtio_net.h>
@@ -437,6 +438,7 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 	rxq->mb_pool = mb_pool;
 	rxq->port_id = dev->data->port_id;
 	rxq->queue_id = queue_id;
+	rxq->intr_fd = -1;
 	dev->data->rx_queues[queue_id] = rxq;
 
 	if (io_uring_queue_init(nb_rx_desc, &rxq->io_ring, 0) != 0) {
@@ -444,10 +446,26 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 		goto error_rxq_free;
 	}
 
+	/*
+	 * Create an eventfd for Rx interrupt notification.
+	 * io_uring will signal this fd whenever a CQE is posted,
+	 * enabling power-aware applications to sleep until packets arrive.
+	 */
+	rxq->intr_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+	if (rxq->intr_fd < 0) {
+		PMD_LOG(ERR, "eventfd failed: %s", strerror(errno));
+		goto error_iouring_exit;
+	}
+
+	if (io_uring_register_eventfd(&rxq->io_ring, rxq->intr_fd) < 0) {
+		PMD_LOG(ERR, "io_uring_register_eventfd failed: %s", strerror(errno));
+		goto error_eventfd_close;
+	}
+
 	mbufs = calloc(nb_rx_desc, sizeof(struct rte_mbuf *));
 	if (mbufs == NULL) {
 		PMD_LOG(ERR, "Rx mbuf pointer alloc failed");
-		goto error_iouring_exit;
+		goto error_eventfd_close;
 	}
 
 	/* open shared tap fd maybe already setup */
@@ -494,6 +512,11 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d
 	rtap_cancel_all(&rxq->io_ring);
 	rtap_queue_close(dev, queue_id);
 	free(mbufs);
+error_eventfd_close:
+	if (rxq->intr_fd >= 0) {
+		close(rxq->intr_fd);
+		rxq->intr_fd = -1;
+	}
 error_iouring_exit:
 	io_uring_queue_exit(&rxq->io_ring);
 error_rxq_free:
@@ -509,6 +532,12 @@ rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)
 	if (rxq == NULL)
 		return;
 
+	if (rxq->intr_fd >= 0) {
+		io_uring_unregister_eventfd(&rxq->io_ring);
+		close(rxq->intr_fd);
+		rxq->intr_fd = -1;
+	}
+
 	rtap_cancel_all(&rxq->io_ring);
 	io_uring_queue_exit(&rxq->io_ring);
 
Signature · 1 line
2.51.0
#68 lore raw

PATCH v6 10/11 net/rtap: add extended statistics support

In reply to Stephen Hemminger

Add xstats support to rtap PMD following the virtio PMD design pattern.

Statistics provided per queue (14 Rx + 12 Tx):
- Packet size distribution (6 buckets: 64, 65-127, 128-255, 256-511, 512-1023, 1024-1518 bytes)
- Packet type classification (broadcast, multicast, unicast)
- Rx offload stats (LRO, checksum validation, mbuf alloc failures)
- Tx offload stats (TSO, checksum offload, multi-segment packets)

Statistics follow DPDK naming convention:
{direction}_q{queue_id}_{detail}_{unit}

Examples:
rx_q0_size_64_packets
rx_q0_broadcast_packets
rx_q0_lro_packets
tx_q0_tso_packets
tx_q0_checksum_offload_packets

---
 doc/guides/nics/features/rtap.ini |   1 +
 drivers/net/rtap/meson.build      |   1 +
 drivers/net/rtap/rtap.h           |  41 +++++
 drivers/net/rtap/rtap_ethdev.c    |   3 +
 drivers/net/rtap/rtap_rxtx.c      |   3 +
 drivers/net/rtap/rtap_xstats.c    | 293 ++++++++++++++++++++++++++++++
 6 files changed, 342 insertions(+)
 create mode 100644 drivers/net/rtap/rtap_xstats.c
doc/guides/nics/features/rtap.ini +1 -0
index 48fe3f1b33..233227e7d4 100644
@@ -15,6 +15,7 @@ Basic stats          = Y
 Stats per queue      = Y
 TSO                  = Y
 L4 checksum offload  = Y
+Extended stats       = P
 Multiprocess aware   = Y
 Linux                = Y
 ARMv7                = Y
drivers/net/rtap/meson.build +1 -0
index 58943e035a..835d1e557d 100644
@@ -22,6 +22,7 @@ sources = files(
         'rtap_intr.c',
         'rtap_netlink.c',
         'rtap_rxtx.c',
+        'rtap_xstats.c',
 )
 
 ext_deps += liburing
drivers/net/rtap/rtap.h +41 -0

65 lines not on this page · raw message

drivers/net/rtap/rtap_ethdev.c +3 -0
index 2cbb66b675..eb581608cc 100644
@@ -541,6 +541,9 @@ static const struct eth_dev_ops rtap_ops = {
 	.allmulticast_disable	= rtap_allmulticast_disable,
 	.stats_get		= rtap_stats_get,
 	.stats_reset		= rtap_stats_reset,
+	.xstats_get		= rtap_xstats_get,
+	.xstats_get_names	= rtap_xstats_get_names,
+	.xstats_reset		= rtap_xstats_reset,
 	.rx_queue_setup		= rtap_rx_queue_setup,
 	.rx_queue_release	= rtap_rx_queue_release,
 	.tx_queue_setup		= rtap_tx_queue_setup,
drivers/net/rtap/rtap_rxtx.c +3 -0
index cd9b4f0bac..f0ea53b8cf 100644
@@ -289,6 +289,7 @@ rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 			PMD_RX_LOG(ERR, "Rx mbuf alloc failed");
 			dev->data->rx_mbuf_alloc_failed++;
+			rxq->xstats.mbuf_alloc_failed++;
 
 			nmb = mb;	 /* Reuse original */
 			goto resubmit;
@@ -317,6 +318,7 @@ rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 		mb->port = rxq->port_id;
 
 		__rte_mbuf_sanity_check(mb, 1);
+		rtap_rx_xstats_update(rxq, mb);
 		num_bytes += mb->pkt_len;
 		bufs[num_rx++] = mb;
 
@@ -735,6 +737,7 @@ rtap_tx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
 
 		io_uring_sqe_set_data(sqe, mb);
 		rtap_tx_offload(hdr, mb);
+		rtap_tx_xstats_update(txq, mb);
 
 		PMD_TX_LOG(DEBUG, "write m=%p segs=%u", mb, mb->nb_segs);
 
drivers/net/rtap/rtap_xstats.c new file +293 -0

293 lines not on this page · raw message

Signature · 1 line
2.51.0
#69 lore raw

PATCH v6 11/11 test: add unit tests for rtap PMD

In reply to Stephen Hemminger

Add a comprehensive test suite for the rtap poll mode driver covering:

- Device creation and configuration
- Device info query
- Link status and link up/down control
- Statistics collection and reset
- Promiscuous and all-multicast mode toggling
- MAC address and MTU configuration
- Multi-queue setup, teardown, and queue count reduction
- Mismatched Rx/Tx queue count rejection
- Queue reconfiguration (stop, reconfigure, restart)
- Rx path: inject packet via AF_PACKET, receive via DPDK
- Tx path: transmit via DPDK, capture via AF_PACKET
- Multi-segment Tx with chained mbufs
- Multi-segment Rx with small-buffer mempool
- Offload capability reporting and configuration
- Tx UDP checksum offload end-to-end verification
- File descriptor leak detection after device close
- Command-line created device testing

Tests require CAP_NET_ADMIN or root privileges

---
 app/test/meson.build     |    1 +
 app/test/test_pmd_rtap.c | 2620 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 2621 insertions(+)
 create mode 100644 app/test/test_pmd_rtap.c
app/test/meson.build +1 -0
index 48874037eb..5855077066 100644
@@ -143,6 +143,7 @@ source_file_deps = {
     'test_pie.c': ['sched'],
     'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps,
     'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'],
+    'test_pmd_rtap.c': ['net_rtap', 'ethdev', 'bus_vdev'],
     'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'],
     'test_pmu.c': ['pmu'],
     'test_power.c': ['power', 'power_acpi', 'power_kvm_vm', 'power_intel_pstate',
app/test/test_pmd_rtap.c new file +2620 -0

2,620 lines not on this page · raw message

Signature · 1 line
2.51.0
#70 lore raw

RE: [PATCH v6 00/11] net/rtap: add io_uring based TAP driver

In reply to Stephen Hemminger
Quoted 10 lines

This series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.

Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.

Quoted 8 lines

Motivation
----------

This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.

Sounds interesting...
Curious did you make any perf comparisons vs our traditional tap?

Quoted 91 lines

The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.

Why not extend net_tap?
-----------------------

The existing net_tap driver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity: rte_flow support emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.

net_rtap takes the opposite approach -- use the kernel efficiently and let it do what it does well. There is no rte_flow support; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.

Given these fundamentally different design goals, a clean implementation was more practical than refactoring net_tap.

Acknowledgement
---------------

Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.

Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)
- liburing >= 2.0

Should work on distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).

v6:
- fix lots of bugs found doing automated review
- convert to use ifindex and netlink to avoid rename issues
- implement xstats

Stephen Hemminger (11):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add multi-process support
net/rtap: add link state change interrupt net/rtap: add Rx interrupt support
net/rtap: add extended statistics support
test: add unit tests for rtap PMD

 MAINTAINERS                            |    7 +
 app/test/meson.build                   |    1 +
 app/test/test_pmd_rtap.c               | 2620 ++++++++++++++++++++++++
 doc/guides/nics/features/rtap.ini      |   26 +
 doc/guides/nics/index.rst              |    1 +
 doc/guides/nics/rtap.rst               |  101 +
 doc/guides/rel_notes/release_26_03.rst |    7 +
 drivers/net/meson.build                |    1 +
 drivers/net/rtap/meson.build           |   30 +
 drivers/net/rtap/rtap.h                |  152 ++
 drivers/net/rtap/rtap_ethdev.c         |  864 ++++++++
 drivers/net/rtap/rtap_intr.c           |  207 ++
 drivers/net/rtap/rtap_netlink.c        |  445 ++++
 drivers/net/rtap/rtap_rxtx.c           |  803 ++++++++
 drivers/net/rtap/rtap_xstats.c         |  293 +++
 15 files changed, 5558 insertions(+)
 create mode 100644 app/test/test_pmd_rtap.c
 create mode 100644 doc/guides/nics/features/rtap.ini
 create mode 100644 doc/guides/nics/rtap.rst
 create mode 100644 drivers/net/rtap/meson.build
 create mode 100644 drivers/net/rtap/rtap.h
 create mode 100644 drivers/net/rtap/rtap_ethdev.c
 create mode 100644 drivers/net/rtap/rtap_intr.c
 create mode 100644 drivers/net/rtap/rtap_netlink.c
 create mode 100644 drivers/net/rtap/rtap_rxtx.c
 create mode 100644 drivers/net/rtap/rtap_xstats.c

--
2.51.0

#71 lore raw

Re: [PATCH v6 00/11] net/rtap: add io_uring based TAP driver

Quoted 9 lines · On Sun, 15 Feb 2026 08:58:14 +0000 Konstantin Ananyev…
Quoted 6 lines

This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.

Sounds interesting...
Curious did you make any perf comparisons vs our traditional tap?

That is next step. Trying to get a testbed that is realistic setup.