ioring: network driver
Subsystems (2)
cover ioring: network driver
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statistics
doc/guides/nics/features/ioring.ini | 14 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 ++ drivers/net/ioring/meson.build | 12 + drivers/net/ioring/rte_eth_ioring.c | 1067 +++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 1161 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.45.2
RFC 1/8 net/ioring: introduce new driver
Add basic driver initialization, documentation, and device creation and basic documentation.
--- doc/guides/nics/features/ioring.ini | 9 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 +++++++ drivers/net/ioring/meson.build | 12 ++ drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 351 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
@@ -0,0 +1,9 @@ +; +; Supported features of the 'ioring' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -41,6 +41,7 @@ Network Interface Controller Drivers igc intel_vf ionic + ioring ipn3ke ixgbe mana
doc/guides/nics/ioring.rst new file +66 -0
66 lines not on this page · raw message
drivers/net/ioring/meson.build new file +12 -0
@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +ext_deps += dependency('liburing', required:true) + +sources = files('rte_eth_ioring.c') +require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0
262 lines not on this page · raw message
drivers/net/meson.build +1 -0
@@ -33,6 +33,7 @@ drivers = [ 'idpf', 'igc', 'ionic', + 'ioring', 'ipn3ke', 'ixgbe', 'mana',
Signature · 1 line
2.45.2
RFC 2/8 net/ioring: implement link state
In reply to Stephen HemmingerAdd hooks to set kernel link up/down and report state.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0
104 lines not on this page · raw message
Signature · 1 line
2.45.2
RFC 3/8 net/ioring: implement control functions
In reply to Stephen HemmingerThese internal ops, just force changes to kernel visible net device.
--- doc/guides/nics/features/ioring.ini | 3 ++ drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
@@ -5,6 +5,9 @@ ; [Features] Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0
93 lines not on this page · raw message
Signature · 1 line
2.45.2
RFC 4/8 net/ioring: implement management functions
In reply to Stephen HemmingerAdd start, stop, configure and info functions.
--- drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
@@ -212,12 +212,16 @@ static int eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu) {struct pmd_internals *pmd = dev->data->dev_private; + +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; +struct ifreq ifr = { .ifr_mtu = mtu }; -int ret;strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); - ret =ioctl (pmd->ctl_sock,SIOCSIFMTU , &ifr); +int ret =ioctl (sock,SIOCSIFMTU , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFMTU) failed: %s" ,strerror (errno)); ret = -errno;
@@ -230,14 +234,17 @@ static int eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr) { struct pmd_internals *pmd = dev->data->dev_private; -struct ifreq ifr = { }; -int ret; +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; + +struct ifreq ifr = { };strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); ifr.ifr_hwaddr.sa_family =ARPHRD_ETHER ;memcpy (ifr.ifr_hwaddr.sa_data, addr,sizeof (*addr)); - ret =ioctl (pmd->ctl_sock,SIOCSIFHWADDR , &ifr); +int ret =ioctl (sock,SIOCSIFHWADDR , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFHWADDR) failed: %s" ,strerror (errno)); ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)return ret; } +static int +eth_dev_start (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ; +eth_dev_set_link_up (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + } + +return 0 ; +} + +static int +eth_dev_stop (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +eth_dev_set_link_down (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + } + +return 0 ; +} + +static int +eth_dev_configure (struct rte_eth_dev *dev) +{ +/* rx/tx must be paired */ +if (dev->data->nb_rx_queues != dev->data->nb_tx_queues) +return -EINVAL ; + +return 0 ; +} + +static int +eth_dev_info (struct rte_eth_dev *dev,struct rte_eth_dev_info *dev_info) +{ +struct pmd_internals *pmd = dev->data->dev_private; + + dev_info->if_index =if_nametoindex (pmd->ifname); + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + +return 0 ; +} +static int eth_dev_close (struct rte_eth_dev *dev) {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)close (pmd->keep_fd); pmd->keep_fd =-1 ; } -return 0 ; }static const struct eth_dev_ops ops = { + .dev_start = eth_dev_start, + .dev_stop = eth_dev_stop, + .dev_configure = eth_dev_configure, + .dev_infos_get = eth_dev_info, .dev_close = eth_dev_close, .link_update = eth_link_update, .dev_set_link_up = eth_dev_set_link_up,
Signature · 1 line
2.45.2
RFC 5/8 net/ioring: implement primary secondary fd passing
In reply to Stephen HemmingerAdd support for communicating fd's from primary to secondary.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++- 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
@@ -8,6 +8,7 @@ Link status = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y +Multiprocess aware = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2
181 lines not on this page · raw message
Signature · 1 line
2.45.2
RFC 6/8 net/ioring: implement receive and transmit
In reply to Stephen HemmingerUse io_uring to read and write from TAP device.
--- drivers/net/ioring/rte_eth_ioring.c | 364 +++++++++++++++++++++++++++- 1 file changed, 363 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +363 -1
@@ -2,6 +2,7 @@* Copyright (c) Stephen Hemminger */ +#include <assert.h> #include <ctype.h> #include <errno.h> #include <fcntl.h>
@@ -9,8 +10,10 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <liburing.h> #include <sys/ioctl.h> #include <sys/socket.h> +#include <sys/uio.h> #include <net/if.h> #include <linux/if.h> #include <linux/if_arp.h>
@@ -27,6 +30,13 @@#include <rte_kvargs.h> #include <rte_log.h> +#define IORING_DEFAULT_BURST 64 +#define IORING_NUM_BUFFERS 1024 +#define IORING_MAX_QUEUES 128 + + +static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_DEFAULT_IFNAME "enio%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -34,6 +44,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);#define RTE_LOGTYPE_IORING ioring_logtype#define PMD_LOG (level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__) +#ifdef RTE_ETHDEV_DEBUG_RX +#define PMD_RX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__) +#else +#define PMD_RX_LOG (...) do { } while (0) +#endif + +#ifdef RTE_ETHDEV_DEBUG_TX +#define PMD_TX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__) +#else +#define PMD_TX_LOG (...) do { } while (0) +#endif +#define IORING_IFACE_ARG "iface"#define IORING_PERSIST_ARG "persist"
@@ -43,6 +67,30 @@ static const char * const valid_arguments[] = {NULL }; +struct rx_queue { +struct rte_mempool *mb_pool;/* rx buffer pool */ +struct io_uring io_ring;/* queue of posted read's */ +uint16_t port_id; +uint16_t queue_id; + +uint64_t rx_packets; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_errors; +}; + +struct tx_queue { +struct io_uring io_ring; + +uint16_t port_id; +uint16_t queue_id; +uint16_t free_thresh; + +uint64_t tx_packets; +uint64_t tx_bytes; +uint64_t tx_errors; +}; +struct pmd_internals {int keep_fd;/* keep alive file descriptor */ char ifname[IFNAMSIZ ];/* name assigned by kernel */
@@ -300,6 +348,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->if_index =if_nametoindex (pmd->ifname); dev_info->max_mac_addrs =1 ; dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + dev_info->max_rx_queues =IORING_MAX_QUEUES ; + dev_info->max_tx_queues =IORING_MAX_QUEUES ; + dev_info->min_rx_bufsize =0 ; + + dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { + .burst_size =IORING_DEFAULT_BURST , + .ring_size =IORING_NUM_BUFFERS , + .nb_queues =1 , + };return 0 ; }
@@ -311,6 +368,14 @@ eth_dev_close(struct rte_eth_dev *dev)PMD_LOG (INFO ,"Closing %s" , pmd->ifname); +int *fds = dev->process_private; +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +if (fds[i] ==-1 ) +continue ; +close (fds[i]); + fds[i] =-1 ; + } +if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -324,6 +389,296 @@ eth_dev_close(struct rte_eth_dev *dev)return 0 ; } +/* Setup another fd to TAP device for the queue */ +static int +eth_queue_setup (struct rte_eth_dev *dev,const char *name,uint16_t queue_id) +{ +int *fds = dev->process_private; + +if (fds[queue_id] !=-1 ) +return 0 ;/* already setup */ + +struct ifreq ifr = { }; +int tap_fd =tap_open (name, &ifr,0 ); +if (tap_fd <0 ) { +PMD_LOG (ERR ,"tap_open failed" ); +return -1 ; + } + +PMD_LOG (DEBUG ,"opened %d for queue %u" , tap_fd, queue_id); + fds[queue_id] = tap_fd; +return 0 ; +} + +static int +eth_queue_fd (uint16_t port_id,uint16_t queue_id) +{ +struct rte_eth_dev *dev = &rte_eth_devices[port_id]; +int *fds = dev->process_private; + +return fds[queue_id]; +} + +/* setup an submit queue to read mbuf */ +static inline void +eth_rx_submit (struct rx_queue *rxq,int fd,struct rte_mbuf *mb) +{ +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); + +if (unlikely (sqe ==NULL )) { +PMD_LOG (DEBUG ,"io_uring no rx sqe" ); + rxq->rx_errors++; + }else { + void *base = rte_pktmbuf_mtod(mb, void *); +size_t len = mb->buf_len; + +io_uring_prep_read (sqe, fd, base, len,0 ); +io_uring_sqe_set_data (sqe, mb); + } +} + +static uint16_t +eth_ioring_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct rx_queue *rxq = queue; +struct io_uring_cqe *cqe; +unsigned int head, num_cqe =0 ; +uint16_t num_rx =0 ; +uint32_t num_bytes =0 ; +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +io_uring_for_each_cqe (&rxq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; +ssize_t len = cqe->res; + +PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); + num_cqe++; + +if (unlikely (len <RTE_ETHER_HDR_LEN )) { +if (len <0 ) +PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len)); +else +PMD_LOG (ERR ,"io_uring_read missing hdr" ); + + rxq->rx_errors++; + goto resubmit; + } + +struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (nmb ==0 )) { +PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); + ++rxq->rx_nombuf; + goto resubmit; + } + + mb->pkt_len = len; + mb->data_len = len; + mb->port = rxq->port_id; +__rte_mbuf_sanity_check (mb,1 ); + + num_bytes += len; + bufs[num_rx++] = mb; + + mb = nmb; +resubmit: +eth_rx_submit (rxq, fd, mb); + +if (num_rx == nb_pkts) +break ; + } +io_uring_cq_advance (&rxq->io_ring, num_cqe); + + rxq->rx_packets += num_rx; + rxq->rx_bytes += num_bytes; +return num_rx; +} + +static int +eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, +unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf __rte_unused, +struct rte_mempool *mb_pool) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +PMD_LOG (DEBUG ,"setup port %u queue %u rx_descriptors %u" , + dev->data->port_id, queue_id, nb_rx_desc); + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct rx_queue *rxq =rte_zmalloc_socket (NULL ,sizeof (*rxq), +RTE_CACHE_LINE_SIZE , socket_id); +if (rxq ==NULL ) { +PMD_LOG (ERR ,"rxq alloc failed" ); +return -1 ; + } + + rxq->mb_pool = mb_pool; + rxq->port_id = dev->data->port_id; + rxq->queue_id = queue_id; + dev->data->rx_queues[queue_id] = rxq; + +struct rte_mbuf *mbufs[nb_rx_desc]; +if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { +PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); +return -1 ; + } + +if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); + goto error; + } + +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +for (uint16_t i =0 ; i < nb_rx_desc; i++) { +struct rte_mbuf *mb = mbufs[i]; + +eth_rx_submit (rxq, fd, mb); + } + +io_uring_submit (&rxq->io_ring); +return 0 ; + +error: +rte_pktmbuf_free_bulk (mbufs, nb_rx_desc); +return -1 ; +} + +static void +eth_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct rx_queue *rxq = dev->data->rx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&rxq->io_ring,1 ); + } + +io_uring_queue_exit (&rxq->io_ring); +} + +static int +eth_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct tx_queue *txq =rte_zmalloc_socket (NULL ,sizeof (*txq),RTE_CACHE_LINE_SIZE , socket_id); +if (txq ==NULL ) { +PMD_LOG (ERR ,"txq alloc failed" ); +return -1 ; + } + + txq->port_id = dev->data->port_id; + txq->queue_id = queue_id; + txq->free_thresh = tx_conf->tx_free_thresh; + dev->data->tx_queues[queue_id] = txq; + +if (io_uring_queue_init (nb_tx_desc, &txq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static void +eth_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct tx_queue *txq = dev->data->tx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&txq->io_ring,1 ); + } + +io_uring_queue_exit (&txq->io_ring); +} + +static void +eth_ioring_tx_cleanup (struct tx_queue *txq) +{ +struct io_uring_cqe *cqe; +unsigned int head; +unsigned int tx_done =0 ; +uint64_t tx_bytes =0 ; + +io_uring_for_each_cqe (&txq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; + +PMD_TX_LOG (DEBUG ," mbuf len %u result: %d" , mb->pkt_len, cqe->res); +if (unlikely (cqe->res <0 )) { + ++txq->tx_errors; + }else { + ++tx_done; + tx_bytes += mb->pkt_len; + } + +rte_pktmbuf_free (mb); + } +io_uring_cq_advance (&txq->io_ring, tx_done); + + txq->tx_packets += tx_done; + txq->tx_bytes += tx_bytes; +} + +static uint16_t +eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct tx_queue *txq = queue; +uint16_t num_tx; + +if (unlikely (nb_pkts ==0 )) +return 0 ; + +PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); + +if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh) +eth_ioring_tx_cleanup (txq); + +int fd =eth_queue_fd (txq->port_id, txq->queue_id); + +for (num_tx =0 ; num_tx < nb_pkts; num_tx++) { +struct rte_mbuf *mb = bufs[num_tx]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) +break ;/* submit ring is full */ + +io_uring_sqe_set_data (sqe, mb); + +if (rte_mbuf_refcnt_read (mb) ==1 && +RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { + void *base = rte_pktmbuf_mtod(mb, void *); +io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); + +PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); + }else { +PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); + ++txq->tx_errors; +continue ; + } + } +if (num_tx >0 ) +io_uring_submit (&txq->io_ring); + +return num_tx; +} +static const struct eth_dev_ops ops = { .dev_start = eth_dev_start, .dev_stop = eth_dev_stop,
@@ -339,9 +694,12 @@ static const struct eth_dev_ops ops = { .promiscuous_disable = eth_dev_promiscuous_disable, .allmulticast_enable = eth_dev_allmulticast_enable, .allmulticast_disable = eth_dev_allmulticast_disable, + .rx_queue_setup = eth_rx_queue_setup, + .rx_queue_release = eth_rx_queue_release, + .tx_queue_setup = eth_tx_queue_setup, + .tx_queue_release = eth_tx_queue_release, }; -static int ioring_create (struct rte_eth_dev *dev,const char *tap_name,uint8_t persist) {
@@ -379,6 +737,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) }PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); + + dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_burst = eth_ioring_tx; +return 0 ; error:
Signature · 1 line
2.45.2
RFC 7/8 net/ioring: add VLAN support
In reply to Stephen HemmingerAdd support for VLAN insert and stripping.
--- drivers/net/ioring/rte_eth_ioring.c | 45 +++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +43 -2
@@ -34,6 +34,8 @@#define IORING_NUM_BUFFERS 1024#define IORING_MAX_QUEUES 128 +#define IORING_TX_OFFLOAD RTE_ETH_TX_OFFLOAD_VLAN_INSERT +#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_VLAN_STRIPstatic_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" );
@@ -70,6 +72,7 @@ static const char * const valid_arguments[] = {struct rx_queue {struct rte_mempool *mb_pool;/* rx buffer pool */ struct io_uring io_ring;/* queue of posted read's */ +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -81,6 +84,7 @@ struct rx_queue {struct tx_queue {struct io_uring io_ring; +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts) goto resubmit; } +if (rxq->offloads &RTE_ETH_RX_OFFLOAD_VLAN_STRIP ) +rte_vlan_strip (mb); + mb->pkt_len = len; mb->data_len = len; mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)static int eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, -unsigned int socket_id, - const struct rte_eth_rxconf *rx_conf __rte_unused, +unsigned int socket_id,const struct rte_eth_rxconf *rx_conf,struct rte_mempool *mb_pool) {struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_dereturn -1 ; } + rxq->offloads = rx_conf->offloads; rxq->mb_pool = mb_pool; rxq->port_id = dev->data->port_id; rxq->queue_id = queue_id;
@@ -582,6 +589,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, txq->port_id = dev->data->port_id; txq->queue_id = queue_id; + txq->offloads = tx_conf->offloads; txq->free_thresh = tx_conf->tx_free_thresh; dev->data->tx_queues[queue_id] = txq;
@@ -636,6 +644,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq) txq->tx_bytes += tx_bytes; } +static uint16_t +eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) +{ +uint16_t nb_tx; +int error; + +for (nb_tx =0 ; nb_tx < nb_pkts; nb_tx++) { +struct rte_mbuf *m = tx_pkts[nb_tx]; + +#ifdef RTE_LIBRTE_ETHDEV_DEBUG + error =rte_validate_tx_offload (m); +if (unlikely (error)) { + rte_errno = -error; +break ; + } +#endif +/* Do VLAN tag insertion */ +if (unlikely (m->ol_flags &RTE_MBUF_F_TX_VLAN )) { + error =rte_vlan_insert (&m); +/* rte_vlan_insert() may change pointer */ + tx_pkts[nb_tx] = m; + +if (unlikely (error)) { + rte_errno = -error; +break ; + } + } + } + +return nb_tx; +} +static uint16_t eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) {
@@ -739,6 +779,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_prepare = eth_ioring_tx_prepare; dev->tx_pkt_burst = eth_ioring_tx;return 0 ;
Signature · 1 line
2.45.2
RFC 8/8 net/ioring: implement statistics
In reply to Stephen HemmingerAdd support for basic statistics
--- drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
drivers/net/ioring/rte_eth_ioring.c +57 -0
@@ -365,6 +365,61 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)return 0 ; } +static int +eth_dev_stats_get (struct rte_eth_dev *dev,struct rte_eth_stats *stats) +{ +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +const struct rx_queue *rxq = dev->data->rx_queues[i]; + + stats->ipackets += rxq->rx_packets; + stats->ibytes += rxq->rx_bytes; + stats->ierrors += rxq->rx_errors; + stats->rx_nombuf += rxq->rx_nombuf; + +if (i <RTE_ETHDEV_QUEUE_STAT_CNTRS ) { + stats->q_ipackets[i] = rxq->rx_packets; + stats->q_ibytes[i] = rxq->rx_bytes; + } + } + +for (uint16_t i =0 ; i < dev->data->nb_tx_queues; i++) { +const struct tx_queue *txq = dev->data->tx_queues[i]; + + stats->opackets += txq->tx_packets; + stats->obytes += txq->tx_bytes; + stats->oerrors += txq->tx_errors; + +if (i <RTE_ETHDEV_QUEUE_STAT_CNTRS ) { + stats->q_opackets[i] = txq->tx_packets; + stats->q_obytes[i] = txq->tx_bytes; + } + } + +return 0 ; +} + +static int +eth_dev_stats_reset (struct rte_eth_dev *dev) +{ +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +struct rx_queue *rxq = dev->data->rx_queues[i]; + + rxq->rx_packets =0 ; + rxq->rx_bytes =0 ; + rxq->rx_nombuf =0 ; + rxq->rx_errors =0 ; + } + +for (uint16_t i =0 ; i < dev->data->nb_tx_queues; i++) { +struct tx_queue *txq = dev->data->tx_queues[i]; + + txq->tx_packets =0 ; + txq->tx_bytes =0 ; + txq->tx_errors =0 ; + } +return 0 ; +} +static int eth_dev_close (struct rte_eth_dev *dev) {
@@ -734,6 +789,8 @@ static const struct eth_dev_ops ops = { .promiscuous_disable = eth_dev_promiscuous_disable, .allmulticast_enable = eth_dev_allmulticast_enable, .allmulticast_disable = eth_dev_allmulticast_disable, + .stats_get = eth_dev_stats_get, + .stats_reset = eth_dev_stats_reset, .rx_queue_setup = eth_rx_queue_setup, .rx_queue_release = eth_rx_queue_release, .tx_queue_setup = eth_tx_queue_setup,
Signature · 1 line
2.45.2
Quoted 12 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
Quoted 4 lines
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, 'tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.
Quoted 24 lines
Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statisticsdoc/guides/nics/features/ioring.ini | 14 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 ++ drivers/net/ioring/meson.build | 12 + drivers/net/ioring/rte_eth_ioring.c | 1067 +++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 1161 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c--
2.45.2
Quoted 15 lines · On Wed, 11 Dec 2024 11:34:39 +0000 Konstantin Ananyev…
Quoted 10 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
Quoted 4 lines
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Quoted 4 lines
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, '
tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.
There are some userspaces that look for "e*" in name for some setups. But names are totally abitrary
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Quoted 6 lines
Quoted 4 lines
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, '
tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.There are some userspaces that look for "e*" in name for some setups.
Didn't get you here, pls try to re-phrase.
Quoted 1 line
But names are totally abitrary
Quoted 21 lines
From: Konstantin Ananyev [mailto:[email protected]]
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
Quoted 4 lines
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, '
tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.
Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "net/tap_ioring".
Quoted 2 lines
Quoted 2 lines
There are some userspaces that look for "e*" in name for some setups.
Stephen is referring to the device name of an instantiated interface, e.g. "eth0".
And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)
Quoted 4 lines
Didn't get you here, pls try to re-phrase.
Quoted 1 line
But names are totally arbitrary
Quoted 19 lines
Quoted 19 lines
From: Konstantin Ananyev [mailto:[email protected]]
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, '
tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "
net/tap_ioring".
Yep, that what I meant.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
There are some userspaces that look for "e*" in name for some setups.
Stephen is referring to the device name of an instantiated interface, e.g. "eth0".
And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)
Ok... and why such practice should prevent us to name PMD itself in a way we think is appropriate?
Quoted 6 lines
Quoted 4 lines
Didn't get you here, pls try to re-phrase.
Quoted 1 line
But names are totally arbitrary
Quoted 17 lines · On Fri, 20 Dec 2024 14:34:27 +0000 Konstantin Ananyev…
Quoted 17 lines
Quoted 17 lines
From: Konstantin Ananyev [mailto:[email protected]]
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
This is first draft of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Makes sense to me, though didn't properly look inside. One thing - probably add a 'tap' into the name, '
tap_ioiring' or so, otherwise 'ioring' is a bit too generic and might be confusing.Konstantin is referring to the name of the driver and the source code file names, "net/ioring" -> "
net/tap_ioring".Yep, that what I meant.
My thoughts, are shorter name is better, and avoids confusion. There are already multiple drivers that create tap devices: tap and virtio_user.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
There are some userspaces that look for "e*" in name for some setups.
Stephen is referring to the device name of an instantiated interface, e.g. "eth0".
And yes, assuming devices named "e*" are Ethernet devices is a common hack in Linux applications. I've done it myself. :-)
Ok... and why such practice should prevent us to name PMD itself in a way we think is appropriate?
I am more leaning towards not having a default name at all. The policy should be done by Linux (udev) not DPDK. If user wants a name they can add it via devargs.
v2 cover ioring: network driver
In reply to Stephen HemmingerThis is initial work of new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
Stephen Hemminger (8):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement primary secondary fd passing net/ioring: implement receive and transmit net/ioring: add VLAN support
net/ioring: implement statistics
doc/guides/nics/features/ioring.ini | 16 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 ++ drivers/net/ioring/meson.build | 15 + drivers/net/ioring/rte_eth_ioring.c | 1068 +++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 1167 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.45.2
PATCH v2 1/8 net/ioring: introduce new driver
Add basic driver initialization, documentation, and device creation and basic documentation.
--- doc/guides/nics/features/ioring.ini | 9 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 +++++++ drivers/net/ioring/meson.build | 15 ++ drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 354 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
@@ -0,0 +1,9 @@ +; +; Supported features of the 'ioring' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -41,6 +41,7 @@ Network Interface Controller Drivers igc intel_vf ionic + ioring ipn3ke ixgbe mana
doc/guides/nics/ioring.rst new file +66 -0
66 lines not on this page · raw message
drivers/net/ioring/meson.build new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +dep = dependency('liburing', required:false) +reason = 'missing dependency, "liburing"' +build = dep.found() +ext_deps += dep + +sources = files('rte_eth_ioring.c') +require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0
262 lines not on this page · raw message
drivers/net/meson.build +1 -0
@@ -33,6 +33,7 @@ drivers = [ 'idpf', 'igc', 'ionic', + 'ioring', 'ipn3ke', 'ixgbe', 'mana',
Signature · 1 line
2.45.2
RE: [PATCH v2 1/8] net/ioring: introduce new driver
drivers/net/ioring/rte_eth_ioring.c line 70
Quoted 8 lines
drivers/net/ioring/rte_eth_ioring.c new file +8
-0+int features =0 ; +if (ioctl (tap_fd,TUNGETFEATURES , &features) <0 ) { +PMD_LOG (ERR ,"ioctl(TUNGETFEATURES) %s" ,strerror (errno)); + goto error; + } + +int flags =IFF_TAP |IFF_MULTI_QUEUE |IFF_NO_PI ; +if ((features & flags) ==0 ) {
Comparison will only fail if all three flags are missing. Should be:
if ((features & flags) != flags) {
drivers/net/ioring/rte_eth_ioring.c line 72
Quoted 2 lines
drivers/net/ioring/rte_eth_ioring.c new file +2
-0+PMD_LOG (ERR ,"TUN features %#x missing support for %#x" , + features, features & flags);
Should be:
features, (features & flags) ^ flags);
Quoted 2 lines
(quoted diff) +2
-0+ goto error; + }
PATCH v2 2/8 net/ioring: implement link state
In reply to Stephen HemmingerAdd hooks to set kernel link up/down and report state.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0
104 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v2 3/8 net/ioring: implement control functions
In reply to Stephen HemmingerThese internal ops, just force changes to kernel visible net device.
--- doc/guides/nics/features/ioring.ini | 3 ++ drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
@@ -5,6 +5,9 @@ ; [Features] Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0
93 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v2 4/8 net/ioring: implement management functions
In reply to Stephen HemmingerAdd start, stop, configure and info functions.
--- drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
@@ -212,12 +212,16 @@ static int eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu) {struct pmd_internals *pmd = dev->data->dev_private; + +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; +struct ifreq ifr = { .ifr_mtu = mtu }; -int ret;strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); - ret =ioctl (pmd->ctl_sock,SIOCSIFMTU , &ifr); +int ret =ioctl (sock,SIOCSIFMTU , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFMTU) failed: %s" ,strerror (errno)); ret = -errno;
@@ -230,14 +234,17 @@ static int eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr) { struct pmd_internals *pmd = dev->data->dev_private; -struct ifreq ifr = { }; -int ret; +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; + +struct ifreq ifr = { };strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); ifr.ifr_hwaddr.sa_family =ARPHRD_ETHER ;memcpy (ifr.ifr_hwaddr.sa_data, addr,sizeof (*addr)); - ret =ioctl (pmd->ctl_sock,SIOCSIFHWADDR , &ifr); +int ret =ioctl (sock,SIOCSIFHWADDR , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFHWADDR) failed: %s" ,strerror (errno)); ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)return ret; } +static int +eth_dev_start (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ; +eth_dev_set_link_up (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + } + +return 0 ; +} + +static int +eth_dev_stop (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +eth_dev_set_link_down (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + } + +return 0 ; +} + +static int +eth_dev_configure (struct rte_eth_dev *dev) +{ +/* rx/tx must be paired */ +if (dev->data->nb_rx_queues != dev->data->nb_tx_queues) +return -EINVAL ; + +return 0 ; +} + +static int +eth_dev_info (struct rte_eth_dev *dev,struct rte_eth_dev_info *dev_info) +{ +struct pmd_internals *pmd = dev->data->dev_private; + + dev_info->if_index =if_nametoindex (pmd->ifname); + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + +return 0 ; +} +static int eth_dev_close (struct rte_eth_dev *dev) {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)close (pmd->keep_fd); pmd->keep_fd =-1 ; } -return 0 ; }static const struct eth_dev_ops ops = { + .dev_start = eth_dev_start, + .dev_stop = eth_dev_stop, + .dev_configure = eth_dev_configure, + .dev_infos_get = eth_dev_info, .dev_close = eth_dev_close, .link_update = eth_link_update, .dev_set_link_up = eth_dev_set_link_up,
Signature · 1 line
2.45.2
PATCH v2 5/8 net/ioring: implement primary secondary fd passing
In reply to Stephen HemmingerAdd support for communicating fd's from primary to secondary.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++- 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
@@ -8,6 +8,7 @@ Link status = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y +Multiprocess aware = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2
181 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v2 6/8 net/ioring: implement receive and transmit
In reply to Stephen HemmingerUse io_uring to read and write from TAP device.
--- drivers/net/ioring/rte_eth_ioring.c | 365 +++++++++++++++++++++++++++- 1 file changed, 364 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +364 -1
@@ -2,6 +2,7 @@* Copyright (c) Stephen Hemminger */ +#include <assert.h> #include <ctype.h> #include <errno.h> #include <fcntl.h>
@@ -9,8 +10,10 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <liburing.h> #include <sys/ioctl.h> #include <sys/socket.h> +#include <sys/uio.h> #include <net/if.h> #include <linux/if.h> #include <linux/if_arp.h>
@@ -27,6 +30,13 @@#include <rte_kvargs.h> #include <rte_log.h> +#define IORING_DEFAULT_BURST 64 +#define IORING_NUM_BUFFERS 1024 +#define IORING_MAX_QUEUES 128 + + +static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_DEFAULT_IFNAME "enio%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -34,6 +44,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);#define RTE_LOGTYPE_IORING ioring_logtype#define PMD_LOG (level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__) +#ifdef RTE_ETHDEV_DEBUG_RX +#define PMD_RX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__) +#else +#define PMD_RX_LOG (...) do { } while (0) +#endif + +#ifdef RTE_ETHDEV_DEBUG_TX +#define PMD_TX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__) +#else +#define PMD_TX_LOG (...) do { } while (0) +#endif +#define IORING_IFACE_ARG "iface"#define IORING_PERSIST_ARG "persist"
@@ -43,6 +67,30 @@ static const char * const valid_arguments[] = {NULL }; +struct rx_queue { +struct rte_mempool *mb_pool;/* rx buffer pool */ +struct io_uring io_ring;/* queue of posted read's */ +uint16_t port_id; +uint16_t queue_id; + +uint64_t rx_packets; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_errors; +}; + +struct tx_queue { +struct io_uring io_ring; + +uint16_t port_id; +uint16_t queue_id; +uint16_t free_thresh; + +uint64_t tx_packets; +uint64_t tx_bytes; +uint64_t tx_errors; +}; +struct pmd_internals {int keep_fd;/* keep alive file descriptor */ char ifname[IFNAMSIZ ];/* name assigned by kernel */
@@ -300,6 +348,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->if_index =if_nametoindex (pmd->ifname); dev_info->max_mac_addrs =1 ; dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + dev_info->max_rx_queues =IORING_MAX_QUEUES ; + dev_info->max_tx_queues =IORING_MAX_QUEUES ; + dev_info->min_rx_bufsize =0 ; + + dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { + .burst_size =IORING_DEFAULT_BURST , + .ring_size =IORING_NUM_BUFFERS , + .nb_queues =1 , + };return 0 ; }
@@ -311,6 +368,14 @@ eth_dev_close(struct rte_eth_dev *dev)PMD_LOG (INFO ,"Closing %s" , pmd->ifname); +int *fds = dev->process_private; +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +if (fds[i] ==-1 ) +continue ; +close (fds[i]); + fds[i] =-1 ; + } +if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -324,6 +389,297 @@ eth_dev_close(struct rte_eth_dev *dev)return 0 ; } +/* Setup another fd to TAP device for the queue */ +static int +eth_queue_setup (struct rte_eth_dev *dev,const char *name,uint16_t queue_id) +{ +int *fds = dev->process_private; + +if (fds[queue_id] !=-1 ) +return 0 ;/* already setup */ + +struct ifreq ifr = { }; +int tap_fd =tap_open (name, &ifr,0 ); +if (tap_fd <0 ) { +PMD_LOG (ERR ,"tap_open failed" ); +return -1 ; + } + +PMD_LOG (DEBUG ,"opened %d for queue %u" , tap_fd, queue_id); + fds[queue_id] = tap_fd; +return 0 ; +} + +static int +eth_queue_fd (uint16_t port_id,uint16_t queue_id) +{ +struct rte_eth_dev *dev = &rte_eth_devices[port_id]; +int *fds = dev->process_private; + +return fds[queue_id]; +} + +/* setup an submit queue to read mbuf */ +static inline void +eth_rx_submit (struct rx_queue *rxq,int fd,struct rte_mbuf *mb) +{ +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); + +if (unlikely (sqe ==NULL )) { +PMD_LOG (DEBUG ,"io_uring no rx sqe" ); + rxq->rx_errors++; + }else { + void *base = rte_pktmbuf_mtod(mb, void *); +size_t len = mb->buf_len; + +io_uring_prep_read (sqe, fd, base, len,0 ); +io_uring_sqe_set_data (sqe, mb); + } +} + +static uint16_t +eth_ioring_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct rx_queue *rxq = queue; +struct io_uring_cqe *cqe; +unsigned int head, num_cqe =0 ; +uint16_t num_rx =0 ; +uint32_t num_bytes =0 ; +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +io_uring_for_each_cqe (&rxq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; +ssize_t len = cqe->res; + +PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); + num_cqe++; + +if (unlikely (len <RTE_ETHER_HDR_LEN )) { +if (len <0 ) +PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len)); +else +PMD_LOG (ERR ,"io_uring_read missing hdr" ); + + rxq->rx_errors++; + goto resubmit; + } + +struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (nmb ==0 )) { +PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); + ++rxq->rx_nombuf; + goto resubmit; + } + + mb->pkt_len = len; + mb->data_len = len; + mb->port = rxq->port_id; +__rte_mbuf_sanity_check (mb,1 ); + + num_bytes += len; + bufs[num_rx++] = mb; + + mb = nmb; +resubmit: +eth_rx_submit (rxq, fd, mb); + +if (num_rx == nb_pkts) +break ; + } +io_uring_cq_advance (&rxq->io_ring, num_cqe); + + rxq->rx_packets += num_rx; + rxq->rx_bytes += num_bytes; +return num_rx; +} + +static int +eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, +unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf __rte_unused, +struct rte_mempool *mb_pool) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +PMD_LOG (DEBUG ,"setup port %u queue %u rx_descriptors %u" , + dev->data->port_id, queue_id, nb_rx_desc); + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct rx_queue *rxq =rte_zmalloc_socket (NULL ,sizeof (*rxq), +RTE_CACHE_LINE_SIZE , socket_id); +if (rxq ==NULL ) { +PMD_LOG (ERR ,"rxq alloc failed" ); +return -1 ; + } + + rxq->mb_pool = mb_pool; + rxq->port_id = dev->data->port_id; + rxq->queue_id = queue_id; + dev->data->rx_queues[queue_id] = rxq; + +struct rte_mbuf *mbufs[nb_rx_desc]; +if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { +PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); +return -1 ; + } + +if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); + goto error; + } + +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +for (uint16_t i =0 ; i < nb_rx_desc; i++) { +struct rte_mbuf *mb = mbufs[i]; + +eth_rx_submit (rxq, fd, mb); + } + +io_uring_submit (&rxq->io_ring); +return 0 ; + +error: +rte_pktmbuf_free_bulk (mbufs, nb_rx_desc); +return -1 ; +} + +static void +eth_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct rx_queue *rxq = dev->data->rx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&rxq->io_ring,1 ); + } + +io_uring_queue_exit (&rxq->io_ring); +} + +static int +eth_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct tx_queue *txq =rte_zmalloc_socket (NULL ,sizeof (*txq), +RTE_CACHE_LINE_SIZE , socket_id); +if (txq ==NULL ) { +PMD_LOG (ERR ,"txq alloc failed" ); +return -1 ; + } + + txq->port_id = dev->data->port_id; + txq->queue_id = queue_id; + txq->free_thresh = tx_conf->tx_free_thresh; + dev->data->tx_queues[queue_id] = txq; + +if (io_uring_queue_init (nb_tx_desc, &txq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static void +eth_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct tx_queue *txq = dev->data->tx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&txq->io_ring,1 ); + } + +io_uring_queue_exit (&txq->io_ring); +} + +static void +eth_ioring_tx_cleanup (struct tx_queue *txq) +{ +struct io_uring_cqe *cqe; +unsigned int head; +unsigned int tx_done =0 ; +uint64_t tx_bytes =0 ; + +io_uring_for_each_cqe (&txq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; + +PMD_TX_LOG (DEBUG ," mbuf len %u result: %d" , mb->pkt_len, cqe->res); +if (unlikely (cqe->res <0 )) { + ++txq->tx_errors; + }else { + ++tx_done; + tx_bytes += mb->pkt_len; + } + +rte_pktmbuf_free (mb); + } +io_uring_cq_advance (&txq->io_ring, tx_done); + + txq->tx_packets += tx_done; + txq->tx_bytes += tx_bytes; +} + +static uint16_t +eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct tx_queue *txq = queue; +uint16_t num_tx; + +if (unlikely (nb_pkts ==0 )) +return 0 ; + +PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); + +if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh) +eth_ioring_tx_cleanup (txq); + +int fd =eth_queue_fd (txq->port_id, txq->queue_id); + +for (num_tx =0 ; num_tx < nb_pkts; num_tx++) { +struct rte_mbuf *mb = bufs[num_tx]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) +break ;/* submit ring is full */ + +io_uring_sqe_set_data (sqe, mb); + +if (rte_mbuf_refcnt_read (mb) ==1 && +RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { + void *base = rte_pktmbuf_mtod(mb, void *); +io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); + +PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); + }else { +PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); + ++txq->tx_errors; +continue ; + } + } +if (num_tx >0 ) +io_uring_submit (&txq->io_ring); + +return num_tx; +} +static const struct eth_dev_ops ops = { .dev_start = eth_dev_start, .dev_stop = eth_dev_stop,
@@ -339,9 +695,12 @@ static const struct eth_dev_ops ops = { .promiscuous_disable = eth_dev_promiscuous_disable, .allmulticast_enable = eth_dev_allmulticast_enable, .allmulticast_disable = eth_dev_allmulticast_disable, + .rx_queue_setup = eth_rx_queue_setup, + .rx_queue_release = eth_rx_queue_release, + .tx_queue_setup = eth_tx_queue_setup, + .tx_queue_release = eth_tx_queue_release, }; -static int ioring_create (struct rte_eth_dev *dev,const char *tap_name,uint8_t persist) {
@@ -379,6 +738,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) }PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); + + dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_burst = eth_ioring_tx; +return 0 ; error:
Signature · 1 line
2.45.2
PATCH v2 7/8 net/ioring: add VLAN support
In reply to Stephen HemmingerAdd support for VLAN insert and stripping.
--- drivers/net/ioring/rte_eth_ioring.c | 45 +++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +43 -2
@@ -34,6 +34,8 @@#define IORING_NUM_BUFFERS 1024#define IORING_MAX_QUEUES 128 +#define IORING_TX_OFFLOAD RTE_ETH_TX_OFFLOAD_VLAN_INSERT +#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_VLAN_STRIPstatic_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" );
@@ -70,6 +72,7 @@ static const char * const valid_arguments[] = {struct rx_queue {struct rte_mempool *mb_pool;/* rx buffer pool */ struct io_uring io_ring;/* queue of posted read's */ +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -81,6 +84,7 @@ struct rx_queue {struct tx_queue {struct io_uring io_ring; +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts) goto resubmit; } +if (rxq->offloads &RTE_ETH_RX_OFFLOAD_VLAN_STRIP ) +rte_vlan_strip (mb); + mb->pkt_len = len; mb->data_len = len; mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)static int eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, -unsigned int socket_id, - const struct rte_eth_rxconf *rx_conf __rte_unused, +unsigned int socket_id,const struct rte_eth_rxconf *rx_conf,struct rte_mempool *mb_pool) {struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_dereturn -1 ; } + rxq->offloads = rx_conf->offloads; rxq->mb_pool = mb_pool; rxq->port_id = dev->data->port_id; rxq->queue_id = queue_id;
@@ -583,6 +590,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, txq->port_id = dev->data->port_id; txq->queue_id = queue_id; + txq->offloads = tx_conf->offloads; txq->free_thresh = tx_conf->tx_free_thresh; dev->data->tx_queues[queue_id] = txq;
@@ -637,6 +645,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq) txq->tx_bytes += tx_bytes; } +static uint16_t +eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) +{ +uint16_t nb_tx; +int error; + +for (nb_tx =0 ; nb_tx < nb_pkts; nb_tx++) { +struct rte_mbuf *m = tx_pkts[nb_tx]; + +#ifdef RTE_LIBRTE_ETHDEV_DEBUG + error =rte_validate_tx_offload (m); +if (unlikely (error)) { + rte_errno = -error; +break ; + } +#endif +/* Do VLAN tag insertion */ +if (unlikely (m->ol_flags &RTE_MBUF_F_TX_VLAN )) { + error =rte_vlan_insert (&m); +/* rte_vlan_insert() may change pointer */ + tx_pkts[nb_tx] = m; + +if (unlikely (error)) { + rte_errno = -error; +break ; + } + } + } + +return nb_tx; +} +static uint16_t eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) {
@@ -740,6 +780,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_prepare = eth_ioring_tx_prepare; dev->tx_pkt_burst = eth_ioring_tx;return 0 ;
Signature · 1 line
2.45.2
PATCH v2 8/8 net/ioring: implement statistics
In reply to Stephen HemmingerAdd support for basic statistics
--- doc/guides/nics/features/ioring.ini | 2 + drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
@@ -9,6 +9,8 @@ MTU update = Y Promiscuous mode = Y Allmulticast mode = Y Multiprocess aware = Y +Basic stats = Y +Stats per queue = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0
69 lines not on this page · raw message
Signature · 1 line
2.45.2
v3 cover ioring PMD device
In reply to Stephen HemmingerThis is a new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
v3 - add multi-segment support
review feedback
Stephen Hemminger (9):
net/ioring: introduce new driver
net/ioring: implement link state
net/ioring: implement control functions
net/ioring: implement management functions
net/ioring: implement secondary process support
net/ioring: implement receive and transmit
net/ioring: add VLAN support
net/ioring: implement statistics
net/ioring: support multi-segment Rx and Tx
doc/guides/nics/features/ioring.ini | 16 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 ++ drivers/net/ioring/meson.build | 15 + drivers/net/ioring/rte_eth_ioring.c | 1129 +++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 1228 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.47.2
PATCH v3 1/9 net/ioring: introduce new driver
Add basic driver initialization, documentation, and device creation and basic documentation.
--- doc/guides/nics/features/ioring.ini | 9 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 +++++++ drivers/net/ioring/meson.build | 15 ++ drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 6 files changed, 354 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
doc/guides/nics/features/ioring.ini new file +9 -0
@@ -0,0 +1,9 @@ +; +; Supported features of the 'ioring' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -41,6 +41,7 @@ Network Interface Controller Drivers igc intel_vf ionic + ioring ipn3ke ixgbe mana
doc/guides/nics/ioring.rst new file +66 -0
66 lines not on this page · raw message
drivers/net/ioring/meson.build new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +dep = dependency('liburing', required:false) +reason = 'missing dependency, "liburing"' +build = dep.found() +ext_deps += dep + +sources = files('rte_eth_ioring.c') +require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0
262 lines not on this page · raw message
drivers/net/meson.build +1 -0
@@ -34,6 +34,7 @@ drivers = [ 'intel/ixgbe', 'intel/cpfl', # depends on idpf, so must come after it 'ionic', + 'ioring', 'mana', 'memif', 'mlx4',
Signature · 1 line
2.47.2
PATCH v3 2/9 net/ioring: implement link state
In reply to Stephen HemmingerAdd hooks to set kernel link up/down and report state.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0
104 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v3 3/9 net/ioring: implement control functions
In reply to Stephen HemmingerThese internal ops, just force changes to kernel visible net device.
--- doc/guides/nics/features/ioring.ini | 3 ++ drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
@@ -5,6 +5,9 @@ ; [Features] Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0
93 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v3 4/9 net/ioring: implement management functions
In reply to Stephen HemmingerAdd start, stop, configure and info functions.
--- drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
@@ -212,12 +212,16 @@ static int eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu) {struct pmd_internals *pmd = dev->data->dev_private; + +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; +struct ifreq ifr = { .ifr_mtu = mtu }; -int ret;strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); - ret =ioctl (pmd->ctl_sock,SIOCSIFMTU , &ifr); +int ret =ioctl (sock,SIOCSIFMTU , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFMTU) failed: %s" ,strerror (errno)); ret = -errno;
@@ -230,14 +234,17 @@ static int eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr) { struct pmd_internals *pmd = dev->data->dev_private; -struct ifreq ifr = { }; -int ret; +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; + +struct ifreq ifr = { };strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); ifr.ifr_hwaddr.sa_family =ARPHRD_ETHER ;memcpy (ifr.ifr_hwaddr.sa_data, addr,sizeof (*addr)); - ret =ioctl (pmd->ctl_sock,SIOCSIFHWADDR , &ifr); +int ret =ioctl (sock,SIOCSIFHWADDR , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFHWADDR) failed: %s" ,strerror (errno)); ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)return ret; } +static int +eth_dev_start (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ; +eth_dev_set_link_up (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + } + +return 0 ; +} + +static int +eth_dev_stop (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +eth_dev_set_link_down (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + } + +return 0 ; +} + +static int +eth_dev_configure (struct rte_eth_dev *dev) +{ +/* rx/tx must be paired */ +if (dev->data->nb_rx_queues != dev->data->nb_tx_queues) +return -EINVAL ; + +return 0 ; +} + +static int +eth_dev_info (struct rte_eth_dev *dev,struct rte_eth_dev_info *dev_info) +{ +struct pmd_internals *pmd = dev->data->dev_private; + + dev_info->if_index =if_nametoindex (pmd->ifname); + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + +return 0 ; +} +static int eth_dev_close (struct rte_eth_dev *dev) {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)close (pmd->keep_fd); pmd->keep_fd =-1 ; } -return 0 ; }static const struct eth_dev_ops ops = { + .dev_start = eth_dev_start, + .dev_stop = eth_dev_stop, + .dev_configure = eth_dev_configure, + .dev_infos_get = eth_dev_info, .dev_close = eth_dev_close, .link_update = eth_link_update, .dev_set_link_up = eth_dev_set_link_up,
Signature · 1 line
2.47.2
PATCH v3 5/9 net/ioring: implement secondary process support
In reply to Stephen HemmingerAdd support for communicating fd's from primary to secondary.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++- 2 files changed, 135 insertions(+), 2 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
@@ -8,6 +8,7 @@ Link status = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y +Multiprocess aware = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +134 -2
181 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v3 6/9 net/ioring: implement receive and transmit
In reply to Stephen HemmingerUse io_uring to read and write from TAP device.
--- drivers/net/ioring/rte_eth_ioring.c | 362 +++++++++++++++++++++++++++- 1 file changed, 361 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +361 -1
@@ -2,6 +2,7 @@* Copyright (c) Stephen Hemminger */ +#include <assert.h> #include <ctype.h> #include <errno.h> #include <fcntl.h>
@@ -9,8 +10,10 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <liburing.h> #include <sys/ioctl.h> #include <sys/socket.h> +#include <sys/uio.h> #include <net/if.h> #include <linux/if.h> #include <linux/if_arp.h>
@@ -27,6 +30,12 @@#include <rte_kvargs.h> #include <rte_log.h> +#define IORING_DEFAULT_BURST 64 +#define IORING_NUM_BUFFERS 1024 +#define IORING_MAX_QUEUES 128 + +static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -34,6 +43,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);#define RTE_LOGTYPE_IORING ioring_logtype#define PMD_LOG (level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__) +#ifdef RTE_ETHDEV_DEBUG_RX +#define PMD_RX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__) +#else +#define PMD_RX_LOG (...) do { } while (0) +#endif + +#ifdef RTE_ETHDEV_DEBUG_TX +#define PMD_TX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__) +#else +#define PMD_TX_LOG (...) do { } while (0) +#endif +#define IORING_IFACE_ARG "iface"#define IORING_PERSIST_ARG "persist"
@@ -43,6 +66,30 @@ static const char * const valid_arguments[] = {NULL }; +struct rx_queue { +struct rte_mempool *mb_pool;/* rx buffer pool */ +struct io_uring io_ring;/* queue of posted read's */ +uint16_t port_id; +uint16_t queue_id; + +uint64_t rx_packets; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_errors; +}; + +struct tx_queue { +struct io_uring io_ring; + +uint16_t port_id; +uint16_t queue_id; +uint16_t free_thresh; + +uint64_t tx_packets; +uint64_t tx_bytes; +uint64_t tx_errors; +}; +struct pmd_internals {int keep_fd;/* keep alive file descriptor */ char ifname[IFNAMSIZ ];/* name assigned by kernel */
@@ -300,6 +347,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->if_index =if_nametoindex (pmd->ifname); dev_info->max_mac_addrs =1 ; dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + dev_info->max_rx_queues =IORING_MAX_QUEUES ; + dev_info->max_tx_queues =IORING_MAX_QUEUES ; + dev_info->min_rx_bufsize =0 ; + + dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { + .burst_size =IORING_DEFAULT_BURST , + .ring_size =IORING_NUM_BUFFERS , + .nb_queues =1 , + };return 0 ; }
@@ -311,6 +367,14 @@ eth_dev_close(struct rte_eth_dev *dev)PMD_LOG (INFO ,"Closing %s" , pmd->ifname); +int *fds = dev->process_private; +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +if (fds[i] ==-1 ) +continue ; +close (fds[i]); + fds[i] =-1 ; + } +if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -324,6 +388,295 @@ eth_dev_close(struct rte_eth_dev *dev)return 0 ; } +/* Setup another fd to TAP device for the queue */ +static int +eth_queue_setup (struct rte_eth_dev *dev,const char *name,uint16_t queue_id) +{ +int *fds = dev->process_private; + +if (fds[queue_id] !=-1 ) +return 0 ;/* already setup */ + +struct ifreq ifr = { }; +int tap_fd =tap_open (name, &ifr,0 ); +if (tap_fd <0 ) { +PMD_LOG (ERR ,"tap_open failed" ); +return -1 ; + } + +PMD_LOG (DEBUG ,"opened %d for queue %u" , tap_fd, queue_id); + fds[queue_id] = tap_fd; +return 0 ; +} + +static int +eth_queue_fd (uint16_t port_id,uint16_t queue_id) +{ +struct rte_eth_dev *dev = &rte_eth_devices[port_id]; +int *fds = dev->process_private; + +return fds[queue_id]; +} + +/* setup an submit queue to read mbuf */ +static inline void +eth_rx_submit (struct rx_queue *rxq,int fd,struct rte_mbuf *mb) +{ +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); + +if (unlikely (sqe ==NULL )) { +PMD_LOG (DEBUG ,"io_uring no rx sqe" ); + rxq->rx_errors++; +rte_pktmbuf_free (mb); + }else { + void *base = rte_pktmbuf_mtod(mb, void *); +size_t len = mb->buf_len; + +io_uring_prep_read (sqe, fd, base, len,0 ); +io_uring_sqe_set_data (sqe, mb); + } +} + +static uint16_t +eth_ioring_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct rx_queue *rxq = queue; +struct io_uring_cqe *cqe; +unsigned int head, num_cqe =0 ; +uint16_t num_rx =0 ; +uint32_t num_bytes =0 ; +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +io_uring_for_each_cqe (&rxq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; +ssize_t len = cqe->res; + +PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); + num_cqe++; + +if (unlikely (len <RTE_ETHER_HDR_LEN )) { +if (len <0 ) +PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len)); +else +PMD_LOG (ERR ,"io_uring_read missing hdr" ); + + rxq->rx_errors++; + goto resubmit; + } + +struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (nmb ==0 )) { +PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); + ++rxq->rx_nombuf; + goto resubmit; + } + + mb->pkt_len = len; + mb->data_len = len; + mb->port = rxq->port_id; +__rte_mbuf_sanity_check (mb,1 ); + + num_bytes += len; + bufs[num_rx++] = mb; + + mb = nmb; +resubmit: +eth_rx_submit (rxq, fd, mb); + +if (num_rx == nb_pkts) +break ; + } +io_uring_cq_advance (&rxq->io_ring, num_cqe); + + rxq->rx_packets += num_rx; + rxq->rx_bytes += num_bytes; +return num_rx; +} + +static int +eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, +unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf __rte_unused, +struct rte_mempool *mb_pool) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +PMD_LOG (DEBUG ,"setup port %u queue %u rx_descriptors %u" , + dev->data->port_id, queue_id, nb_rx_desc); + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct rx_queue *rxq =rte_zmalloc_socket (NULL ,sizeof (*rxq), +RTE_CACHE_LINE_SIZE , socket_id); +if (rxq ==NULL ) { +PMD_LOG (ERR ,"rxq alloc failed" ); +return -1 ; + } + + rxq->mb_pool = mb_pool; + rxq->port_id = dev->data->port_id; + rxq->queue_id = queue_id; + dev->data->rx_queues[queue_id] = rxq; + +if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +struct rte_mbuf **mbufs =alloca (nb_rx_desc *sizeof (struct rte_mbuf *)); +if (mbufs ==NULL ) { +PMD_LOG (ERR ,"alloca for %u failed" , nb_rx_desc); +return -1 ; + } + +if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { +PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); +return -1 ; + } + +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); +for (uint16_t i =0 ; i < nb_rx_desc; i++) +eth_rx_submit (rxq, fd, mbufs[i]); + +io_uring_submit (&rxq->io_ring); +return 0 ; +} + +static void +eth_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct rx_queue *rxq = dev->data->rx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&rxq->io_ring,1 ); + } + +io_uring_queue_exit (&rxq->io_ring); +} + +static int +eth_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct tx_queue *txq =rte_zmalloc_socket (NULL ,sizeof (*txq), +RTE_CACHE_LINE_SIZE , socket_id); +if (txq ==NULL ) { +PMD_LOG (ERR ,"txq alloc failed" ); +return -1 ; + } + + txq->port_id = dev->data->port_id; + txq->queue_id = queue_id; + txq->free_thresh = tx_conf->tx_free_thresh; + dev->data->tx_queues[queue_id] = txq; + +if (io_uring_queue_init (nb_tx_desc, &txq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static void +eth_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct tx_queue *txq = dev->data->tx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&txq->io_ring,1 ); + } + +io_uring_queue_exit (&txq->io_ring); +} + +static void +eth_ioring_tx_cleanup (struct tx_queue *txq) +{ +struct io_uring_cqe *cqe; +unsigned int head; +unsigned int tx_done =0 ; +uint64_t tx_bytes =0 ; + +io_uring_for_each_cqe (&txq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; + +PMD_TX_LOG (DEBUG ," mbuf len %u result: %d" , mb->pkt_len, cqe->res); +if (unlikely (cqe->res <0 )) { + ++txq->tx_errors; + }else { + ++tx_done; + tx_bytes += mb->pkt_len; + } + +rte_pktmbuf_free (mb); + } +io_uring_cq_advance (&txq->io_ring, tx_done); + + txq->tx_packets += tx_done; + txq->tx_bytes += tx_bytes; +} + +static uint16_t +eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct tx_queue *txq = queue; +uint16_t num_tx; + +if (unlikely (nb_pkts ==0 )) +return 0 ; + +PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); + +if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh) +eth_ioring_tx_cleanup (txq); + +int fd =eth_queue_fd (txq->port_id, txq->queue_id); + +for (num_tx =0 ; num_tx < nb_pkts; num_tx++) { +struct rte_mbuf *mb = bufs[num_tx]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) +break ;/* submit ring is full */ + +io_uring_sqe_set_data (sqe, mb); + +if (rte_mbuf_refcnt_read (mb) ==1 && +RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { + void *base = rte_pktmbuf_mtod(mb, void *); +io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); + +PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); + }else { +PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); + ++txq->tx_errors; +continue ; + } + } +if (num_tx >0 ) +io_uring_submit (&txq->io_ring); + +return num_tx; +} +static const struct eth_dev_ops ops = { .dev_start = eth_dev_start, .dev_stop = eth_dev_stop,
@@ -339,9 +692,12 @@ static const struct eth_dev_ops ops = { .promiscuous_disable = eth_dev_promiscuous_disable, .allmulticast_enable = eth_dev_allmulticast_enable, .allmulticast_disable = eth_dev_allmulticast_disable, + .rx_queue_setup = eth_rx_queue_setup, + .rx_queue_release = eth_rx_queue_release, + .tx_queue_setup = eth_tx_queue_setup, + .tx_queue_release = eth_tx_queue_release, }; -static int ioring_create (struct rte_eth_dev *dev,const char *tap_name,uint8_t persist) {
@@ -379,6 +735,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) }PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); + + dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_burst = eth_ioring_tx; +return 0 ; error:
Signature · 1 line
2.47.2
PATCH v3 7/9 net/ioring: add VLAN support
In reply to Stephen HemmingerAdd support for VLAN insert and stripping.
--- drivers/net/ioring/rte_eth_ioring.c | 47 +++++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 3 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +44 -3
@@ -33,9 +33,11 @@#define IORING_DEFAULT_BURST 64#define IORING_NUM_BUFFERS 1024#define IORING_MAX_QUEUES 128 -static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_TX_OFFLOAD RTE_ETH_TX_OFFLOAD_VLAN_INSERT +#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_VLAN_STRIP +#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -69,6 +71,7 @@ static const char * const valid_arguments[] = {struct rx_queue {struct rte_mempool *mb_pool;/* rx buffer pool */ struct io_uring io_ring;/* queue of posted read's */ +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -80,6 +83,7 @@ struct rx_queue {struct tx_queue {struct io_uring io_ring; +uint64_t offloads;uint16_t port_id;uint16_t queue_id;
@@ -471,6 +475,9 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts) goto resubmit; } +if (rxq->offloads &RTE_ETH_RX_OFFLOAD_VLAN_STRIP ) +rte_vlan_strip (mb); + mb->pkt_len = len; mb->data_len = len; mb->port = rxq->port_id;
@@ -495,8 +502,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)static int eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, -unsigned int socket_id, - const struct rte_eth_rxconf *rx_conf __rte_unused, +unsigned int socket_id,const struct rte_eth_rxconf *rx_conf,struct rte_mempool *mb_pool) {struct pmd_internals *pmd = dev->data->dev_private;
@@ -515,6 +521,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_dereturn -1 ; } + rxq->offloads = rx_conf->offloads; rxq->mb_pool = mb_pool; rxq->port_id = dev->data->port_id; rxq->queue_id = queue_id;
@@ -580,6 +587,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, txq->port_id = dev->data->port_id; txq->queue_id = queue_id; + txq->offloads = tx_conf->offloads; txq->free_thresh = tx_conf->tx_free_thresh; dev->data->tx_queues[queue_id] = txq;
@@ -634,6 +642,38 @@ eth_ioring_tx_cleanup(struct tx_queue *txq) txq->tx_bytes += tx_bytes; } +static uint16_t +eth_ioring_tx_prepare(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) +{ +uint16_t nb_tx; +int error; + +for (nb_tx =0 ; nb_tx < nb_pkts; nb_tx++) { +struct rte_mbuf *m = tx_pkts[nb_tx]; + +#ifdef RTE_LIBRTE_ETHDEV_DEBUG + error =rte_validate_tx_offload (m); +if (unlikely (error)) { + rte_errno = -error; +break ; + } +#endif +/* Do VLAN tag insertion */ +if (unlikely (m->ol_flags &RTE_MBUF_F_TX_VLAN )) { + error =rte_vlan_insert (&m); +/* rte_vlan_insert() may change pointer */ + tx_pkts[nb_tx] = m; + +if (unlikely (error)) { + rte_errno = -error; +break ; + } + } + } + +return nb_tx; +} +static uint16_t eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) {
@@ -737,6 +777,7 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_prepare = eth_ioring_tx_prepare; dev->tx_pkt_burst = eth_ioring_tx;return 0 ;
Signature · 1 line
2.47.2
PATCH v3 8/9 net/ioring: implement statistics
In reply to Stephen HemmingerAdd support for basic statistics
--- doc/guides/nics/features/ioring.ini | 2 + drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
@@ -9,6 +9,8 @@ MTU update = Y Promiscuous mode = Y Allmulticast mode = Y Multiprocess aware = Y +Basic stats = Y +Stats per queue = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0
69 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v3 9/9 net/ioring: support multi-segment Rx and Tx
In reply to Stephen HemmingerUse readv/writev to handle multi-segment transmit and receive. Account for virtio header that will be used for offload (later).
--- drivers/net/ioring/rte_eth_ioring.c | 140 ++++++++++++++++++++-------- 1 file changed, 102 insertions(+), 38 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +102 -38
@@ -18,6 +18,7 @@#include <linux/if.h> #include <linux/if_arp.h> #include <linux/if_tun.h> +#include <linux/virtio_net.h> #include <bus_vdev_driver.h> #include <ethdev_driver.h>
@@ -35,8 +36,11 @@#define IORING_MAX_QUEUES 128static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); -#define IORING_TX_OFFLOAD RTE_ETH_TX_OFFLOAD_VLAN_INSERT -#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_VLAN_STRIP +#define IORING_TX_OFFLOAD (RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ + RTE_ETH_TX_OFFLOAD_MULTI_SEGS) + +#define IORING_RX_OFFLOAD (RTE_ETH_RX_OFFLOAD_VLAN_STRIP | \ + RTE_ETH_RX_OFFLOAD_SCATTER)#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -166,7 +170,7 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist) goto error; } -int flags =IFF_TAP |IFF_MULTI_QUEUE |IFF_NO_PI ; +int flags =IFF_TAP |IFF_MULTI_QUEUE |IFF_NO_PI |IFF_VNET_HDR ;if ((features & flags) != flags) {PMD_LOG (ERR ,"TUN features %#x missing support for %#x" , features, features & flags);
@@ -354,6 +358,8 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->max_rx_queues =IORING_MAX_QUEUES ; dev_info->max_tx_queues =IORING_MAX_QUEUES ; dev_info->min_rx_bufsize =0 ; + dev_info->tx_queue_offload_capa =IORING_TX_OFFLOAD ; + dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa; dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { .burst_size =IORING_DEFAULT_BURST ,
@@ -487,13 +493,44 @@ eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb)PMD_LOG (DEBUG ,"io_uring no rx sqe" ); rxq->rx_errors++;rte_pktmbuf_free (mb); - }else { - void *base = rte_pktmbuf_mtod(mb, void *); -size_t len = mb->buf_len; +return ; + } -io_uring_prep_read (sqe, fd, base, len,0 ); -io_uring_sqe_set_data (sqe, mb); +RTE_VERIFY (mb->nb_segs <IOV_MAX ); +struct iovec iovs[IOV_MAX ]; + +for (uint16_t i =0 ; i < mb->nb_segs; i++) { + iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *); + iovs[i].iov_len =rte_pktmbuf_tailroom (mb); + mb = mb->next; } +io_uring_sqe_set_data (sqe, mb); +io_uring_prep_readv (sqe, fd, iovs, mb->nb_segs,0 ); +} + +static struct rte_mbuf * +eth_ioring_rx_alloc (struct rx_queue *rxq) +{ +const struct rte_eth_dev *dev = &rte_eth_devices[rxq->port_id]; +int buf_size = dev->data->mtu +sizeof (struct virtio_net_hdr ); +struct rte_mbuf *m =NULL ; +struct rte_mbuf **tail = &m; + +do { +struct rte_mbuf *seg =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (seg ==NULL )) { +rte_pktmbuf_free (m); +return NULL ; + } + *tail = seg; + tail = &seg->next; +if (seg != m) + ++m->nb_segs; + + buf_size -=rte_pktmbuf_tailroom (seg); + }while (buf_size >0 ); + +return m; } static uint16_t
@@ -513,7 +550,8 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); num_cqe++; -if (unlikely (len <RTE_ETHER_HDR_LEN )) { +struct virtio_net_hdr *hdr; +if (unlikely (len < (ssize_t )(sizeof (*hdr) +RTE_ETHER_HDR_LEN ))) {if (len <0 )PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len));else
@@ -523,19 +561,31 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts) goto resubmit; } -struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); -if (unlikely (nmb ==0 )) { -PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); + hdr = rte_pktmbuf_mtod(mb, struct virtio_net_hdr *); + +struct rte_mbuf *nmb =eth_ioring_rx_alloc (rxq); +if (!nmb) { ++rxq->rx_nombuf; goto resubmit; } -if (rxq->offloads &RTE_ETH_RX_OFFLOAD_VLAN_STRIP ) -rte_vlan_strip (mb); + len -=sizeof (*hdr); + mb->data_off +=sizeof (*hdr); mb->pkt_len = len; - mb->data_len = len; mb->port = rxq->port_id; +struct rte_mbuf *seg = mb; +for (;;) { + seg->data_len =RTE_MIN (len, mb->buf_len); + seg = seg->next; + len -= seg->data_len; + }while (len >0 && seg !=NULL ); + +RTE_VERIFY (!(seg ==NULL && len >0 )); + +if (rxq->offloads &RTE_ETH_RX_OFFLOAD_VLAN_STRIP ) +rte_vlan_strip (mb); +__rte_mbuf_sanity_check (mb,1 ); num_bytes += len;
@@ -555,6 +605,7 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)return num_rx; } + static int eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_desc, unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,
@@ -587,20 +638,17 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_dereturn -1 ; } -struct rte_mbuf **mbufs =alloca (nb_rx_desc *sizeof (struct rte_mbuf *)); -if (mbufs ==NULL ) { -PMD_LOG (ERR ,"alloca for %u failed" , nb_rx_desc); -return -1 ; - } +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); -if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { -PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); -return -1 ; - } +for (uint16_t i =0 ; i < nb_rx_desc; i++) { +struct rte_mbuf *mb =eth_ioring_rx_alloc (rxq); +if (mb ==NULL ) { +PMD_LOG (ERR ,"Rx mbuf alloc buf failed" ); +return -1 ; + } -int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); -for (uint16_t i =0 ; i < nb_rx_desc; i++) -eth_rx_submit (rxq, fd, mbufs[i]); +eth_rx_submit (rxq, fd, mb); + }io_uring_submit (&rxq->io_ring);return 0 ;
@@ -740,31 +788,47 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); -if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh) +if (likely (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh))eth_ioring_tx_cleanup (txq);int fd =eth_queue_fd (txq->port_id, txq->queue_id);for (num_tx =0 ; num_tx < nb_pkts; num_tx++) {struct rte_mbuf *mb = bufs[num_tx]; +struct virtio_net_hdr *hdr;struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring);if (sqe ==NULL )break ;/* submit ring is full */ -io_uring_sqe_set_data (sqe, mb); +if (rte_mbuf_refcnt_read (mb) ==1 &&RTE_MBUF_DIRECT (mb) && +rte_pktmbuf_headroom (mb) >=sizeof (*hdr)) { + hdr = (struct virtio_net_hdr *)rte_pktmbuf_prepend (mb,sizeof (*hdr)); + }else { +struct rte_mbuf *mh =rte_pktmbuf_alloc (mb->pool); +if (unlikely (mh ==NULL )) +break ; + hdr = (struct virtio_net_hdr *)rte_pktmbuf_append (mh,sizeof (*hdr)); -if (rte_mbuf_refcnt_read (mb) ==1 && -RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { - void *base = rte_pktmbuf_mtod(mb, void *); -io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); + mh->next = mb; + mh->nb_segs = mb->nb_segs +1 ; + mh->pkt_len += mb->pkt_len; + mh->ol_flags = mb->ol_flags &RTE_MBUF_F_TX_OFFLOAD_MASK ; + mb = mh; + } +memset (hdr,0 ,sizeof (*hdr)); -PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); - }else { -PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); - ++txq->tx_errors; -continue ; +io_uring_sqe_set_data (sqe, mb); + +struct iovec iovs[RTE_MBUF_MAX_NB_SEGS +1 ]; +unsigned int niov = mb->nb_segs; +for (unsigned int i =0 ; i < niov; i++) { + iovs[i].iov_base = rte_pktmbuf_mtod(mb, char *); + iovs[i].iov_len = mb->data_len; + mb = mb->next; } + +io_uring_prep_writev (sqe, fd, iovs, niov,0 ); }if (num_tx >0 )io_uring_submit (&txq->io_ring);
Signature · 1 line
2.47.2
v4 cover new ioring PMD
In reply to Stephen HemmingerThis is a new simplified TAP device that uses the Linux kernel ioring API to provide a read/write ring with kernel.
This is split from tap device because there are so many unnecessary things in existing tap, and supporting ioring is better without ifdefs etc. The default name of the tap device is different that other uses in DPDK but the driver tries to keep the same relevant devargs as before.
This driver will only provide features that match what kernel does, so no flow support etc. The next version will add checksum and multi-segment packets. Some of the doc files may need update as well.
v4 - more testing and offload support
Stephen Hemminger (10):
net/ioring: introduce new driver net/ioring: implement link state net/ioring: implement control functions net/ioring: implement management functions net/ioring: implement secondary process support net/ioring: implement receive and transmit net/ioring: implement statistics net/ioring: support multi-segment Rx and Tx net/ioring: support Tx checksum and segment offload net/ioring: add support for Rx offload
MAINTAINERS | 6 + doc/guides/nics/features/ioring.ini | 18 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 60 ++ drivers/net/ioring/meson.build | 15 + drivers/net/ioring/rte_eth_ioring.c | 1288 +++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 7 files changed, 1389 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
Signature · 1 line
2.47.2
PATCH v4 1/10 net/ioring: introduce new driver
Add basic driver initialization, documentation, and device creation and basic documentation.
--- MAINTAINERS | 6 + doc/guides/nics/features/ioring.ini | 9 + doc/guides/nics/index.rst | 1 + doc/guides/nics/ioring.rst | 66 +++++++ drivers/net/ioring/meson.build | 15 ++ drivers/net/ioring/rte_eth_ioring.c | 262 ++++++++++++++++++++++++++++ drivers/net/meson.build | 1 + 7 files changed, 360 insertions(+) create mode 100644 doc/guides/nics/features/ioring.ini create mode 100644 doc/guides/nics/ioring.rst create mode 100644 drivers/net/ioring/meson.build create mode 100644 drivers/net/ioring/rte_eth_ioring.c
MAINTAINERS +6 -0
@@ -853,6 +853,12 @@ F: drivers/net/intel/ipn3ke/ F: doc/guides/nics/ipn3ke.rst F: doc/guides/nics/features/ipn3ke.ini +Ioring - EXPERIMENTAL +M: Stephen Hemminger <[email protected]> +F: drivers/net/ioring/ +F: doc/guides/nics/ioring.rst +F: doc/guides/nics/features/ioring.ini + Marvell cnxk M: Nithin Dabilpuram <[email protected]> M: Kiran Kumar K <[email protected]>
doc/guides/nics/features/ioring.ini new file +9 -0
@@ -0,0 +1,9 @@ +; +; Supported features of the 'ioring' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -41,6 +41,7 @@ Network Interface Controller Drivers igc intel_vf ionic + ioring ipn3ke ixgbe mana
doc/guides/nics/ioring.rst new file +66 -0
66 lines not on this page · raw message
drivers/net/ioring/meson.build new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +dep = dependency('liburing', required:false) +reason = 'missing dependency, "liburing"' +build = dep.found() +ext_deps += dep + +sources = files('rte_eth_ioring.c') +require_iova_in_mbuf = false
drivers/net/ioring/rte_eth_ioring.c new file +262 -0
262 lines not on this page · raw message
drivers/net/meson.build +1 -0
@@ -34,6 +34,7 @@ drivers = [ 'intel/ixgbe', 'intel/cpfl', # depends on idpf, so must come after it 'ionic', + 'ioring', 'mana', 'memif', 'mlx4',
Signature · 1 line
2.47.2
PATCH v4 2/10 net/ioring: implement link state
In reply to Stephen HemmingerAdd hooks to set kernel link up/down and report state.
--- doc/guides/nics/features/ioring.ini | 1 + drivers/net/ioring/rte_eth_ioring.c | 84 +++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)
doc/guides/nics/features/ioring.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +84 -0
104 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v4 3/10 net/ioring: implement control functions
In reply to Stephen HemmingerThese internal ops, just force changes to kernel visible net device.
--- doc/guides/nics/features/ioring.ini | 3 ++ drivers/net/ioring/rte_eth_ioring.c | 69 +++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+)
doc/guides/nics/features/ioring.ini +3 -0
@@ -5,6 +5,9 @@ ; [Features] Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +69 -0
93 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v4 4/10 net/ioring: implement management functions
In reply to Stephen HemmingerAdd start, stop, configure and info functions.
--- drivers/net/ioring/rte_eth_ioring.c | 72 ++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 6 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +66 -6
@@ -212,12 +212,16 @@ static int eth_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu) {struct pmd_internals *pmd = dev->data->dev_private; + +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; +struct ifreq ifr = { .ifr_mtu = mtu }; -int ret;strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); - ret =ioctl (pmd->ctl_sock,SIOCSIFMTU , &ifr); +int ret =ioctl (sock,SIOCSIFMTU , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFMTU) failed: %s" ,strerror (errno)); ret = -errno;
@@ -230,14 +234,17 @@ static int eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr) { struct pmd_internals *pmd = dev->data->dev_private; -struct ifreq ifr = { }; -int ret; +int sock =socket (AF_INET ,SOCK_DGRAM ,0 ); +if (sock <0 ) +return -errno; + +struct ifreq ifr = { };strlcpy (ifr.ifr_name, pmd->ifname,IFNAMSIZ ); ifr.ifr_hwaddr.sa_family =ARPHRD_ETHER ;memcpy (ifr.ifr_hwaddr.sa_data, addr,sizeof (*addr)); - ret =ioctl (pmd->ctl_sock,SIOCSIFHWADDR , &ifr); +int ret =ioctl (sock,SIOCSIFHWADDR , &ifr);if (ret <0 ) {PMD_LOG (ERR ,"ioctl(SIOCSIFHWADDR) failed: %s" ,strerror (errno)); ret = -errno;
@@ -246,6 +253,56 @@ eth_dev_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)return ret; } +static int +eth_dev_start (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ; +eth_dev_set_link_up (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ; + } + +return 0 ; +} + +static int +eth_dev_stop (struct rte_eth_dev *dev) +{ + dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +eth_dev_set_link_down (dev); + +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { + dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + dev->data->tx_queue_state[i] =RTE_ETH_QUEUE_STATE_STOPPED ; + } + +return 0 ; +} + +static int +eth_dev_configure (struct rte_eth_dev *dev) +{ +/* rx/tx must be paired */ +if (dev->data->nb_rx_queues != dev->data->nb_tx_queues) +return -EINVAL ; + +return 0 ; +} + +static int +eth_dev_info (struct rte_eth_dev *dev,struct rte_eth_dev_info *dev_info) +{ +struct pmd_internals *pmd = dev->data->dev_private; + + dev_info->if_index =if_nametoindex (pmd->ifname); + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + +return 0 ; +} +static int eth_dev_close (struct rte_eth_dev *dev) {
@@ -263,11 +320,14 @@ eth_dev_close(struct rte_eth_dev *dev)close (pmd->keep_fd); pmd->keep_fd =-1 ; } -return 0 ; }static const struct eth_dev_ops ops = { + .dev_start = eth_dev_start, + .dev_stop = eth_dev_stop, + .dev_configure = eth_dev_configure, + .dev_infos_get = eth_dev_info, .dev_close = eth_dev_close, .link_update = eth_link_update, .dev_set_link_up = eth_dev_set_link_up,
Signature · 1 line
2.47.2
PATCH v4 5/10 net/ioring: implement secondary process support
In reply to Stephen HemmingerAdd support for communicating fd's from primary to secondary.
--- doc/guides/nics/features/ioring.ini | 1 + doc/guides/nics/ioring.rst | 6 -- drivers/net/ioring/rte_eth_ioring.c | 136 +++++++++++++++++++++++++++- 3 files changed, 135 insertions(+), 8 deletions(-)
doc/guides/nics/features/ioring.ini +1 -0
@@ -8,6 +8,7 @@ Link status = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y +Multiprocess aware = Y Linux = Y x86-64 = Y Usage doc = Y
doc/guides/nics/ioring.rst +0 -6
@@ -46,12 +46,6 @@ But this behavior can be overridden by the use of the persist flag, example:: --vdev=net_ioring0,iface=io0,persist ... -Multi-process sharing ---------------------- - -The IORING device does not support secondary process (yet). - - Limitations -----------
drivers/net/ioring/rte_eth_ioring.c +134 -2
181 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v4 6/10 net/ioring: implement receive and transmit
In reply to Stephen HemmingerUse io_uring to read and write from TAP device.
--- drivers/net/ioring/rte_eth_ioring.c | 366 +++++++++++++++++++++++++++- 1 file changed, 365 insertions(+), 1 deletion(-)
drivers/net/ioring/rte_eth_ioring.c +365 -1
@@ -2,6 +2,7 @@* Copyright (c) Stephen Hemminger */ +#include <assert.h> #include <ctype.h> #include <errno.h> #include <fcntl.h>
@@ -9,8 +10,10 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <liburing.h> #include <sys/ioctl.h> #include <sys/socket.h> +#include <sys/uio.h> #include <net/if.h> #include <linux/if.h> #include <linux/if_arp.h>
@@ -27,6 +30,12 @@#include <rte_kvargs.h> #include <rte_log.h> +#define IORING_DEFAULT_BURST 64 +#define IORING_NUM_BUFFERS 1024 +#define IORING_MAX_QUEUES 128 + +static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -34,6 +43,20 @@ RTE_LOG_REGISTER_DEFAULT(ioring_logtype, NOTICE);#define RTE_LOGTYPE_IORING ioring_logtype#define PMD_LOG (level, ...) RTE_LOG_LINE_PREFIX(level, IORING, "%s(): ", __func__, __VA_ARGS__) +#ifdef RTE_ETHDEV_DEBUG_RX +#define PMD_RX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() rx: ", __func__, __VA_ARGS__) +#else +#define PMD_RX_LOG (...) do { } while (0) +#endif + +#ifdef RTE_ETHDEV_DEBUG_TX +#define PMD_TX_LOG (level, ...) \ + RTE_LOG_LINE_PREFIX(level, IORING, "%s() tx: ", __func__, __VA_ARGS__) +#else +#define PMD_TX_LOG (...) do { } while (0) +#endif +#define IORING_IFACE_ARG "iface"#define IORING_PERSIST_ARG "persist"
@@ -43,6 +66,30 @@ static const char * const valid_arguments[] = {NULL }; +struct rx_queue { +struct rte_mempool *mb_pool;/* rx buffer pool */ +struct io_uring io_ring;/* queue of posted read's */ +uint16_t port_id; +uint16_t queue_id; + +uint64_t rx_packets; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_errors; +}; + +struct tx_queue { +struct io_uring io_ring; + +uint16_t port_id; +uint16_t queue_id; +uint16_t free_thresh; + +uint64_t tx_packets; +uint64_t tx_bytes; +uint64_t tx_errors; +}; +struct pmd_internals {int keep_fd;/* keep alive file descriptor */ char ifname[IFNAMSIZ ];/* name assigned by kernel */
@@ -300,6 +347,15 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->if_index =if_nametoindex (pmd->ifname); dev_info->max_mac_addrs =1 ; dev_info->max_rx_pktlen =RTE_ETHER_MAX_LEN ; + dev_info->max_rx_queues =IORING_MAX_QUEUES ; + dev_info->max_tx_queues =IORING_MAX_QUEUES ; + dev_info->min_rx_bufsize =0 ; + + dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { + .burst_size =IORING_DEFAULT_BURST , + .ring_size =IORING_NUM_BUFFERS , + .nb_queues =1 , + };return 0 ; }
@@ -311,6 +367,14 @@ eth_dev_close(struct rte_eth_dev *dev)PMD_LOG (INFO ,"Closing %s" , pmd->ifname); +int *fds = dev->process_private; +for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { +if (fds[i] ==-1 ) +continue ; +close (fds[i]); + fds[i] =-1 ; + } +if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -324,6 +388,299 @@ eth_dev_close(struct rte_eth_dev *dev)return 0 ; } +/* Setup another fd to TAP device for the queue */ +static int +eth_queue_setup (struct rte_eth_dev *dev,const char *name,uint16_t queue_id) +{ +int *fds = dev->process_private; + +if (fds[queue_id] !=-1 ) +return 0 ;/* already setup */ + +struct ifreq ifr = { }; +int tap_fd =tap_open (name, &ifr,0 ); +if (tap_fd <0 ) { +PMD_LOG (ERR ,"tap_open failed" ); +return -1 ; + } + +PMD_LOG (DEBUG ,"opened %d for queue %u" , tap_fd, queue_id); + fds[queue_id] = tap_fd; +return 0 ; +} + +static int +eth_queue_fd (uint16_t port_id,uint16_t queue_id) +{ +struct rte_eth_dev *dev = &rte_eth_devices[port_id]; +int *fds = dev->process_private; + +return fds[queue_id]; +} + +/* setup an submit queue to read mbuf */ +static inline void +eth_rx_submit (struct rx_queue *rxq,int fd,struct rte_mbuf *mb) +{ +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); + +if (unlikely (sqe ==NULL )) { +PMD_LOG (DEBUG ,"io_uring no rx sqe" ); + rxq->rx_errors++; +rte_pktmbuf_free (mb); +return ; + } +io_uring_sqe_set_data (sqe, mb); + + void *buf = rte_pktmbuf_mtod_offset(mb, void *, 0); +unsigned int nbytes =rte_pktmbuf_tailroom (mb); + +io_uring_prep_read (sqe, fd, buf, nbytes,0 ); +} + +static uint16_t +eth_ioring_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct rx_queue *rxq = queue; +struct io_uring_cqe *cqe; +unsigned int head, num_cqe =0 ; +uint16_t num_rx =0 ; +uint32_t num_bytes =0 ; +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); + +io_uring_for_each_cqe (&rxq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; +ssize_t len = cqe->res; + +PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); + num_cqe++; + +if (unlikely (len <RTE_ETHER_HDR_LEN )) { +if (len <0 ) +PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len)); +else +PMD_LOG (ERR ,"io_uring_read missing hdr" ); + + rxq->rx_errors++; + goto resubmit; + } + +struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (nmb ==0 )) { +PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); + ++rxq->rx_nombuf; + goto resubmit; + } + + mb->pkt_len = len; + mb->data_len = len; + mb->port = rxq->port_id; +__rte_mbuf_sanity_check (mb,1 ); + + num_bytes += len; + bufs[num_rx++] = mb; + + mb = nmb; +resubmit: +eth_rx_submit (rxq, fd, mb); + +if (num_rx == nb_pkts) +break ; + } +io_uring_cq_advance (&rxq->io_ring, num_cqe); + + rxq->rx_packets += num_rx; + rxq->rx_bytes += num_bytes; +return num_rx; +} + +static int +eth_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_rx_desc, +unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf __rte_unused, +struct rte_mempool *mb_pool) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +PMD_LOG (DEBUG ,"setup port %u queue %u rx_descriptors %u" , + dev->data->port_id, queue_id, nb_rx_desc); + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct rx_queue *rxq =rte_zmalloc_socket (NULL ,sizeof (*rxq), +RTE_CACHE_LINE_SIZE , socket_id); +if (rxq ==NULL ) { +PMD_LOG (ERR ,"rxq alloc failed" ); +return -1 ; + } + + rxq->mb_pool = mb_pool; + rxq->port_id = dev->data->port_id; + rxq->queue_id = queue_id; + dev->data->rx_queues[queue_id] = rxq; + +if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +struct rte_mbuf **mbufs =alloca (nb_rx_desc *sizeof (struct rte_mbuf *)); +if (mbufs ==NULL ) { +PMD_LOG (ERR ,"alloca for %u failed" , nb_rx_desc); +return -1 ; + } + +if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { +PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); +return -1 ; + } + +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); +for (uint16_t i =0 ; i < nb_rx_desc; i++) +eth_rx_submit (rxq, fd, mbufs[i]); + +io_uring_submit (&rxq->io_ring); +return 0 ; +} + +static void +eth_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct rx_queue *rxq = dev->data->rx_queues[queue_id]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&rxq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&rxq->io_ring,1 ); + } + +io_uring_queue_exit (&rxq->io_ring); +} + +static int +eth_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf) +{ +struct pmd_internals *pmd = dev->data->dev_private; + +/* open shared tap fd maybe already setup */ +if (eth_queue_setup (dev, pmd->ifname, queue_id) <0 ) +return -1 ; + +struct tx_queue *txq =rte_zmalloc_socket (NULL ,sizeof (*txq), +RTE_CACHE_LINE_SIZE , socket_id); +if (txq ==NULL ) { +PMD_LOG (ERR ,"txq alloc failed" ); +return -1 ; + } + + txq->port_id = dev->data->port_id; + txq->queue_id = queue_id; + txq->free_thresh = tx_conf->tx_free_thresh; + dev->data->tx_queues[queue_id] = txq; + +if (io_uring_queue_init (nb_tx_desc, &txq->io_ring,0 ) !=0 ) { +PMD_LOG (ERR ,"io_uring_queue_init failed: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static void +eth_ioring_tx_cleanup (struct tx_queue *txq) +{ +struct io_uring_cqe *cqe; +unsigned int head; +unsigned int tx_done =0 ; +uint64_t tx_bytes =0 ; + +io_uring_for_each_cqe (&txq->io_ring, head, cqe) { +struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; + +PMD_TX_LOG (DEBUG ," mbuf len %u result: %d" , mb->pkt_len, cqe->res); +if (unlikely (cqe->res <0 )) { + ++txq->tx_errors; + }else { + ++tx_done; + tx_bytes += mb->pkt_len; + } + +rte_pktmbuf_free (mb); + } +io_uring_cq_advance (&txq->io_ring, tx_done); + + txq->tx_packets += tx_done; + txq->tx_bytes += tx_bytes; +} + +static void +eth_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id) +{ +struct tx_queue *txq = dev->data->tx_queues[queue_id]; + +eth_ioring_tx_cleanup (txq); + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) { +PMD_LOG (ERR ,"io_uring_get_sqe failed: %s" ,strerror (errno)); + }else { +io_uring_prep_cancel (sqe,NULL ,IORING_ASYNC_CANCEL_ANY ); +io_uring_submit_and_wait (&txq->io_ring,1 ); + } + +io_uring_queue_exit (&txq->io_ring); +} + +static uint16_t +eth_ioring_tx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{ +struct tx_queue *txq = queue; +uint16_t num_tx; + +if (unlikely (nb_pkts ==0 )) +return 0 ; + +PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); + +if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh) +eth_ioring_tx_cleanup (txq); + +int fd =eth_queue_fd (txq->port_id, txq->queue_id); + +for (num_tx =0 ; num_tx < nb_pkts; num_tx++) { +struct rte_mbuf *mb = bufs[num_tx]; + +struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring); +if (sqe ==NULL ) +break ;/* submit ring is full */ + +io_uring_sqe_set_data (sqe, mb); + +if (rte_mbuf_refcnt_read (mb) ==1 && +RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { + void *base = rte_pktmbuf_mtod(mb, void *); +io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); + +PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); + }else { +PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); + ++txq->tx_errors; +continue ; + } + } + +if (likely (num_tx >0 )) +io_uring_submit (&txq->io_ring); + +return num_tx; +} +static const struct eth_dev_ops ops = { .dev_start = eth_dev_start, .dev_stop = eth_dev_stop,
@@ -339,9 +696,12 @@ static const struct eth_dev_ops ops = { .promiscuous_disable = eth_dev_promiscuous_disable, .allmulticast_enable = eth_dev_allmulticast_enable, .allmulticast_disable = eth_dev_allmulticast_disable, + .rx_queue_setup = eth_rx_queue_setup, + .rx_queue_release = eth_rx_queue_release, + .tx_queue_setup = eth_tx_queue_setup, + .tx_queue_release = eth_tx_queue_release, }; -static int ioring_create (struct rte_eth_dev *dev,const char *tap_name,uint8_t persist) {
@@ -379,6 +739,10 @@ ioring_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) }PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); + + dev->rx_pkt_burst = eth_ioring_rx; + dev->tx_pkt_burst = eth_ioring_tx; +return 0 ; error:
Signature · 1 line
2.47.2
PATCH v4 7/10 net/ioring: implement statistics
In reply to Stephen HemmingerAdd support for basic statistics
--- doc/guides/nics/features/ioring.ini | 2 + drivers/net/ioring/rte_eth_ioring.c | 57 +++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
doc/guides/nics/features/ioring.ini +2 -0
@@ -9,6 +9,8 @@ MTU update = Y Promiscuous mode = Y Allmulticast mode = Y Multiprocess aware = Y +Basic stats = Y +Stats per queue = Y Linux = Y x86-64 = Y Usage doc = Y
drivers/net/ioring/rte_eth_ioring.c +57 -0
69 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v4 8/10 net/ioring: support multi-segment Rx and Tx
In reply to Stephen HemmingerUse readv/writev to handle multi-segment transmit and receive. Account for virtio header that will be used for offload (later).
--- drivers/net/ioring/rte_eth_ioring.c | 203 ++++++++++++++++++++++------ 1 file changed, 160 insertions(+), 43 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +160 -43
@@ -18,6 +18,7 @@#include <linux/if.h> #include <linux/if_arp.h> #include <linux/if_tun.h> +#include <linux/virtio_net.h> #include <bus_vdev_driver.h> #include <ethdev_driver.h>
@@ -30,12 +31,18 @@#include <rte_kvargs.h> #include <rte_log.h> +static_assert (RTE_PKTMBUF_HEADROOM >=sizeof (struct virtio_net_hdr )); +#define IORING_DEFAULT_BURST 64#define IORING_NUM_BUFFERS 1024#define IORING_MAX_QUEUES 128static_assert (IORING_MAX_QUEUES <=RTE_MP_MAX_FD_NUM ,"Max queues exceeds MP fd limit" ); +#define IORING_TX_OFFLOAD RTE_ETH_TX_OFFLOAD_MULTI_SEGS + +#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_SCATTER +#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -162,7 +169,7 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist) goto error; } -int flags =IFF_TAP |IFF_MULTI_QUEUE |IFF_NO_PI ; +int flags =IFF_TAP |IFF_MULTI_QUEUE |IFF_NO_PI |IFF_VNET_HDR ;if ((features & flags) != flags) {PMD_LOG (ERR ,"TUN features %#x missing support for %#x" , features, features & flags);
@@ -193,6 +200,13 @@ tap_open(const char *name, struct ifreq *ifr, uint8_t persist) goto error; } + +int hdr_size =sizeof (struct virtio_net_hdr ); +if (ioctl (tap_fd,TUNSETVNETHDRSZ , &hdr_size) <0 ) { +PMD_LOG (ERR ,"ioctl(TUNSETVNETHDRSZ) %s" ,strerror (errno)); + goto error; + } +return tap_fd; error:close (tap_fd);
@@ -350,6 +364,8 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->max_rx_queues =IORING_MAX_QUEUES ; dev_info->max_tx_queues =IORING_MAX_QUEUES ; dev_info->min_rx_bufsize =0 ; + dev_info->tx_queue_offload_capa =IORING_TX_OFFLOAD ; + dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa; dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { .burst_size =IORING_DEFAULT_BURST ,
@@ -478,7 +494,6 @@ static inline void eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb) { struct io_uring_sqe *sqe = io_uring_get_sqe(&rxq->io_ring); -if (unlikely (sqe ==NULL )) {PMD_LOG (DEBUG ,"io_uring no rx sqe" ); rxq->rx_errors++;
@@ -487,10 +502,81 @@ eth_rx_submit(struct rx_queue *rxq, int fd, struct rte_mbuf *mb) }io_uring_sqe_set_data (sqe, mb); - void *buf = rte_pktmbuf_mtod_offset(mb, void *, 0); -unsigned int nbytes =rte_pktmbuf_tailroom (mb); +RTE_ASSERT (rte_pktmbuf_headroom (mb) >=sizeof (struct virtio_net_hdr )); + void *buf = rte_pktmbuf_mtod_offset(mb, void *, -sizeof(struct virtio_net_hdr)); +unsigned int nbytes =sizeof (struct virtio_net_hdr ) +rte_pktmbuf_tailroom (mb); + +/* optimize for the case where packet fits in one mbuf */ +if (mb->nb_segs ==1 ) { +io_uring_prep_read (sqe, fd, buf, nbytes,0 ); + }else { +uint16_t nsegs = mb->nb_segs; +RTE_ASSERT (nsegs >0 && nsegs <IOV_MAX ); +struct iovec iovs[RTE_MBUF_MAX_NB_SEGS ]; + + iovs[0 ].iov_base = buf; + iovs[0 ].iov_len = nbytes; + +for (uint16_t i =1 ; i < nsegs; i++) { + mb = mb->next; + iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *); + iovs[i].iov_len =rte_pktmbuf_tailroom (mb); + } +io_uring_prep_readv (sqe, fd, iovs, nsegs,0 ); + } + +} + + +/* Allocates one or more mbuf's to be used for reading packets */ +static struct rte_mbuf * +eth_ioring_rx_alloc (struct rx_queue *rxq) +{ +const struct rte_eth_dev *dev = &rte_eth_devices[rxq->port_id]; +int buf_size = dev->data->mtu; +struct rte_mbuf *m =NULL ; +struct rte_mbuf **tail = &m; + +do { +struct rte_mbuf *seg =rte_pktmbuf_alloc (rxq->mb_pool); +if (unlikely (seg ==NULL )) { +rte_pktmbuf_free (m); +return NULL ; + } + *tail = seg; + tail = &seg->next; +if (seg != m) + ++m->nb_segs; + + buf_size -=rte_pktmbuf_tailroom (seg); + }while (buf_size >0 ); + +__rte_mbuf_sanity_check (m,1 ); +return m; +} + + +/* set length of received mbuf segments */ +static inline void +eth_ioring_rx_adjust (struct rte_mbuf *mb,size_t len) +{ +struct rte_mbuf *seg; +unsigned int nsegs =0 ; + +for (seg = mb; seg !=NULL && len >0 ; seg = seg->next) { +uint16_t seg_len =RTE_MIN (len,rte_pktmbuf_tailroom (mb)); + + seg->data_len = seg_len; + len -= seg_len; + ++nsegs; + } -io_uring_prep_read (sqe, fd, buf, nbytes,0 ); + mb->nb_segs = nsegs; +if (len ==0 && seg !=NULL ) { +/* free any residual */ +rte_pktmbuf_free (seg->next); + seg->next =NULL ; + } } static uint16_t
@@ -505,37 +591,42 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)io_uring_for_each_cqe (&rxq->io_ring, head, cqe) {struct rte_mbuf *mb = (void *)(uintptr_t )cqe->user_data; -ssize_t len = cqe->res; +int32_t len = cqe->res; -PMD_RX_LOG (DEBUG ,"cqe %u len %zd" , num_cqe, len); - num_cqe++; +PMD_RX_LOG (DEBUG ,"complete m=%p len=%d" , mb, len); -if (unlikely (len <RTE_ETHER_HDR_LEN )) { -if (len <0 ) -PMD_LOG (ERR ,"io_uring_read: %s" ,strerror (-len)); -else -PMD_LOG (ERR ,"io_uring_read missing hdr" ); + num_cqe++; +struct virtio_net_hdr *hdr; +if (unlikely (len < (ssize_t )(sizeof (*hdr) +RTE_ETHER_HDR_LEN ))) { +PMD_LOG (ERR ,"io_uring_read result = %d" , len); rxq->rx_errors++; goto resubmit; } -struct rte_mbuf *nmb =rte_pktmbuf_alloc (rxq->mb_pool); -if (unlikely (nmb ==0 )) { -PMD_LOG (DEBUG ,"Rx mbuf alloc failed" ); +/* virtio header is before packet data */ + hdr = rte_pktmbuf_mtod_offset(mb, struct virtio_net_hdr *, -sizeof(*hdr)); + len -=sizeof (*hdr); + +struct rte_mbuf *nmb =eth_ioring_rx_alloc (rxq); +if (!nmb) { +PMD_RX_LOG (NOTICE ,"alloc failed" ); ++rxq->rx_nombuf; goto resubmit; } - mb->pkt_len = len; - mb->data_len = len; mb->port = rxq->port_id; -__rte_mbuf_sanity_check (mb,1 ); + mb->pkt_len = len; + +if (mb->nb_segs ==1 ) + mb->data_len = len; +else +eth_ioring_rx_adjust (mb, len); - num_bytes += len; + num_bytes += mb->pkt_len; bufs[num_rx++] = mb; - mb = nmb; + mb = nmb;/* use the new buffer when resubmitting */ resubmit:eth_rx_submit (rxq, fd, mb);
@@ -581,20 +672,17 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_dereturn -1 ; } -struct rte_mbuf **mbufs =alloca (nb_rx_desc *sizeof (struct rte_mbuf *)); -if (mbufs ==NULL ) { -PMD_LOG (ERR ,"alloca for %u failed" , nb_rx_desc); -return -1 ; - } +int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); -if (rte_pktmbuf_alloc_bulk (mb_pool, mbufs, nb_rx_desc) <0 ) { -PMD_LOG (ERR ,"Rx mbuf alloc %u bufs failed" , nb_rx_desc); -return -1 ; - } +for (uint16_t i =0 ; i < nb_rx_desc; i++) { +struct rte_mbuf *mb =eth_ioring_rx_alloc (rxq); +if (mb ==NULL ) { +PMD_LOG (ERR ,"Rx mbuf alloc buf failed" ); +return -1 ; + } -int fd =eth_queue_fd (rxq->port_id, rxq->queue_id); -for (uint16_t i =0 ; i < nb_rx_desc; i++) -eth_rx_submit (rxq, fd, mbufs[i]); +eth_rx_submit (rxq, fd, mb); + }io_uring_submit (&rxq->io_ring);return 0 ;
@@ -701,8 +789,6 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)if (unlikely (nb_pkts ==0 ))return 0 ; -PMD_TX_LOG (DEBUG ,"%d packets to xmit" , nb_pkts); -if (io_uring_sq_space_left (&txq->io_ring) < txq->free_thresh)eth_ioring_tx_cleanup (txq);
@@ -710,23 +796,54 @@ eth_ioring_tx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)for (num_tx =0 ; num_tx < nb_pkts; num_tx++) {struct rte_mbuf *mb = bufs[num_tx]; +struct virtio_net_hdr *hdr;struct io_uring_sqe *sqe =io_uring_get_sqe (&txq->io_ring);if (sqe ==NULL )break ;/* submit ring is full */ +if (rte_mbuf_refcnt_read (mb) ==1 && +RTE_MBUF_DIRECT (mb) && +rte_pktmbuf_headroom (mb) >=sizeof (*hdr)) { + hdr = rte_pktmbuf_mtod_offset(mb, struct virtio_net_hdr *, sizeof(*hdr)); + }else { +struct rte_mbuf *mh =rte_pktmbuf_alloc (mb->pool); +if (unlikely (mh ==NULL )) { + ++txq->tx_errors; +rte_pktmbuf_free (mb); +continue ; + } + + hdr = rte_pktmbuf_mtod_offset(mh, struct virtio_net_hdr *, sizeof(*hdr)); + mh->next = mb; + mh->nb_segs = mb->nb_segs +1 ; + mh->pkt_len = mb->pkt_len; + mh->ol_flags = mb->ol_flags &RTE_MBUF_F_TX_OFFLOAD_MASK ; + mb = mh; + } +io_uring_sqe_set_data (sqe, mb); -if (rte_mbuf_refcnt_read (mb) ==1 && -RTE_MBUF_DIRECT (mb) && mb->nb_segs ==1 ) { - void *base = rte_pktmbuf_mtod(mb, void *); -io_uring_prep_write (sqe, fd, base, mb->pkt_len,0 ); +PMD_TX_LOG (DEBUG ,"write m=%p segs=%u" , mb, mb->nb_segs); + void *buf = rte_pktmbuf_mtod_offset(mb, void *, -sizeof(*hdr)); +unsigned int nbytes =sizeof (struct virtio_net_hdr ) + mb->data_len; -PMD_TX_LOG (DEBUG ,"tx mbuf: %p submit" , mb); +if (mb->nb_segs ==1 ) { +io_uring_prep_write (sqe, fd, buf, nbytes,0 ); }else { -PMD_LOG (ERR ,"Can't do mbuf without space yet!" ); - ++txq->tx_errors; -continue ; +struct iovec iovs[RTE_MBUF_MAX_NB_SEGS +1 ]; +unsigned int niov = mb->nb_segs; + + iovs[0 ].iov_base = buf; + iovs[0 ].iov_len = nbytes; + +for (unsigned int i =1 ; i < niov; i++) { + mb = mb->next; + iovs[i].iov_base = rte_pktmbuf_mtod(mb, void *); + iovs[i].iov_len = mb->data_len; + } + +io_uring_prep_writev (sqe, fd, iovs, niov,0 ); } }
Signature · 1 line
2.47.2
PATCH v4 9/10 net/ioring: support Tx checksum and segment offload
In reply to Stephen HemmingerThe code for transmit flag mapping can be copied from virtio driver. The TAP device does not support querying features of the virtio net header, so the driver assumes checksum offload is allowed.
--- doc/guides/nics/features/ioring.ini | 2 ++ drivers/net/ioring/rte_eth_ioring.c | 51 ++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-)
doc/guides/nics/features/ioring.ini +2 -0
@@ -10,6 +10,8 @@ Promiscuous mode = Y Allmulticast mode = Y Multiprocess aware = Y Basic stats = Y +L3 checksum offload = Y +L4 checksum offload = Y Stats per queue = Y Linux = Y x86-64 = Y
drivers/net/ioring/rte_eth_ioring.c +50 -1
69 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH v4 10/10 net/ioring: add support for Rx offload
In reply to Stephen HemmingerThe TAP device supports receive offload.
--- drivers/net/ioring/rte_eth_ioring.c | 98 ++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 2 deletions(-)
drivers/net/ioring/rte_eth_ioring.c +96 -2
@@ -30,6 +30,7 @@#include <rte_ether.h> #include <rte_kvargs.h> #include <rte_log.h> +#include <rte_net.h> static_assert (RTE_PKTMBUF_HEADROOM >=sizeof (struct virtio_net_hdr ));
@@ -44,7 +45,10 @@ static_assert(IORING_MAX_QUEUES <= RTE_MP_MAX_FD_NUM, "Max queues exceeds MP fd RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_TCP_TSO) -#define IORING_RX_OFFLOAD RTE_ETH_RX_OFFLOAD_SCATTER +#define IORING_RX_OFFLOAD (RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_LRO | \ + RTE_ETH_RX_OFFLOAD_SCATTER)#define IORING_DEFAULT_IFNAME "itap%d"#define IORING_MP_KEY "ioring_mp_send_fds"
@@ -349,10 +353,31 @@ eth_dev_stop(struct rte_eth_dev *dev)static int eth_dev_configure (struct rte_eth_dev *dev) { +struct pmd_internals *pmd = dev->data->dev_private; +/* rx/tx must be paired */ if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)return -EINVAL ; +/* +* Set offload flags visible on the kernel network interface. +* This controls whether kernel will use checksum offload etc. +* Note: kernel transmit is DPDK receive. +*/ +const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode; +unsigned int offload =0 ; +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_CHECKSUM ) { + offload |=TUN_F_CSUM ; + +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_TCP_LRO ) + offload |=TUN_F_TSO4 |TUN_F_TSO6 |TUN_F_TSO_ECN ; + } + +if (ioctl (pmd->keep_fd,TUNSETOFFLOAD , offload) !=0 ) { +PMD_LOG (ERR ,"ioctl(TUNSETOFFLOAD) failed: %s" ,strerror (errno)); +return -1 ; + } +return 0 ; }
@@ -558,7 +583,6 @@ eth_ioring_rx_alloc(struct rx_queue *rxq)return m; } -/* set length of received mbuf segments */ static inline void eth_ioring_rx_adjust (struct rte_mbuf *mb,size_t len)
@@ -582,6 +606,69 @@ eth_ioring_rx_adjust(struct rte_mbuf *mb, size_t len) } } +static int +eth_ioring_rx_offload (struct rte_mbuf *m,const struct virtio_net_hdr *hdr) +{ +uint32_t ptype; +bool l4_supported = false; +struct rte_net_hdr_lens hdr_lens; + +/* nothing to do */ +if (hdr->flags ==0 && hdr->gso_type ==VIRTIO_NET_HDR_GSO_NONE ) +return 0 ; + + m->ol_flags |=RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN ; + + ptype =rte_net_get_ptype (m, &hdr_lens,RTE_PTYPE_ALL_MASK ); + m->packet_type = ptype; +if ((ptype &RTE_PTYPE_L4_MASK ) ==RTE_PTYPE_L4_TCP || + (ptype &RTE_PTYPE_L4_MASK ) ==RTE_PTYPE_L4_UDP || + (ptype &RTE_PTYPE_L4_MASK ) ==RTE_PTYPE_L4_SCTP ) + l4_supported = true; + +if (hdr->flags &VIRTIO_NET_HDR_F_NEEDS_CSUM ) { +uint32_t hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len; +if (hdr->csum_start <= hdrlen && l4_supported) { + m->ol_flags |=RTE_MBUF_F_RX_L4_CKSUM_NONE ; + }else { +/* Unknown proto or tunnel, do sw cksum. */ +uint16_t csum =0 , off; + +if (rte_raw_cksum_mbuf (m, hdr->csum_start, +rte_pktmbuf_pkt_len (m) - hdr->csum_start, + &csum) <0 ) +return -EINVAL ; +if (likely (csum !=0xffff )) + csum = ~csum; + off = hdr->csum_offset + hdr->csum_start; +if (rte_pktmbuf_data_len (m) >= off +1 ) + *rte_pktmbuf_mtod_offset(m, uint16_t *, off) = csum; + } + }else if ((hdr->flags &VIRTIO_NET_HDR_F_DATA_VALID ) && l4_supported) { + m->ol_flags |=RTE_MBUF_F_RX_L4_CKSUM_GOOD ; + } + +/* GSO request, save required information in mbuf */ +if (hdr->gso_type !=VIRTIO_NET_HDR_GSO_NONE ) { +/* Check unsupported modes */ +if ((hdr->gso_type &VIRTIO_NET_HDR_GSO_ECN ) || hdr->gso_size ==0 ) +return -EINVAL ; + +/* Update mss lengths in mbuf */ + m->tso_segsz = hdr->gso_size; +switch (hdr->gso_type & ~VIRTIO_NET_HDR_GSO_ECN ) { +case VIRTIO_NET_HDR_GSO_TCPV4 : +case VIRTIO_NET_HDR_GSO_TCPV6 : + m->ol_flags |=RTE_MBUF_F_RX_LRO |RTE_MBUF_F_RX_L4_CKSUM_NONE ; +break ; +default : +return -EINVAL ; + } + } + +return 0 ; +} +static uint16_t eth_ioring_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) {
@@ -626,6 +713,13 @@ eth_ioring_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)else eth_ioring_rx_adjust (mb, len); +if (unlikely (eth_ioring_rx_offload (mb, hdr) <0 )) { +PMD_RX_LOG (ERR ,"invalid rx offload" ); + ++rxq->rx_errors; + goto resubmit; + } + +__rte_mbuf_sanity_check (mb,1 ); num_bytes += mb->pkt_len; bufs[num_rx++] = mb;
Signature · 1 line
2.47.2
v5 cover net/rtap: add io_uring based TAP driver
In reply to Stephen HemmingerThis series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.
Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.
Motivation
----------
This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.
The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.
Why not extend net_tap?
-----------------------
The existing net_tap driver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity: rte_flow support emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.
net_rtap takes the opposite approach — use the kernel efficiently and let it do what it does well. There is no rte_flow support; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.
Given these fundamentally different design goals, a clean implementation was more practical than refactoring net_tap.
Acknowledgement
---------------
Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.
Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)
- liburing >= 2.0
Known working distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).
v5 - revised, renamed and expanded from the v4 ioring PMD
- more complete testing and dependency handling
Stephen Hemminger (10):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add link state change interrupt net/rtap: add multi-process support
net/rtap: add Rx interrupt support
test: add unit tests for rtap PMD
MAINTAINERS | 7 + app/test/meson.build | 1 + app/test/test_pmd_rtap.c | 2044 ++++++++++++++++++++++++ doc/guides/nics/features/rtap.ini | 25 + doc/guides/nics/index.rst | 1 + doc/guides/nics/rtap.rst | 101 ++ doc/guides/rel_notes/release_26_03.rst | 6 + drivers/net/meson.build | 1 + drivers/net/rtap/meson.build | 28 + drivers/net/rtap/rtap.h | 100 ++ drivers/net/rtap/rtap_ethdev.c | 908 +++++++++++ drivers/net/rtap/rtap_intr.c | 267 ++++ drivers/net/rtap/rtap_rxtx.c | 784 +++++++++ 13 files changed, 4273 insertions(+) create mode 100644 app/test/test_pmd_rtap.c create mode 100644 doc/guides/nics/features/rtap.ini create mode 100644 doc/guides/nics/rtap.rst create mode 100644 drivers/net/rtap/meson.build create mode 100644 drivers/net/rtap/rtap.h create mode 100644 drivers/net/rtap/rtap_ethdev.c create mode 100644 drivers/net/rtap/rtap_intr.c create mode 100644 drivers/net/rtap/rtap_rxtx.c
Signature · 1 line
2.51.0
PATCH v5 1/10 net/rtap: add driver skeleton and documentation
Add the initial skeleton for the rtap poll mode driver, a virtual ethernet device that uses Linux io_uring for packet I/O with kernel TAP devices.
This patch includes:
- MAINTAINERS entry
- Driver documentation (doc/guides/nics/rtap.rst)
- Feature matrix (doc/guides/nics/features/rtap.ini)
- Release notes update
- Meson build integration with liburing dependency
- Header file with shared data structures and declarations
- Stub probe/remove handlers that register the vdev driver
- Empty dev_ops with only dev_close implemented
The driver registers as net_rtap and is Linux-only.
It requires the liburing library version 2.0 or later. Earlier versions have known security and build issues. The library is available in all currently supported distributions (Debian 12+, Ubuntu 22.04+, RHEL 9+, Fedora 35+)
--- MAINTAINERS | 7 + doc/guides/nics/features/rtap.ini | 13 ++ doc/guides/nics/index.rst | 1 + doc/guides/nics/rtap.rst | 101 +++++++++++++++ doc/guides/rel_notes/release_26_03.rst | 6 + drivers/net/meson.build | 1 + drivers/net/rtap/meson.build | 26 ++++ drivers/net/rtap/rtap.h | 69 ++++++++++ drivers/net/rtap/rtap_ethdev.c | 172 +++++++++++++++++++++++++ 9 files changed, 396 insertions(+) create mode 100644 doc/guides/nics/features/rtap.ini create mode 100644 doc/guides/nics/rtap.rst create mode 100644 drivers/net/rtap/meson.build create mode 100644 drivers/net/rtap/rtap.h create mode 100644 drivers/net/rtap/rtap_ethdev.c
MAINTAINERS +7 -0
@@ -1135,6 +1135,13 @@ F: doc/guides/nics/pcap_ring.rst F: app/test/test_pmd_ring.c F: app/test/test_pmd_ring_perf.c +Rtap PMD - EXPERIMENTAL +M: Stephen Hemminger <[email protected]> +F: drivers/net/rtap/ +F: app/test/test_pmd_rtap.c +F: doc/guides/nics/rtap.rst +F: doc/guides/nics/features/rtap.ini + Null Networking PMD M: Tetsuya Mukawa <[email protected]> F: drivers/net/null/
doc/guides/nics/features/rtap.ini new file +13 -0
@@ -0,0 +1,13 @@ +; +; Supported features of the 'rtap' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +ARMv7 = Y +ARMv8 = Y +Power8 = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -65,6 +65,7 @@ Network Interface Controller Drivers qede r8169 rnp + rtap sfc_efx softnic tap
doc/guides/nics/rtap.rst new file +101 -0
101 lines not on this page · raw message
doc/guides/rel_notes/release_26_03.rst +6 -0
@@ -63,6 +63,12 @@ New Features * Added support for pre and post VF reset callbacks. +* **Added rtap virtual ethernet driver.** + + Added a new experimental virtual device driver that uses Linux io_uring + for packet injection into the kernel network stack. + It requires Linux kernel 5.1 or later and the liburing library. + Removed Items -------------
drivers/net/meson.build +1 -0
@@ -56,6 +56,7 @@ drivers = [ 'r8169', 'ring', 'rnp', + 'rtap', 'sfc', 'softnic', 'tap',
drivers/net/rtap/meson.build new file +26 -0
@@ -0,0 +1,26 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2026 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +liburing = dependency('liburing', version: '>= 2.0', required: false) +if not liburing.found() + build = false + reason = 'missing dependency, "liburing"' +endif + +if build and not cc.has_header_symbol('linux/io_uring.h', 'IORING_ASYNC_CANCEL_ALL') + build = false + reason = 'kernel headers missing IORING_ASYNC_CANCEL_ALL (need kernel >= 5.19 headers)' +endif + +sources = files( + 'rtap_ethdev.c', +) + +ext_deps += liburing + +require_iova_in_mbuf = false
drivers/net/rtap/rtap.h new file +69 -0
69 lines not on this page · raw message
drivers/net/rtap/rtap_ethdev.c new file +172 -0
172 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 2/10 net/rtap: add TAP device creation and queue management
In reply to Stephen HemmingerAdd TAP device creation using the Linux TUN/TAP interface with IFF_MULTI_QUEUE, IFF_NO_PI, and IFF_VNET_HDR flags. Enable NAPI mode when the kernel supports it.
The driver maintains a keep-alive file descriptor to the TAP device and opens additional per-queue file descriptors for data path I/O. This mirrors the multi-queue TAP architecture where each queue pair (rx + tx) shares a single TAP fd.
Add the rtap_create() function that:
- Opens the TAP device with configurable interface name
- Configures the virtio-net header size
- Reads the kernel-assigned MAC address
- Supports the 'persist' option to keep the interface after exit
- Detaches the keep-alive queue from data traffic
Add rtap_queue_open() and rtap_queue_close() for per-queue fd management used during queue setup and teardown.
--- drivers/net/rtap/rtap.h | 4 + drivers/net/rtap/rtap_ethdev.c | 216 ++++++++++++++++++++++++++++++++- 2 files changed, 219 insertions(+), 1 deletion(-)
drivers/net/rtap/rtap.h +4 -0
@@ -66,4 +66,8 @@ struct rtap_pmd { struct rte_ether_addr eth_addr; /* address assigned by kernel */ }; +/* rtap_ethdev.c */ +int rtap_queue_open (struct rte_eth_dev *dev,uint16_t queue_id); +void rtap_queue_close (struct rte_eth_dev *dev,uint16_t queue_id); +#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +215 -1
245 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 3/10 net/rtap: add Rx/Tx with scatter/gather support
In reply to Stephen HemmingerImplement packet receive and transmit using io_uring asynchronous I/O, with full support for both single-segment and multi-segment mbufs.
Rx path:
- rtap_rx_alloc() chains multiple mbufs when the MTU exceeds a single mbuf's tailroom capacity
- Pre-post read/readv requests to the io_uring submission queue, each backed by a pre-allocated (possibly chained) mbuf
- On rx_burst, harvest completed CQEs and replace each consumed mbuf with a freshly allocated one
- rtap_rx_adjust() distributes received data across segments and frees unused trailing segments
- Parse the prepended virtio-net header (offload fields are ignored until the offload patch)
Tx path:
- For single-segment mbufs, use io_uring write and batch submits
- For multi-segment mbufs, use writev via io_uring with immediate submit (iovec is stack-allocated)
- When the mbuf headroom is not writable (shared or indirect), chain a new header mbuf for the virtio-net header
- Prepend a zeroed virtio-net header (offload population deferred)
- Clean completed tx CQEs to free transmitted mbufs
Add io_uring cancel-all logic using IORING_ASYNC_CANCEL_ALL for clean queue teardown, draining all pending CQEs and freeing mbufs.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 13 + drivers/net/rtap/rtap_ethdev.c | 7 + drivers/net/rtap/rtap_rxtx.c | 755 ++++++++++++++++++++++++++++++ 5 files changed, 777 insertions(+) create mode 100644 drivers/net/rtap/rtap_rxtx.c
doc/guides/nics/features/rtap.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Scattered Rx = P Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/meson.build +1 -0
@@ -19,6 +19,7 @@ endif sources = files( 'rtap_ethdev.c', + 'rtap_rxtx.c', ) ext_deps += liburing
drivers/net/rtap/rtap.h +13 -0
@@ -70,4 +70,17 @@ struct rtap_pmd {int rtap_queue_open (struct rte_eth_dev *dev,uint16_t queue_id);void rtap_queue_close (struct rte_eth_dev *dev,uint16_t queue_id); +/* rtap_rxtx.c */ +uint16_t rtap_rx_burst (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts); +uint16_t rtap_tx_burst (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts); +int rtap_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_rx_desc,unsigned int socket_id, +const struct rte_eth_rxconf *rx_conf, +struct rte_mempool *mb_pool); +void rtap_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf); +void rtap_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +7 -0
@@ -232,6 +232,10 @@ static const struct eth_dev_ops rtap_ops = { .dev_stop = rtap_dev_stop, .dev_configure = rtap_dev_configure, .dev_close = rtap_dev_close, + .rx_queue_setup = rtap_rx_queue_setup, + .rx_queue_release = rtap_rx_queue_release, + .tx_queue_setup = rtap_tx_queue_setup, + .tx_queue_release = rtap_tx_queue_release, }; static int
@@ -272,6 +276,9 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)PMD_LOG (DEBUG ,"%s setup" , ifr.ifr_name); + dev->rx_pkt_burst = rtap_rx_burst; + dev->tx_pkt_burst = rtap_tx_burst; +return 0 ; error:
drivers/net/rtap/rtap_rxtx.c new file +755 -0
755 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 4/10 net/rtap: add statistics and device info
In reply to Stephen HemmingerImplement basic and per-queue statistics collection and the device info query.
Stats tracked per rx queue: packets, bytes, errors, and missed. Stats tracked per tx queue: packets, bytes, errors.
Device info reports:
- Interface index from the kernel TAP device
- Maximum queue counts
- Default burst size and ring size
Enable AUTOFILL_QUEUE_XSTATS for automatic per-queue extended statistics.
--- doc/guides/nics/features/rtap.ini | 2 + drivers/net/rtap/rtap.h | 2 + drivers/net/rtap/rtap_ethdev.c | 139 ++++++++++++++++++++++++++++++ 3 files changed, 143 insertions(+)
doc/guides/nics/features/rtap.ini +2 -0
@@ -5,6 +5,8 @@ ; [Features] Scattered Rx = P +Basic stats = Y +Stats per queue = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap.h +2 -0
@@ -64,6 +64,8 @@ struct rtap_pmd {int keep_fd;/* keep alive file descriptor */ char ifname[IFNAMSIZ ];/* name assigned by kernel */ struct rte_ether_addr eth_addr;/* address assigned by kernel */ + +uint64_t rx_drop_base;/* value of rx_dropped when reset */ };/* rtap_ethdev.c */
drivers/net/rtap/rtap_ethdev.c +139 -0
170 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 5/10 net/rtap: add link and device management operations
In reply to Stephen HemmingerAdd device management operations for the TAP interface:
- Link up/down control via SIOCSIFFLAGS
- Link status query via SIOCGIFFLAGS
- Promiscuous mode enable/disable
- All-multicast mode enable/disable
- MTU configuration via SIOCSIFMTU
- MAC address configuration via SIOCSIFHWADDR
These operations manipulate the kernel TAP interface flags and parameters through ioctl calls.
--- doc/guides/nics/features/rtap.ini | 4 + drivers/net/rtap/rtap.h | 1 + drivers/net/rtap/rtap_ethdev.c | 172 ++++++++++++++++++++++++++++++ 3 files changed, 177 insertions(+)
doc/guides/nics/features/rtap.ini +4 -0
@@ -4,6 +4,10 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Scattered Rx = P Basic stats = Y Stats per queue = Y
drivers/net/rtap/rtap.h +1 -0
@@ -71,6 +71,7 @@ struct rtap_pmd {/* rtap_ethdev.c */ int rtap_queue_open (struct rte_eth_dev *dev,uint16_t queue_id);void rtap_queue_close (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_link_update (struct rte_eth_dev *dev,int wait_to_complete);/* rtap_rxtx.c */ uint16_t rtap_rx_burst (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts);
drivers/net/rtap/rtap_ethdev.c +172 -0
193 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 6/10 net/rtap: add checksum and TSO offload support
In reply to Stephen HemmingerAdd transmit and receive offload support using the virtio-net header that is exchanged with the kernel TAP device.
Tx offloads:
- UDP and TCP checksum offload: set VIRTIO_NET_HDR_F_NEEDS_CSUM with appropriate csum_start and csum_offset fields
- TCP segmentation offload (TSO): set gso_type and gso_size
Rx offloads:
- Parse virtio-net header flags to determine checksum status
- Handle NEEDS_CSUM by either marking L4_CKSUM_NONE for known protocols or computing the checksum in software for tunnels
- Handle DATA_VALID flag to mark L4_CKSUM_GOOD
- Parse GSO headers to populate LRO metadata
Configure kernel-side offloads via TUNSETOFFLOAD in dev_configure based on the requested rx offload flags.
Report offload capabilities in dev_info.
--- doc/guides/nics/features/rtap.ini | 2 ++ drivers/net/rtap/rtap_ethdev.c | 30 +++++++++++++++++++++++++++++- 2 files changed, 31 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +2 -0
@@ -11,6 +11,8 @@ Allmulticast mode = Y Scattered Rx = P Basic stats = Y Stats per queue = Y +TSO = Y +L4 checksum offload = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap_ethdev.c +29 -1
@@ -31,6 +31,16 @@#define RTAP_DEFAULT_IFNAME "rtap%d" +#define RTAP_TX_OFFLOAD (RTE_ETH_TX_OFFLOAD_MULTI_SEGS | \ + RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_TSO) + +#define RTAP_RX_OFFLOAD (RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_LRO | \ + RTE_ETH_RX_OFFLOAD_SCATTER) +#define RTAP_DEFAULT_BURST 64#define RTAP_NUM_BUFFERS 1024#define RTAP_MAX_QUEUES 128
@@ -317,7 +327,21 @@ rtap_dev_configure(struct rte_eth_dev *dev)if (dev->data->nb_rx_queues != dev->data->nb_tx_queues)return -EINVAL ; -if (ioctl (pmd->keep_fd,TUNSETOFFLOAD ,0 ) !=0 ) { +/* +* Set offload flags visible on the kernel network interface. +* This controls whether kernel will use checksum offload etc. +* Note: kernel transmit is DPDK receive. +*/ +const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode; +unsigned int offload =0 ; +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_CHECKSUM ) { + offload |=TUN_F_CSUM ; + +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_TCP_LRO ) + offload |=TUN_F_TSO4 |TUN_F_TSO6 |TUN_F_TSO_ECN ; + } + +if (ioctl (pmd->keep_fd,TUNSETOFFLOAD , offload) !=0 ) {PMD_LOG (ERR ,"ioctl(TUNSETOFFLOAD) failed: %s" ,strerror (errno));return -1 ; }
@@ -336,6 +360,10 @@ rtap_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->min_rx_bufsize =RTAP_MIN_RX_BUFSIZE ; dev_info->max_rx_queues =RTAP_MAX_QUEUES ; dev_info->max_tx_queues =RTAP_MAX_QUEUES ; + dev_info->rx_queue_offload_capa =RTAP_RX_OFFLOAD ; + dev_info->rx_offload_capa = dev_info->rx_queue_offload_capa; + dev_info->tx_queue_offload_capa =RTAP_TX_OFFLOAD ; + dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa; dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { .burst_size =RTAP_DEFAULT_BURST ,
Signature · 1 line
2.51.0
PATCH v5 7/10 net/rtap: add link state change interrupt
In reply to Stephen HemmingerAdd link state change (LSC) notification support using a netlink socket subscribed to RTMGRP_LINK events.
When LSC is enabled in the device configuration, the driver:
- Creates a NETLINK_ROUTE socket filtering for RTM_NEWLINK and RTM_DELLINK messages
- Registers the socket fd with the EAL interrupt handler
- On interrupt, drains all pending netlink messages and updates the link status for matching interface index changes
The interrupt is enabled during dev_start and disabled during dev_stop, with proper cleanup of the netlink socket and EAL callback registration.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 5 + drivers/net/rtap/rtap_ethdev.c | 36 +++++++- drivers/net/rtap/rtap_intr.c | 147 ++++++++++++++++++++++++++++++ 5 files changed, 187 insertions(+), 3 deletions(-) create mode 100644 drivers/net/rtap/rtap_intr.c
doc/guides/nics/features/rtap.ini +1 -0
@@ -5,6 +5,7 @@ ; [Features] Link status = Y +Link status event = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y
drivers/net/rtap/meson.build +1 -0
@@ -19,6 +19,7 @@ endif sources = files( 'rtap_ethdev.c', + 'rtap_intr.c', 'rtap_rxtx.c', )
drivers/net/rtap/rtap.h +5 -0
@@ -13,6 +13,7 @@#include <ethdev_driver.h> #include <rte_ether.h> +#include <rte_interrupts.h> #include <rte_log.h>
@@ -62,6 +63,7 @@ struct rtap_tx_queue {struct rtap_pmd {int keep_fd;/* keep alive file descriptor */ +struct rte_intr_handle *intr_handle;/* LSC interrupt handle */ char ifname[IFNAMSIZ ];/* name assigned by kernel */ struct rte_ether_addr eth_addr;/* address assigned by kernel */
@@ -86,4 +88,7 @@ int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, const struct rte_eth_txconf *tx_conf);void rtap_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +/* rtap_intr.c */ +int rtap_lsc_set (struct rte_eth_dev *dev,int set); +#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +33 -3
@@ -24,6 +24,7 @@#include <rte_eal.h> #include <rte_ethdev.h> #include <rte_ether.h> +#include <rte_interrupts.h> #include <rte_kvargs.h> #include <rte_log.h>
@@ -141,11 +142,16 @@ rtap_change_flags(struct rte_eth_dev *dev, uint32_t flags, uint32_t mask) ifr.ifr_flags |= flags; ret =ioctl (sock,SIOCSIFFLAGS , &ifr); -if (ret <0 ) +if (ret <0 ) {PMD_LOG_ERRNO (ERR ,"Unable to set flags for %s" , ifr.ifr_name); + goto error; + } +close (sock); +return 0 ; error: + ret = -errno;close (sock); -return (ret <0 ) ? -errno :0 ; +return ret; } static int
@@ -277,11 +283,18 @@ rtap_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)static int rtap_dev_start (struct rte_eth_dev *dev) { -int ret =rtap_set_link_up (dev); +int ret; + ret =rtap_lsc_set (dev,1 );if (ret !=0 )return ret; + ret =rtap_set_link_up (dev); +if (ret !=0 ) { +rtap_lsc_set (dev,0 ); +return ret; + } + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ;for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ;
@@ -298,6 +311,7 @@ rtap_dev_stop(struct rte_eth_dev *dev) dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +rtap_lsc_set (dev,0 );rtap_set_link_down (dev);for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) {
@@ -508,6 +522,9 @@ rtap_dev_close(struct rte_eth_dev *dev)close (pmd->keep_fd); pmd->keep_fd =-1 ; } + +rte_intr_instance_free (pmd->intr_handle); + pmd->intr_handle =NULL ; }free (dev->process_private);
@@ -585,6 +602,17 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) pmd->keep_fd =-1 ; pmd->rx_drop_base =0 ; +/* Allocate interrupt instance for link state change events */ + pmd->intr_handle =rte_intr_instance_alloc (RTE_INTR_INSTANCE_F_SHARED ); +if (pmd->intr_handle ==NULL ) { +PMD_LOG (ERR ,"Failed to allocate intr handle" ); + goto error; + } +rte_intr_type_set (pmd->intr_handle,RTE_INTR_HANDLE_EXT ); +rte_intr_fd_set (pmd->intr_handle,-1 ); + dev->intr_handle = pmd->intr_handle; + data->dev_flags |=RTE_ETH_DEV_INTR_LSC ; + dev->dev_ops = &rtap_ops;/* Get the initial fd used to keep the tap device around */
@@ -623,6 +651,8 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) error:if (pmd->keep_fd !=-1 )close (pmd->keep_fd); +rte_intr_instance_free (pmd->intr_handle); + pmd->intr_handle =NULL ;return -1 ; }
drivers/net/rtap/rtap_intr.c new file +147 -0
147 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 8/10 net/rtap: add multi-process support
In reply to Stephen HemmingerAdd support for DPDK multi-process operation. Secondary processes need access to the per-queue TAP file descriptors owned by the primary process.
Implement fd sharing using the DPDK multi-process IPC mechanism:
- Primary registers an MP action handler that responds to fd requests by sending queue fds via rte_mp_reply()
- Secondary process attaches to the existing device and requests fds from primary via rte_mp_request_sync()
The MP action is registered on the first device probe and unregistered when the last device is closed.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/rtap_ethdev.c | 130 ++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+)
doc/guides/nics/features/rtap.ini +1 -0
@@ -14,6 +14,7 @@ Basic stats = Y Stats per queue = Y TSO = Y L4 checksum offload = Y +Multiprocess aware = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap_ethdev.c +130 -0
166 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v5 9/10 net/rtap: add Rx interrupt support
In reply to Stephen HemmingerAdd support for the DPDK Rx interrupt mechanism, enabling power-aware applications (e.g. l3fwd-power) to sleep until packets arrive rather than busy-polling.
Each Rx queue creates an eventfd during queue setup and registers it with its io_uring instance via io_uring_register_eventfd(). When the kernel posts a CQE (completing a read, i.e. a packet arrived), it signals the eventfd. These per-queue eventfds are wired into a VDEV interrupt handle during dev_start when the application has set intr_conf.rxq.
The enable op drains the eventfd counter to re-arm notification; disable is a no-op since the application simply stops polling. The eventfd is always created unconditionally so it is available if the application enables Rx interrupts later.
The Rx interrupt handle is kept separate from the existing LSC netlink interrupt handle to avoid coupling the two mechanisms.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/rtap.h | 6 ++ drivers/net/rtap/rtap_ethdev.c | 16 ++++ drivers/net/rtap/rtap_intr.c | 120 ++++++++++++++++++++++++++++++ drivers/net/rtap/rtap_rxtx.c | 31 +++++++- 5 files changed, 173 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +1 -0
@@ -6,6 +6,7 @@ [Features] Link status = Y Link status event = Y +Rx interrupt = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y
drivers/net/rtap/rtap.h +6 -0
@@ -42,6 +42,7 @@ extern int rtap_logtype;struct rtap_rx_queue {struct rte_mempool *mb_pool;/* rx buffer pool */ struct io_uring io_ring;/* queue of posted read's */ +int intr_fd;/* eventfd for Rx interrupt */ uint16_t port_id;uint16_t queue_id;
@@ -64,6 +65,7 @@ struct rtap_tx_queue {struct rtap_pmd {int keep_fd;/* keep alive file descriptor */ struct rte_intr_handle *intr_handle;/* LSC interrupt handle */ +struct rte_intr_handle *rx_intr_handle;/* Rx queue interrupt handle */ char ifname[IFNAMSIZ ];/* name assigned by kernel */ struct rte_ether_addr eth_addr;/* address assigned by kernel */
@@ -90,5 +92,9 @@ void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);/* rtap_intr.c */ int rtap_lsc_set (struct rte_eth_dev *dev,int set); +int rtap_rx_intr_vec_install (struct rte_eth_dev *dev); +void rtap_rx_intr_vec_uninstall (struct rte_eth_dev *dev); +int rtap_rx_queue_intr_enable (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_rx_queue_intr_disable (struct rte_eth_dev *dev,uint16_t queue_id);#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +16 -0
@@ -293,8 +293,18 @@ rtap_dev_start(struct rte_eth_dev *dev)if (ret !=0 )return ret; +/* Install Rx interrupt vector if requested by application */ +if (dev->data->dev_conf.intr_conf.rxq) { + ret =rtap_rx_intr_vec_install (dev); +if (ret !=0 ) { +rtap_lsc_set (dev,0 ); +return ret; + } + } + ret =rtap_set_link_up (dev);if (ret !=0 ) { +rtap_rx_intr_vec_uninstall (dev);rtap_lsc_set (dev,0 );return ret; }
@@ -315,6 +325,7 @@ rtap_dev_stop(struct rte_eth_dev *dev) dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +rtap_rx_intr_vec_uninstall (dev);rtap_lsc_set (dev,0 );rtap_set_link_down (dev);
@@ -527,6 +538,9 @@ rtap_dev_close(struct rte_eth_dev *dev) pmd->keep_fd =-1 ; } +rte_intr_instance_free (pmd->rx_intr_handle); + pmd->rx_intr_handle =NULL ; +rte_intr_instance_free (pmd->intr_handle); pmd->intr_handle =NULL ; }
@@ -597,6 +611,8 @@ static const struct eth_dev_ops rtap_ops = { .rx_queue_release = rtap_rx_queue_release, .tx_queue_setup = rtap_tx_queue_setup, .tx_queue_release = rtap_tx_queue_release, + .rx_queue_intr_enable = rtap_rx_queue_intr_enable, + .rx_queue_intr_disable = rtap_rx_queue_intr_disable, }; static int
drivers/net/rtap/rtap_intr.c +120 -0
123 lines not on this page · raw message
drivers/net/rtap/rtap_rxtx.c +30 -1
@@ -8,6 +8,7 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <sys/eventfd.h> #include <liburing.h> #include <sys/uio.h> #include <linux/virtio_net.h>
@@ -369,6 +370,7 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d rxq->mb_pool = mb_pool; rxq->port_id = dev->data->port_id; rxq->queue_id = queue_id; + rxq->intr_fd =-1 ; dev->data->rx_queues[queue_id] = rxq;if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) {
@@ -376,10 +378,26 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d goto error_rxq_free; } +/* +* Create an eventfd for Rx interrupt notification. +* io_uring will signal this fd whenever a CQE is posted, +* enabling power-aware applications to sleep until packets arrive. +*/ + rxq->intr_fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC ); +if (rxq->intr_fd <0 ) { +PMD_LOG (ERR ,"eventfd failed: %s" ,strerror (errno)); + goto error_iouring_exit; + } + +if (io_uring_register_eventfd (&rxq->io_ring, rxq->intr_fd) <0 ) { +PMD_LOG (ERR ,"io_uring_register_eventfd failed: %s" ,strerror (errno)); + goto error_eventfd_close; + } + mbufs =calloc (nb_rx_desc,sizeof (struct rte_mbuf *));if (mbufs ==NULL ) {PMD_LOG (ERR ,"Rx mbuf pointer alloc failed" ); - goto error_iouring_exit; + goto error_eventfd_close; }/* open shared tap fd maybe already setup */
@@ -429,6 +447,11 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d }rtap_queue_close (dev, queue_id);free (mbufs); +error_eventfd_close: +if (rxq->intr_fd >=0 ) { +close (rxq->intr_fd); + rxq->intr_fd =-1 ; + } error_iouring_exit:io_uring_queue_exit (&rxq->io_ring); error_rxq_free:
@@ -503,6 +526,12 @@ rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)if (rxq ==NULL )return ; +if (rxq->intr_fd >=0 ) { +io_uring_unregister_eventfd (&rxq->io_ring); +close (rxq->intr_fd); + rxq->intr_fd =-1 ; + } +rtap_cancel_all (&rxq->io_ring);io_uring_queue_exit (&rxq->io_ring);
Signature · 1 line
2.51.0
PATCH v5 10/10 test: add unit tests for rtap PMD
In reply to Stephen HemmingerAdd a comprehensive test suite for the rtap poll mode driver covering:
- Device creation and configuration
- Device info query
- Link status and link up/down control
- Statistics collection and reset
- Promiscuous and all-multicast mode toggling
- MAC address and MTU configuration
- Multi-queue setup, teardown, and queue count reduction
- Mismatched rx/tx queue count rejection
- Queue reconfiguration (stop, reconfigure, restart)
- Rx path: inject packet via AF_PACKET, receive via DPDK
- Tx path: transmit via DPDK, capture via AF_PACKET
- Multi-segment Tx with chained mbufs
- Multi-segment Rx with small-buffer mempool
- Offload capability reporting and configuration
- Tx UDP checksum offload end-to-end verification
- File descriptor leak detection after device close
- Command-line created device testing
Tests require CAP_NET_ADMIN or root privileges
--- app/test/meson.build | 1 + app/test/test_pmd_rtap.c | 2044 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 2045 insertions(+) create mode 100644 app/test/test_pmd_rtap.c
app/test/meson.build +1 -0
@@ -143,6 +143,7 @@ source_file_deps = { 'test_pie.c': ['sched'], 'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps, 'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'], + 'test_pmd_rtap.c': ['net_rtap', 'ethdev', 'bus_vdev'], 'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'], 'test_pmu.c': ['pmu'], 'test_power.c': ['power', 'power_acpi', 'power_kvm_vm', 'power_intel_pstate',
app/test/test_pmd_rtap.c new file +2044 -0
2,044 lines not on this page · raw message
Signature · 1 line
2.51.0
RE: [PATCH v5 00/10] net/rtap: add io_uring based TAP driver
In reply to Stephen HemmingerQuoted 6 lines
This series adds
net_rtap, an experimental poll mode driver that uses Linuxio_uringfor asynchronous packet I/O with kernel TAP interfaces.Like
net_tap,net_rtapcreates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.
Robin Jarry recently added Network Namespace support to the TAP driver [1]. Does the rtap driver support Network Namespaces?
[1]: https://inbox.dpdk.org/dev/[email protected]/
Quoted 2 lines
Requirements:
- Kernel headers withIORING_ASYNC_CANCEL_ALL(upstream since 5.19)
The DPDK 25.11 minimum requirement [2] is Kernel 5.4.
Since this driver requires a higher version, it should be prominently mentioned in relevant documentation.
In December 2026, shortly after the release of DPDK 26.11, the oldest supported Kernel LTS version will be 6.1. For now, it is 5.10.
[2]: https://doc.dpdk.org/guides-25.11/linux_gsg/sys_reqs.html#system-software
Quoted 5 lines
- liburing >= 2.0
Known working distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (
io_uringis disabled by default).
v6 cover net/rtap: add io_uring based TAP driver
In reply to Stephen HemmingerThis series adds net_rtap, an experimental poll mode driver that uses Linux io_uring for asynchronous packet I/O with kernel TAP interfaces.
Like net_tap, net_rtap creates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.
Motivation
----------
This driver started as an experiment to determine whether Linux io_uring could deliver better packet I/O performance than the traditional read()/write() system calls used by net_tap. By posting batches of I/O requests asynchronously, io_uring amortizes system call overhead across multiple packets.
The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.
Why not extend net_tap?
-----------------------
The existing net_tap driver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity: rte_flow support emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.
net_rtap takes the opposite approach -- use the kernel efficiently and let it do what it does well. There is no rte_flow support; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.
Given these fundamentally different design goals, a clean implementation was more practical than refactoring net_tap.
Acknowledgement
---------------
Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.
Requirements:
- Kernel headers with IORING_ASYNC_CANCEL_ALL (upstream since 5.19)
- liburing >= 2.0
Should work on distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (io_uring is disabled by default).
v6:
- fix lots of bugs found doing automated review
- convert to use ifindex and netlink to avoid rename issues
- implement xstats
Stephen Hemminger (11):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add multi-process support
net/rtap: add link state change interrupt net/rtap: add Rx interrupt support
net/rtap: add extended statistics support
test: add unit tests for rtap PMD
MAINTAINERS | 7 + app/test/meson.build | 1 + app/test/test_pmd_rtap.c | 2620 ++++++++++++++++++++++++ doc/guides/nics/features/rtap.ini | 26 + doc/guides/nics/index.rst | 1 + doc/guides/nics/rtap.rst | 101 + doc/guides/rel_notes/release_26_03.rst | 7 + drivers/net/meson.build | 1 + drivers/net/rtap/meson.build | 30 + drivers/net/rtap/rtap.h | 152 ++ drivers/net/rtap/rtap_ethdev.c | 864 ++++++++ drivers/net/rtap/rtap_intr.c | 207 ++ drivers/net/rtap/rtap_netlink.c | 445 ++++ drivers/net/rtap/rtap_rxtx.c | 803 ++++++++ drivers/net/rtap/rtap_xstats.c | 293 +++ 15 files changed, 5558 insertions(+) create mode 100644 app/test/test_pmd_rtap.c create mode 100644 doc/guides/nics/features/rtap.ini create mode 100644 doc/guides/nics/rtap.rst create mode 100644 drivers/net/rtap/meson.build create mode 100644 drivers/net/rtap/rtap.h create mode 100644 drivers/net/rtap/rtap_ethdev.c create mode 100644 drivers/net/rtap/rtap_intr.c create mode 100644 drivers/net/rtap/rtap_netlink.c create mode 100644 drivers/net/rtap/rtap_rxtx.c create mode 100644 drivers/net/rtap/rtap_xstats.c
Signature · 1 line
2.51.0
PATCH v6 1/11 net/rtap: add driver skeleton and documentation
Add the initial skeleton for the rtap poll mode driver, a virtual ethernet device that uses Linux io_uring for packet I/O with kernel TAP devices.
This patch includes:
- MAINTAINERS entry
- Driver documentation (doc/guides/nics/rtap.rst)
- Feature matrix (doc/guides/nics/features/rtap.ini)
- Release notes update
- Meson build integration with liburing dependency
- Header file with shared data structures and declarations
- Stub probe/remove handlers that register the vdev driver
- Empty dev_ops with only dev_close implemented
The driver registers as net_rtap and is Linux-only.
Requires the liburing library version 2.0 or later. Earlier versions have known security and build issues.
--- MAINTAINERS | 7 + doc/guides/nics/features/rtap.ini | 13 ++ doc/guides/nics/index.rst | 1 + doc/guides/nics/rtap.rst | 101 ++++++++++++++ doc/guides/rel_notes/release_26_03.rst | 7 + drivers/net/meson.build | 1 + drivers/net/rtap/meson.build | 26 ++++ drivers/net/rtap/rtap.h | 81 +++++++++++ drivers/net/rtap/rtap_ethdev.c | 177 +++++++++++++++++++++++++ 9 files changed, 414 insertions(+) create mode 100644 doc/guides/nics/features/rtap.ini create mode 100644 doc/guides/nics/rtap.rst create mode 100644 drivers/net/rtap/meson.build create mode 100644 drivers/net/rtap/rtap.h create mode 100644 drivers/net/rtap/rtap_ethdev.c
MAINTAINERS +7 -0
@@ -1135,6 +1135,13 @@ F: doc/guides/nics/ring.rst F: app/test/test_pmd_ring.c F: app/test/test_pmd_ring_perf.c +Rtap PMD - EXPERIMENTAL +M: Stephen Hemminger <[email protected]> +F: drivers/net/rtap/ +F: app/test/test_pmd_rtap.c +F: doc/guides/nics/rtap.rst +F: doc/guides/nics/features/rtap.ini + Null Networking PMD M: Tetsuya Mukawa <[email protected]> F: drivers/net/null/
doc/guides/nics/features/rtap.ini new file +13 -0
@@ -0,0 +1,13 @@ +; +; Supported features of the 'rtap' driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +ARMv7 = Y +ARMv8 = Y +Power8 = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/index.rst +1 -0
@@ -66,6 +66,7 @@ Network Interface Controller Drivers r8169 ring rnp + rtap sfc_efx softnic tap
doc/guides/nics/rtap.rst new file +101 -0
101 lines not on this page · raw message
doc/guides/rel_notes/release_26_03.rst +7 -0
@@ -87,6 +87,13 @@ New Features * Added support for AES-XTS cipher algorithm. * Added support for SHAKE-128 and SHAKE-256 authentication algorithms. +* **Added rtap virtual ethernet driver.** + + Added a new experimental virtual device driver that uses Linux io_uring + for packet injection into the kernel network stack. + It requires Linux kernel 5.19 or later for IORING_ASYNC_CANCEL + and liburing 2.0 or later. + Removed Items -------------
drivers/net/meson.build +1 -0
@@ -56,6 +56,7 @@ drivers = [ 'r8169', 'ring', 'rnp', + 'rtap', 'sfc', 'softnic', 'tap',
drivers/net/rtap/meson.build new file +26 -0
@@ -0,0 +1,26 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2026 Stephen Hemminger + +if not is_linux + build = false + reason = 'only supported on Linux' +endif + +liburing = dependency('liburing', version: '>= 2.0', required: false) +if not liburing.found() + build = false + reason = 'missing dependency, "liburing"' +endif + +if build and not cc.has_header_symbol('linux/io_uring.h', 'IORING_ASYNC_CANCEL_ALL') + build = false + reason = 'kernel headers missing IORING_ASYNC_CANCEL_ALL (need kernel >= 5.19 headers)' +endif + +sources = files( + 'rtap_ethdev.c', +) + +ext_deps += liburing + +require_iova_in_mbuf = false
drivers/net/rtap/rtap.h new file +81 -0
81 lines not on this page · raw message
drivers/net/rtap/rtap_ethdev.c new file +177 -0
177 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 2/11 net/rtap: add TAP device creation and queue management
In reply to Stephen HemmingerAdd TAP device creation using /dev/net/tun with IFF_MULTI_QUEUE, IFF_NO_PI, IFF_VNET_HDR, and optional IFF_NAPI flags.
The driver maintains a keep-alive fd and opens additional per-queue fds for I/O. Each queue pair (rx+tx) shares a single TAP fd.
Key operations:
- rtap_create(): Opens TAP, gets stable ifindex, opens netlink socket, retrieves MAC via netlink, detaches keep-alive queue
- rtap_queue_open/close(): Per-queue fd management (converts ifindex to name for TUNSETIFF ioctl)
- rtap_dev_configure(): Validates paired queues, clears offloads
- rtap_dev_start/stop(): Manages link status and queue states
The driver uses netlink (RTM_GETLINK/RTM_NEWLINK) and ifindex for interface control rather than ioctl() and interface names. This avoids issues with interface renames and namespace moves.
--- drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 4 + drivers/net/rtap/rtap_ethdev.c | 241 ++++++++++++++++- drivers/net/rtap/rtap_netlink.c | 445 ++++++++++++++++++++++++++++++++ 4 files changed, 689 insertions(+), 2 deletions(-) create mode 100644 drivers/net/rtap/rtap_netlink.c
drivers/net/rtap/meson.build +1 -0
@@ -19,6 +19,7 @@ endif sources = files( 'rtap_ethdev.c', + 'rtap_netlink.c', ) ext_deps += liburing
drivers/net/rtap/rtap.h +4 -0
@@ -64,6 +64,10 @@ struct rtap_pmd {struct rte_ether_addr eth_addr;/* address assigned by kernel */ }; +/* rtap_ethdev.c */ +int rtap_queue_open (struct rte_eth_dev *dev,uint16_t queue_id); +void rtap_queue_close (struct rte_eth_dev *dev,uint16_t queue_id); +/* rtap_netlink.c */ int rtap_nl_open (unsigned int groups);struct rte_eth_dev ;
drivers/net/rtap/rtap_ethdev.c +239 -2
276 lines not on this page · raw message
drivers/net/rtap/rtap_netlink.c new file +445 -0
445 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 3/11 net/rtap: add Rx/Tx with scatter/gather support
In reply to Stephen HemmingerImplement packet receive and transmit using io_uring asynchronous I/O, with full support for both single-segment and multi-segment mbufs.
Rx path:
- rtap_rx_alloc() chains multiple mbufs when the MTU exceeds a single mbuf's tailroom capacity
- Pre-post read/readv requests to the io_uring submission queue, each backed by a pre-allocated (possibly chained) mbuf
- On rx_burst, harvest completed CQEs and replace each consumed mbuf with a freshly allocated one
- rtap_rx_adjust() distributes received data across segments and frees unused trailing segments
- Parse the prepended virtio-net header
Tx path:
- For single-segment mbufs, use io_uring write and batch submits
- For multi-segment mbufs, use writev via io_uring with immediate submit (iovec is stack-allocated)
- When the mbuf headroom is not writable (shared or indirect), chain a new header mbuf for the virtio-net header
- Prepend a zeroed virtio-net header
- Clean completed Tx CQEs to free transmitted mbufs
Add io_uring cancel-all logic using IORING_ASYNC_CANCEL_ALL for clean queue teardown, draining all pending CQEs and freeing mbufs.
Full offload configuration is not enabled until later.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 13 + drivers/net/rtap/rtap_ethdev.c | 7 + drivers/net/rtap/rtap_rxtx.c | 771 ++++++++++++++++++++++++++++++ 5 files changed, 793 insertions(+) create mode 100644 drivers/net/rtap/rtap_rxtx.c
doc/guides/nics/features/rtap.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Scattered Rx = P Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/meson.build +1 -0
@@ -20,6 +20,7 @@ endif sources = files( 'rtap_ethdev.c', 'rtap_netlink.c', + 'rtap_rxtx.c', ) ext_deps += liburing
drivers/net/rtap/rtap.h +13 -0
@@ -82,4 +82,17 @@ int rtap_nl_get_mac(int nlsk_fd, int if_index, struct rte_ether_addr *addr);struct rtnl_link_stats64 ;int rtap_nl_get_stats (int if_index,struct rtnl_link_stats64 *stats); +/* rtap_rxtx.c */ +uint16_t rtap_rx_burst (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts); +uint16_t rtap_tx_burst (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts); +int rtap_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_rx_desc,unsigned int socket_id, +const struct rte_eth_rxconf *rx_conf, +struct rte_mempool *mb_pool); +void rtap_rx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id, +uint16_t nb_tx_desc,unsigned int socket_id, +const struct rte_eth_txconf *tx_conf); +void rtap_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +7 -0
@@ -248,6 +248,10 @@ static const struct eth_dev_ops rtap_ops = { .dev_stop = rtap_dev_stop, .dev_configure = rtap_dev_configure, .dev_close = rtap_dev_close, + .rx_queue_setup = rtap_rx_queue_setup, + .rx_queue_release = rtap_rx_queue_release, + .tx_queue_setup = rtap_tx_queue_setup, + .tx_queue_release = rtap_tx_queue_release, }; static int
@@ -298,6 +302,9 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)PMD_LOG (DEBUG ,"ifindex %d setup" , pmd->if_index); + dev->rx_pkt_burst = rtap_rx_burst; + dev->tx_pkt_burst = rtap_tx_burst; +return 0 ; error:
drivers/net/rtap/rtap_rxtx.c new file +771 -0
771 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 4/11 net/rtap: add statistics and device info
In reply to Stephen HemmingerImplement basic and per-queue statistics collection. Usual statistics per queue: packets, bytes, errors.
The imissed counter reports kernel rx_dropped (packets dropped before reaching the interface) retrieved via netlink IFLA_STATS64. The rx_drop_base field captures the baseline value at stats_reset.
Device info reports ifindex, max queue counts, and default burst/ring sizes.
--- doc/guides/nics/features/rtap.ini | 2 + drivers/net/rtap/rtap.h | 2 + drivers/net/rtap/rtap_ethdev.c | 120 ++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+)
doc/guides/nics/features/rtap.ini +2 -0
@@ -5,6 +5,8 @@ ; [Features] Scattered Rx = P +Basic stats = Y +Stats per queue = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap.h +2 -0
@@ -62,6 +62,8 @@ struct rtap_pmd {int if_index;/* interface index */ int nlsk_fd;/* netlink control socket */ struct rte_ether_addr eth_addr;/* address assigned by kernel */ + +uint64_t rx_drop_base;/* value of rx_dropped when reset */ };/* rtap_ethdev.c */
drivers/net/rtap/rtap_ethdev.c +120 -0
157 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 5/11 net/rtap: add link and device management operations
In reply to Stephen HemmingerAdd device management operations via netlink:
- Link up/down control
- Link status query (checks IFF_UP and IFF_RUNNING)
- Promiscuous mode enable/disable
- All-multicast mode enable/disable
- MTU configuration
- MAC address configuration
All operations use the persistent netlink socket (pmd->nlsk_fd) and ifindex for stable interface identification across renames and namespace moves.
--- doc/guides/nics/features/rtap.ini | 4 ++ drivers/net/rtap/rtap.h | 1 + drivers/net/rtap/rtap_ethdev.c | 102 ++++++++++++++++++++++++++++++ 3 files changed, 107 insertions(+)
doc/guides/nics/features/rtap.ini +4 -0
@@ -4,6 +4,10 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y +MTU update = Y +Promiscuous mode = Y +Allmulticast mode = Y Scattered Rx = P Basic stats = Y Stats per queue = Y
drivers/net/rtap/rtap.h +1 -0
@@ -69,6 +69,7 @@ struct rtap_pmd {/* rtap_ethdev.c */ int rtap_queue_open (struct rte_eth_dev *dev,uint16_t queue_id);void rtap_queue_close (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_link_update (struct rte_eth_dev *dev,int wait_to_complete);/* rtap_netlink.c */ int rtap_nl_open (unsigned int groups);
drivers/net/rtap/rtap_ethdev.c +102 -0
123 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 6/11 net/rtap: add checksum and TSO offload support
In reply to Stephen HemmingerAdd transmit and receive offload support using the virtio-net header that is exchanged with the kernel TAP device.
Tx offloads:
- UDP and TCP checksum offload: set VIRTIO_NET_HDR_F_NEEDS_CSUM with appropriate csum_start and csum_offset fields
- TCP segmentation offload (TSO): set gso_type and gso_size
Rx offloads:
- Enable receive offloads based on configuration
Report offload capabilities in dev_info.
--- doc/guides/nics/features/rtap.ini | 2 ++ drivers/net/rtap/rtap_ethdev.c | 30 +++++++++++++++++++++++++++++- 2 files changed, 31 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +2 -0
@@ -11,6 +11,8 @@ Allmulticast mode = Y Scattered Rx = P Basic stats = Y Stats per queue = Y +TSO = Y +L4 checksum offload = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap_ethdev.c +29 -1
@@ -31,6 +31,16 @@#define RTAP_DEFAULT_IFNAME "rtap%d" +#define RTAP_TX_OFFLOAD (RTE_ETH_TX_OFFLOAD_MULTI_SEGS | \ + RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_TSO) + +#define RTAP_RX_OFFLOAD (RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \ + RTE_ETH_RX_OFFLOAD_TCP_LRO | \ + RTE_ETH_RX_OFFLOAD_SCATTER) +#define RTAP_DEFAULT_BURST 64#define RTAP_NUM_BUFFERS 1024#define RTAP_MAX_QUEUES 128
@@ -250,7 +260,21 @@ rtap_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } -if (ioctl (pmd->keep_fd,TUNSETOFFLOAD ,0 ) !=0 ) { +/* +* Set offload flags visible on the kernel network interface. +* This controls whether kernel will use checksum offload etc. +* Note: kernel transmit is DPDK receive. +*/ +const struct rte_eth_rxmode *rx_mode = &dev->data->dev_conf.rxmode; +unsigned int offload =0 ; +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_CHECKSUM ) { + offload |=TUN_F_CSUM ; + +if (rx_mode->offloads &RTE_ETH_RX_OFFLOAD_TCP_LRO ) + offload |=TUN_F_TSO4 |TUN_F_TSO6 |TUN_F_TSO_ECN ; + } + +if (ioctl (pmd->keep_fd,TUNSETOFFLOAD , offload) !=0 ) {int ret = -errno;PMD_LOG (ERR ,"ioctl(TUNSETOFFLOAD) failed: %s" ,strerror (errno));
@@ -271,6 +295,10 @@ rtap_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->min_rx_bufsize =RTAP_MIN_RX_BUFSIZE ; dev_info->max_rx_queues =RTAP_MAX_QUEUES ; dev_info->max_tx_queues =RTAP_MAX_QUEUES ; + dev_info->rx_queue_offload_capa =RTAP_RX_OFFLOAD ; + dev_info->rx_offload_capa = dev_info->rx_queue_offload_capa; + dev_info->tx_queue_offload_capa =RTAP_TX_OFFLOAD ; + dev_info->tx_offload_capa = dev_info->tx_queue_offload_capa; dev_info->default_rxportconf = (struct rte_eth_dev_portconf ) { .burst_size =RTAP_DEFAULT_BURST ,
Signature · 1 line
2.51.0
PATCH v6 7/11 net/rtap: add multi-process support
In reply to Stephen HemmingerAdd support for DPDK multi-process operation. Secondary processes need access to the per-queue TAP file descriptors owned by the primary process.
Implement fd sharing using the DPDK multi-process IPC mechanism:
- Primary registers an MP action handler that responds to fd requests by sending queue fds via rte_mp_reply()
- Secondary process attaches to the existing device and requests fds from primary via rte_mp_request_sync()
The MP action is registered on the first device probe and unregistered when the last device is closed.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/rtap_ethdev.c | 140 ++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+)
doc/guides/nics/features/rtap.ini +1 -0
@@ -13,6 +13,7 @@ Basic stats = Y Stats per queue = Y TSO = Y L4 checksum offload = Y +Multiprocess aware = Y Linux = Y ARMv7 = Y ARMv8 = Y
drivers/net/rtap/rtap_ethdev.c +140 -0
192 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 8/11 net/rtap: add link state change interrupt
In reply to Stephen HemmingerAdd link state change (LSC) notification support using a netlink socket subscribed to RTMGRP_LINK events.
When enabled, the driver registers a netlink socket with the EAL interrupt handler. On receiving RTM_NEWLINK/RTM_DELLINK messages matching the interface ifindex, the driver updates link status and triggers RTE_ETH_EVENT_INTR_LSC callbacks.
LSC is enabled in dev_start and disabled in dev_stop.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 4 ++ drivers/net/rtap/rtap_ethdev.c | 26 ++++++++- drivers/net/rtap/rtap_intr.c | 87 +++++++++++++++++++++++++++++++ 5 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 drivers/net/rtap/rtap_intr.c
doc/guides/nics/features/rtap.ini +1 -0
@@ -5,6 +5,7 @@ ; [Features] Link status = Y +Link status event = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y
drivers/net/rtap/meson.build +1 -0
@@ -19,6 +19,7 @@ endif sources = files( 'rtap_ethdev.c', + 'rtap_intr.c', 'rtap_netlink.c', 'rtap_rxtx.c', )
drivers/net/rtap/rtap.h +4 -0
@@ -61,6 +61,7 @@ struct rtap_pmd {int keep_fd;/* keep alive file descriptor */ int if_index;/* interface index */ int nlsk_fd;/* netlink control socket */ +struct rte_intr_handle *intr_handle;/* LSC interrupt handle */ struct rte_ether_addr eth_addr;/* address assigned by kernel */ uint64_t rx_drop_base;/* value of rx_dropped when reset */
@@ -98,4 +99,7 @@ int rtap_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, const struct rte_eth_txconf *tx_conf);void rtap_tx_queue_release (struct rte_eth_dev *dev,uint16_t queue_id); +/* rtap_intr.c */ +int rtap_lsc_set (struct rte_eth_dev *dev,int set); +#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +25 -1
@@ -214,11 +214,18 @@ rtap_macaddr_set(struct rte_eth_dev *dev, struct rte_ether_addr *addr)static int rtap_dev_start (struct rte_eth_dev *dev) { -int ret =rtap_set_link_up (dev); +int ret; + ret =rtap_lsc_set (dev,1 );if (ret !=0 )return ret; + ret =rtap_set_link_up (dev); +if (ret !=0 ) { +rtap_lsc_set (dev,0 ); +return ret; + } + dev->data->dev_link.link_status =RTE_ETH_LINK_UP ;for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) { dev->data->rx_queue_state[i] =RTE_ETH_QUEUE_STATE_STARTED ;
@@ -235,6 +242,7 @@ rtap_dev_stop(struct rte_eth_dev *dev) dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +rtap_lsc_set (dev,0 );rtap_set_link_down (dev);for (uint16_t i =0 ; i < dev->data->nb_rx_queues; i++) {
@@ -435,6 +443,9 @@ rtap_dev_close(struct rte_eth_dev *dev)close (pmd->nlsk_fd); pmd->nlsk_fd =-1 ; } + +rte_intr_instance_free (pmd->intr_handle); + pmd->intr_handle =NULL ; }free (dev->process_private);
@@ -522,6 +533,17 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist) pmd->nlsk_fd =-1 ; pmd->rx_drop_base =0 ; +/* Allocate interrupt instance for link state change events */ + pmd->intr_handle =rte_intr_instance_alloc (RTE_INTR_INSTANCE_F_SHARED ); +if (pmd->intr_handle ==NULL ) { +PMD_LOG (ERR ,"Failed to allocate intr handle" ); + goto error; + } +rte_intr_type_set (pmd->intr_handle,RTE_INTR_HANDLE_EXT ); +rte_intr_fd_set (pmd->intr_handle,-1 ); + dev->intr_handle = pmd->intr_handle; + data->dev_flags |=RTE_ETH_DEV_INTR_LSC ; + dev->dev_ops = &rtap_ops;/* Get the initial fd used to keep the tap device around */
@@ -571,6 +593,8 @@ rtap_create(struct rte_eth_dev *dev, const char *tap_name, uint8_t persist)close (pmd->nlsk_fd);if (pmd->keep_fd !=-1 )close (pmd->keep_fd); +rte_intr_instance_free (pmd->intr_handle); + pmd->intr_handle =NULL ;return -1 ; }
drivers/net/rtap/rtap_intr.c new file +87 -0
87 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 9/11 net/rtap: add Rx interrupt support
In reply to Stephen HemmingerAdd support for the DPDK Rx interrupt mechanism, enabling power-aware applications (e.g. l3fwd-power) to sleep until packets arrive rather than busy-polling.
Each Rx queue creates an eventfd during queue setup and registers it with its io_uring instance via io_uring_register_eventfd(). When the kernel posts a CQE (completing a read, i.e. a packet arrived), it signals the eventfd. These per-queue eventfds are wired into a VDEV interrupt handle during dev_start when the application has set intr_conf.rxq.
The enable op drains the eventfd counter to re-arm notification; disable is a no-op since the application simply stops polling. The eventfd is always created unconditionally so it is available if the application enables Rx interrupts later.
The Rx interrupt handle is kept separate from the existing LSC netlink interrupt handle to avoid coupling the two mechanisms.
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/rtap.h | 6 ++ drivers/net/rtap/rtap_ethdev.c | 26 +++++++ drivers/net/rtap/rtap_intr.c | 120 ++++++++++++++++++++++++++++++ drivers/net/rtap/rtap_rxtx.c | 31 +++++++- 5 files changed, 183 insertions(+), 1 deletion(-)
doc/guides/nics/features/rtap.ini +1 -0
@@ -6,6 +6,7 @@ [Features] Link status = Y Link status event = Y +Rx interrupt = Y MTU update = Y Promiscuous mode = Y Allmulticast mode = Y
drivers/net/rtap/rtap.h +6 -0
@@ -38,6 +38,7 @@ extern int rtap_logtype;struct rtap_rx_queue {struct rte_mempool *mb_pool;/* rx buffer pool */ struct io_uring io_ring;/* queue of posted read's */ +int intr_fd;/* eventfd for Rx interrupt */ uint16_t port_id;uint16_t queue_id;
@@ -62,6 +63,7 @@ struct rtap_pmd {int if_index;/* interface index */ int nlsk_fd;/* netlink control socket */ struct rte_intr_handle *intr_handle;/* LSC interrupt handle */ +struct rte_intr_handle *rx_intr_handle;/* Rx queue interrupt handle */ struct rte_ether_addr eth_addr;/* address assigned by kernel */ uint64_t rx_drop_base;/* value of rx_dropped when reset */
@@ -101,5 +103,9 @@ void rtap_tx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id);/* rtap_intr.c */ int rtap_lsc_set (struct rte_eth_dev *dev,int set); +int rtap_rx_intr_vec_install (struct rte_eth_dev *dev); +void rtap_rx_intr_vec_uninstall (struct rte_eth_dev *dev); +int rtap_rx_queue_intr_enable (struct rte_eth_dev *dev,uint16_t queue_id); +int rtap_rx_queue_intr_disable (struct rte_eth_dev *dev,uint16_t queue_id);#endif /* _RTAP_H_ */
drivers/net/rtap/rtap_ethdev.c +26 -0
@@ -220,8 +220,18 @@ rtap_dev_start(struct rte_eth_dev *dev)if (ret !=0 )return ret; +/* Install Rx interrupt vector if requested by application */ +if (dev->data->dev_conf.intr_conf.rxq) { + ret =rtap_rx_intr_vec_install (dev); +if (ret !=0 ) { +rtap_lsc_set (dev,0 ); +return ret; + } + } + ret =rtap_set_link_up (dev);if (ret !=0 ) { +rtap_rx_intr_vec_uninstall (dev);rtap_lsc_set (dev,0 );return ret; }
@@ -242,6 +252,7 @@ rtap_dev_stop(struct rte_eth_dev *dev) dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; +rtap_rx_intr_vec_uninstall (dev);rtap_lsc_set (dev,0 );rtap_set_link_down (dev);
@@ -275,6 +286,16 @@ rtap_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } +/* +* LSC and Rx queue interrupts both need dev->intr_handle, +* so they cannot be enabled simultaneously. +*/ +if (dev->data->dev_conf.intr_conf.lsc && + dev->data->dev_conf.intr_conf.rxq) { +PMD_LOG (ERR ,"LSC and Rx queue interrupts are mutually exclusive" ); +return -ENOTSUP ; + } +/* * Set offload flags visible on the kernel network interface. * This controls whether kernel will use checksum offload etc.
@@ -444,6 +465,9 @@ rtap_dev_close(struct rte_eth_dev *dev) pmd->nlsk_fd =-1 ; } +rte_intr_instance_free (pmd->rx_intr_handle); + pmd->rx_intr_handle =NULL ; +rte_intr_instance_free (pmd->intr_handle); pmd->intr_handle =NULL ; }
@@ -521,6 +545,8 @@ static const struct eth_dev_ops rtap_ops = { .rx_queue_release = rtap_rx_queue_release, .tx_queue_setup = rtap_tx_queue_setup, .tx_queue_release = rtap_tx_queue_release, + .rx_queue_intr_enable = rtap_rx_queue_intr_enable, + .rx_queue_intr_disable = rtap_rx_queue_intr_disable, }; static int
drivers/net/rtap/rtap_intr.c +120 -0
123 lines not on this page · raw message
drivers/net/rtap/rtap_rxtx.c +30 -1
@@ -10,6 +10,7 @@#include <stddef.h> #include <string.h> #include <liburing.h> +#include <sys/eventfd.h> #include <sys/types.h> #include <sys/uio.h> #include <linux/virtio_net.h>
@@ -437,6 +438,7 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d rxq->mb_pool = mb_pool; rxq->port_id = dev->data->port_id; rxq->queue_id = queue_id; + rxq->intr_fd =-1 ; dev->data->rx_queues[queue_id] = rxq;if (io_uring_queue_init (nb_rx_desc, &rxq->io_ring,0 ) !=0 ) {
@@ -444,10 +446,26 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_d goto error_rxq_free; } +/* +* Create an eventfd for Rx interrupt notification. +* io_uring will signal this fd whenever a CQE is posted, +* enabling power-aware applications to sleep until packets arrive. +*/ + rxq->intr_fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC ); +if (rxq->intr_fd <0 ) { +PMD_LOG (ERR ,"eventfd failed: %s" ,strerror (errno)); + goto error_iouring_exit; + } + +if (io_uring_register_eventfd (&rxq->io_ring, rxq->intr_fd) <0 ) { +PMD_LOG (ERR ,"io_uring_register_eventfd failed: %s" ,strerror (errno)); + goto error_eventfd_close; + } + mbufs =calloc (nb_rx_desc,sizeof (struct rte_mbuf *));if (mbufs ==NULL ) {PMD_LOG (ERR ,"Rx mbuf pointer alloc failed" ); - goto error_iouring_exit; + goto error_eventfd_close; }/* open shared tap fd maybe already setup */
@@ -494,6 +512,11 @@ rtap_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_rx_drtap_cancel_all (&rxq->io_ring);rtap_queue_close (dev, queue_id);free (mbufs); +error_eventfd_close: +if (rxq->intr_fd >=0 ) { +close (rxq->intr_fd); + rxq->intr_fd =-1 ; + } error_iouring_exit:io_uring_queue_exit (&rxq->io_ring); error_rxq_free:
@@ -509,6 +532,12 @@ rtap_rx_queue_release(struct rte_eth_dev *dev, uint16_t queue_id)if (rxq ==NULL )return ; +if (rxq->intr_fd >=0 ) { +io_uring_unregister_eventfd (&rxq->io_ring); +close (rxq->intr_fd); + rxq->intr_fd =-1 ; + } +rtap_cancel_all (&rxq->io_ring);io_uring_queue_exit (&rxq->io_ring);
Signature · 1 line
2.51.0
PATCH v6 10/11 net/rtap: add extended statistics support
In reply to Stephen HemmingerAdd xstats support to rtap PMD following the virtio PMD design pattern.
Statistics provided per queue (14 Rx + 12 Tx):
- Packet size distribution (6 buckets: 64, 65-127, 128-255, 256-511, 512-1023, 1024-1518 bytes)
- Packet type classification (broadcast, multicast, unicast)
- Rx offload stats (LRO, checksum validation, mbuf alloc failures)
- Tx offload stats (TSO, checksum offload, multi-segment packets)
Statistics follow DPDK naming convention:
{direction}_q{queue_id}_{detail}_{unit}
Examples:
rx_q0_size_64_packets
rx_q0_broadcast_packets
rx_q0_lro_packets
tx_q0_tso_packets
tx_q0_checksum_offload_packets
--- doc/guides/nics/features/rtap.ini | 1 + drivers/net/rtap/meson.build | 1 + drivers/net/rtap/rtap.h | 41 +++++ drivers/net/rtap/rtap_ethdev.c | 3 + drivers/net/rtap/rtap_rxtx.c | 3 + drivers/net/rtap/rtap_xstats.c | 293 ++++++++++++++++++++++++++++++ 6 files changed, 342 insertions(+) create mode 100644 drivers/net/rtap/rtap_xstats.c
doc/guides/nics/features/rtap.ini +1 -0
@@ -15,6 +15,7 @@ Basic stats = Y Stats per queue = Y TSO = Y L4 checksum offload = Y +Extended stats = P Multiprocess aware = Y Linux = Y ARMv7 = Y
drivers/net/rtap/meson.build +1 -0
@@ -22,6 +22,7 @@ sources = files( 'rtap_intr.c', 'rtap_netlink.c', 'rtap_rxtx.c', + 'rtap_xstats.c', ) ext_deps += liburing
drivers/net/rtap/rtap.h +41 -0
65 lines not on this page · raw message
drivers/net/rtap/rtap_ethdev.c +3 -0
@@ -541,6 +541,9 @@ static const struct eth_dev_ops rtap_ops = { .allmulticast_disable = rtap_allmulticast_disable, .stats_get = rtap_stats_get, .stats_reset = rtap_stats_reset, + .xstats_get = rtap_xstats_get, + .xstats_get_names = rtap_xstats_get_names, + .xstats_reset = rtap_xstats_reset, .rx_queue_setup = rtap_rx_queue_setup, .rx_queue_release = rtap_rx_queue_release, .tx_queue_setup = rtap_tx_queue_setup,
drivers/net/rtap/rtap_rxtx.c +3 -0
@@ -289,6 +289,7 @@ rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)PMD_RX_LOG (ERR ,"Rx mbuf alloc failed" ); dev->data->rx_mbuf_alloc_failed++; + rxq->xstats.mbuf_alloc_failed++; nmb = mb;/* Reuse original */ goto resubmit;
@@ -317,6 +318,7 @@ rtap_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts) mb->port = rxq->port_id;__rte_mbuf_sanity_check (mb,1 ); +rtap_rx_xstats_update (rxq, mb); num_bytes += mb->pkt_len; bufs[num_rx++] = mb;
@@ -735,6 +737,7 @@ rtap_tx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)io_uring_sqe_set_data (sqe, mb);rtap_tx_offload (hdr, mb); +rtap_tx_xstats_update (txq, mb);PMD_TX_LOG (DEBUG ,"write m=%p segs=%u" , mb, mb->nb_segs);
drivers/net/rtap/rtap_xstats.c new file +293 -0
293 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 11/11 test: add unit tests for rtap PMD
In reply to Stephen HemmingerAdd a comprehensive test suite for the rtap poll mode driver covering:
- Device creation and configuration
- Device info query
- Link status and link up/down control
- Statistics collection and reset
- Promiscuous and all-multicast mode toggling
- MAC address and MTU configuration
- Multi-queue setup, teardown, and queue count reduction
- Mismatched Rx/Tx queue count rejection
- Queue reconfiguration (stop, reconfigure, restart)
- Rx path: inject packet via AF_PACKET, receive via DPDK
- Tx path: transmit via DPDK, capture via AF_PACKET
- Multi-segment Tx with chained mbufs
- Multi-segment Rx with small-buffer mempool
- Offload capability reporting and configuration
- Tx UDP checksum offload end-to-end verification
- File descriptor leak detection after device close
- Command-line created device testing
Tests require CAP_NET_ADMIN or root privileges
--- app/test/meson.build | 1 + app/test/test_pmd_rtap.c | 2620 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 2621 insertions(+) create mode 100644 app/test/test_pmd_rtap.c
app/test/meson.build +1 -0
@@ -143,6 +143,7 @@ source_file_deps = { 'test_pie.c': ['sched'], 'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps, 'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'], + 'test_pmd_rtap.c': ['net_rtap', 'ethdev', 'bus_vdev'], 'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'], 'test_pmu.c': ['pmu'], 'test_power.c': ['power', 'power_acpi', 'power_kvm_vm', 'power_intel_pstate',
app/test/test_pmd_rtap.c new file +2620 -0
2,620 lines not on this page · raw message
Signature · 1 line
2.51.0
RE: [PATCH v6 00/11] net/rtap: add io_uring based TAP driver
In reply to Stephen HemmingerQuoted 10 lines
This series adds
net_rtap, an experimental poll mode driver that uses Linuxio_uringfor asynchronous packet I/O with kernel TAP interfaces.Like
net_tap,net_rtapcreates a kernel network interface visible to standard tools (ip, ethtool) and the Linux TCP/IP stack. From DPDK it is an ordinary ethdev.
Quoted 8 lines
Motivation
----------This driver started as an experiment to determine whether Linux
io_uringcould deliver better packet I/O performance than the traditional read()/write() system calls used bynet_tap. By posting batches of I/O requests asynchronously,io_uringamortizes system call overhead across multiple packets.
Sounds interesting...
Curious did you make any perf comparisons vs our traditional tap?
Quoted 91 lines
The project also served as a testbed for using AI tooling to help build a comprehensive test suite, refactor code, and improve documentation. The result is intended as an example for other PMD
authors: the driver has thorough unit tests covering data path, offloads, multi-queue, fd lifecycle, and more, along with detailed code comments explaining design choices.Why not extend
net_tap?
-----------------------The existing
net_tapdriver was designed to provide feature parity with mlx5 when used behind the failsafe PMD. That goal led to significant complexity:rte_flowsupport emulated via eBPF programs, software GSO implementation, and other features that duplicate in user space what the kernel already does.
net_rtaptakes the opposite approach -- use the kernel efficiently and let it do what it does well. There is norte_flowsupport; receive queue selection is left to the kernel's native RSS/steering. There is no software GSO; the driver passes segmentation requests to the kernel via the virtio-net header and lets the kernel handle it. The result is a much simpler driver that is easier to maintain and reason about.Given these fundamentally different design goals, a clean implementation was more practical than refactoring
net_tap.Acknowledgement
---------------Parts of the test suite, code review, and refactoring were done with the assistance of Anthropic Claude (AI). All generated code was reviewed and tested by the author.
Requirements:
- Kernel headers withIORING_ASYNC_CANCEL_ALL(upstream since 5.19)
- liburing >= 2.0Should work on distributions: Debian 12+, Ubuntu 24.04+, Fedora 37+, SLES 15 SP6+ / openSUSE Tumbleweed. RHEL 9 is not supported (
io_uringis disabled by default).v6:
- fix lots of bugs found doing automated review
- convert to use ifindex and netlink to avoid rename issues
- implement xstatsStephen Hemminger (11):
net/rtap: add driver skeleton and documentation net/rtap: add TAP device creation and queue management net/rtap: add Rx/Tx with scatter/gather support net/rtap: add statistics and device info net/rtap: add link and device management operations net/rtap: add checksum and TSO offload support net/rtap: add multi-process support
net/rtap: add link state change interrupt net/rtap: add Rx interrupt support
net/rtap: add extended statistics support
test: add unit tests for rtap PMDMAINTAINERS | 7 + app/test/meson.build | 1 + app/test/test_pmd_rtap.c | 2620 ++++++++++++++++++++++++ doc/guides/nics/features/rtap.ini | 26 + doc/guides/nics/index.rst | 1 + doc/guides/nics/rtap.rst | 101 + doc/guides/rel_notes/release_26_03.rst | 7 + drivers/net/meson.build | 1 + drivers/net/rtap/meson.build | 30 + drivers/net/rtap/rtap.h | 152 ++ drivers/net/rtap/rtap_ethdev.c | 864 ++++++++ drivers/net/rtap/rtap_intr.c | 207 ++ drivers/net/rtap/rtap_netlink.c | 445 ++++ drivers/net/rtap/rtap_rxtx.c | 803 ++++++++ drivers/net/rtap/rtap_xstats.c | 293 +++ 15 files changed, 5558 insertions(+) create mode 100644 app/test/test_pmd_rtap.c create mode 100644 doc/guides/nics/features/rtap.ini create mode 100644 doc/guides/nics/rtap.rst create mode 100644 drivers/net/rtap/meson.build create mode 100644 drivers/net/rtap/rtap.h create mode 100644 drivers/net/rtap/rtap_ethdev.c create mode 100644 drivers/net/rtap/rtap_intr.c create mode 100644 drivers/net/rtap/rtap_netlink.c create mode 100644 drivers/net/rtap/rtap_rxtx.c create mode 100644 drivers/net/rtap/rtap_xstats.c--
2.51.0
Re: [PATCH v6 00/11] net/rtap: add io_uring based TAP driver
Quoted 9 lines · On Sun, 15 Feb 2026 08:58:14 +0000 Konstantin Ananyev…
Quoted 6 lines
This driver started as an experiment to determine whether Linux
io_uringcould deliver better packet I/O performance than the traditional read()/write() system calls used bynet_tap. By posting batches of I/O requests asynchronously,io_uringamortizes system call overhead across multiple packets.Sounds interesting...
Curious did you make any perf comparisons vs our traditional tap?
That is next step. Trying to get a testbed that is realistic setup.