app/testpmd: fix multi-pool Rx setup with --no-NUMA
PATCH 1/1 app/testpmd: fix multi-pool Rx setup with --no-NUMA
When NUMA is disabled, mbuf pools are allocated on SOCKET_ID_ANY, but rx_queue_setup() looked them up using port->socket_id. Per-segment pool lookup failed and all segments fell back to the first pool, causing Rx queue setup to fail when a later segment needed a larger mbuf (e.g. --mbuf-size=314,978 --rxpkts=186,978).
Check numa_support inside mbuf_pool_find() and drop the duplicate NUMA socket remapping from start_port().
--- app/test-pmd/testpmd.c | 3 +-- app/test-pmd/testpmd.h | 3 +++ 2 files changed, 4 insertions(+), 2 deletions(-)
app/test-pmd/testpmd.c +1 -2
@@ -3153,8 +3153,7 @@ start_port(portid_t pid) }else {struct rte_mempool *mp =mbuf_pool_find_first - ((numa_support ? port->socket_id : - (unsigned int )SOCKET_ID_ANY )); + (port->socket_id);if (mp ==NULL ) { fprintf(stderr, "Failed to setup RX queue: No mempool allocation on the socket %d\n",
app/test-pmd/testpmd.h +3 -0
@@ -891,6 +891,9 @@ mbuf_pool_find(unsigned int sock_id, uint16_t idx) {char pool_name[RTE_MEMPOOL_NAMESIZE ]; +if (!numa_support) + sock_id = (unsigned int )SOCKET_ID_ANY ; +mbuf_poolname_build (sock_id, pool_name,sizeof (pool_name), idx);return rte_mempool_lookup ((const char *)pool_name); }
Signature · 1 line
2.21.0
cover app/testpmd: fix socket id handling with NUMA disabled
Alternative to the patch "app/testpmd: fix multi-pool Rx setup with --no-NUMA" from Maayan Kashani, fixing the problem at the source
rather than remapping inside mbuf_pool_find().
Since commit 835fd4893a31 ("app/testpmd: support cross-NUMA allocations"), mbuf pools are created (and named) on SOCKET_ID_ANY
when NUMA support is disabled, but port->socket_id still holds 0/--socket-num. Every name-based pool lookup that does not go through the single workaround in start_port() misses. The reported multi-pool Rx failure is one instance; plain "testpmd --no-numa --forward-mode=txonly" also crashes with a NULL mempool dereference, and --socket-num no longer affects pool allocation.
Patch 1 restores the invariant that port->socket_id equals the socket id the pools are created and named with, and removes the start_port() workaround. It should go to stable (the regression shipped in 25.07 and 25.11); it cherry-picks onto 25.11 with trivial context conflicts and was build- and run-tested there.
Patch 2 consolidates the three diverging copies of the port socket id policy into one helper; this also fixes hot-attached ports ignoring --port-numa-config, --no-numa and --socket-num.
Patch 3 prints SOCKET_ID_ANY as "any" instead of 4294967295.
Patch 4 adds the error string to the interactive queue setup command failure message; rte_eth_rx/tx_queue_setup() can return -EBUSY silently when the port is started and the PMD lacks runtime queue setup support.
Stephen Hemminger (4):
app/testpmd: fix port socket ID with NUMA disabled app/testpmd: consolidate port socket ID computation app/testpmd: display any socket ID as text app/testpmd: report reason when queue setup command fails
app/test-pmd/cmdline.c | 12 +++-- app/test-pmd/config.c | 18 ++++--- app/test-pmd/testpmd.c | 120 +++++++++++++++++++++-------------------- app/test-pmd/testpmd.h | 11 ++++ 4 files changed, 92 insertions(+), 69 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/4 app/testpmd: fix port socket ID with NUMA disabled
When NUMA support is disabled, mbuf pools are created on SOCKET_ID_ANY but port->socket_id is still set to 0 (or --socket-num). Pool lookup is by name, and the name embeds the socket id the pool was created with, so every lookup that does not go through the one workaround in start_port() misses:
- the per-lcore pool assignment in init_config() leaves fwd_lcores[]->mbp NULL, so "testpmd --no-numa --forward-mode=txonly" crashes with a NULL dereference in pkt_burst_transmit();
- the per-segment and multi-mempool lookups in rx_queue_setup() fall back to the first pool, so buffer split fails when a later segment needs a larger mbuf
(e.g. --mbuf-size=314,978 --rxpkts=186,978);
- the interactive "port <id> rxq|txq <id> setup" command cannot find a pool.
The --socket-num option is also ignored for pool allocation since the offending commit.
Fix this at the source instead of adding another workaround at a lookup site: make port->socket_id hold the same socket id the pools are created and named with. Add uma_socket_id() returning the pool socket key used when NUMA support is disabled (--socket-num if given, otherwise SOCKET_ID_ANY) and use it for pool creation, port->socket_id and the per-lcore pool fallback. The workaround in start_port() is then unnecessary.
Passing SOCKET_ID_ANY to rte_eth_rx/tx_queue_setup() is allowed by the ethdev API and matches the cross-NUMA intent of the offending commit.
--- app/test-pmd/testpmd.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
app/test-pmd/testpmd.c +17 -10
@@ -1729,6 +1729,17 @@ init_config_port_offloads(portid_t pid, uint32_t socket_id) } } +/* +* Socket id that mbuf pools are created (and named) with when NUMA +* support is disabled: --socket-num if given, otherwise SOCKET_ID_ANY. +*/ +static unsigned int +uma_socket_id (void ) +{ +return socket_num ==UMA_NO_CONFIG ? + (unsigned int )SOCKET_ID_ANY : socket_num; +} +static void init_config (void ) {
@@ -1778,8 +1789,7 @@ init_config(void) socket_id = socket_ids[0 ]; } }else { - socket_id = (socket_num ==UMA_NO_CONFIG ) ? -0 : socket_num; + socket_id =uma_socket_id (); }/* Apply default TxRx configuration for all ports */ init_config_port_offloads (pid, socket_id);
@@ -1823,7 +1833,7 @@ init_config(void) mempools[i] =mbuf_pool_create (mbuf_data_size[i], nb_mbuf_per_pool, -SOCKET_ID_ANY , i); +uma_socket_id (), i); } }
@@ -1841,7 +1851,8 @@ init_config(void) rte_lcore_to_socket_id(fwd_lcores_cpuids[lc_id]));if (mbp ==NULL ) - mbp =mbuf_pool_find_first (0 ); + mbp =mbuf_pool_find_first (numa_support ? +0 :uma_socket_id ()); fwd_lcores[lc_id]->mbp = mbp;#ifdef RTE_LIB_GSO /* initialize GSO context */
@@ -1920,10 +1931,7 @@ init_fwd_streams(void) } }else { -if (socket_num ==UMA_NO_CONFIG ) - port->socket_id =0 ; -else - port->socket_id = socket_num; + port->socket_id =uma_socket_id (); } }
@@ -3153,8 +3161,7 @@ start_port(portid_t pid) }else {struct rte_mempool *mp =mbuf_pool_find_first - ((numa_support ? port->socket_id : - (unsigned int )SOCKET_ID_ANY )); + (port->socket_id);if (mp ==NULL ) { fprintf(stderr, "Failed to setup RX queue: No mempool allocation on the socket %d\n",
Signature · 1 line
2.53.0
PATCH 2/4 app/testpmd: consolidate port socket ID computation
In reply to Stephen HemmingerThe socket id of a port was computed in three places with diverging copies of the same policy:
- init_config() and init_fwd_streams() duplicated the full policy (--port-numa-config, device socket id, first available socket, --socket-num);
- setup_attached_port() used only the device socket id and ignored --port-numa-config, --no-numa and --socket-num, so a hot-attached port could be keyed to a socket where no mbuf pool exists.
Move the policy into a single helper, port_socket_id(), and use it in all three places.
--- app/test-pmd/testpmd.c | 69 +++++++++++++++++------------------------- 1 file changed, 28 insertions(+), 41 deletions(-)
app/test-pmd/testpmd.c +28 -41
@@ -1740,6 +1740,31 @@ uma_socket_id(void) (unsigned int)SOCKET_ID_ANY : socket_num; } +/* +* Socket id to use for a port: the socket its mbuf pools are created +* on and where its queue resources are placed. +*/ +static unsigned int +port_socket_id (portid_t pid) +{ +unsigned int socket_id; + +if (!numa_support) +return uma_socket_id (); + + socket_id = port_numa[pid]; +if (socket_id ==NUMA_NO_CONFIG ) { + socket_id =rte_eth_dev_socket_id (pid); +/* +* If socket_id is invalid, +* set to the first available socket. +*/ +if (check_socket_id (socket_id) <0 ) + socket_id = socket_ids[0 ]; + } +return socket_id; +} +static void init_config (void ) {
@@ -1774,25 +1799,8 @@ init_config(void) }RTE_ETH_FOREACH_DEV (pid) { -uint32_t socket_id; - -if (numa_support) { - socket_id = port_numa[pid]; -if (port_numa[pid] ==NUMA_NO_CONFIG ) { - socket_id =rte_eth_dev_socket_id (pid); - -/* -* if socket_id is invalid, -* set to the first available socket. -*/ -if (check_socket_id (socket_id) <0 ) - socket_id = socket_ids[0 ]; - } - }else { - socket_id =uma_socket_id (); - }/* Apply default TxRx configuration for all ports */ -init_config_port_offloads (pid, socket_id); +init_config_port_offloads (pid,port_socket_id (pid)); }/* * Create pools of mbuf.
@@ -1916,23 +1924,7 @@ init_fwd_streams(void) nb_txq, port->dev_info.max_tx_queues);return -1 ; } -if (numa_support) { -if (port_numa[pid] !=NUMA_NO_CONFIG ) - port->socket_id = port_numa[pid]; -else { - port->socket_id =rte_eth_dev_socket_id (pid); - -/* -* if socket_id is invalid, -* set to the first available socket. -*/ -if (check_socket_id (port->socket_id) <0 ) - port->socket_id = socket_ids[0 ]; - } - } -else { - port->socket_id =uma_socket_id (); - } + port->socket_id =port_socket_id (pid); } q =RTE_MAX (nb_rxq, nb_txq);
@@ -3604,14 +3596,9 @@ attach_port(char *identifier)static void setup_attached_port (portid_t pi) { -unsigned int socket_id;int ret; - socket_id = (unsigned )rte_eth_dev_socket_id (pi); -/* if socket_id is invalid, set to the first available socket. */ -if (check_socket_id (socket_id) <0 ) - socket_id = socket_ids[0 ]; -reconfig (pi, socket_id); +reconfig (pi,port_socket_id (pi)); ret =rte_eth_promiscuous_enable (pi);if (ret !=0 ) fprintf(stderr,
Signature · 1 line
2.53.0
PATCH 3/4 app/testpmd: display any socket ID as text
In reply to Stephen HemmingerWhen NUMA support is disabled the port socket id is SOCKET_ID_ANY, which was displayed as 4294967295 in "show port info", the "Configuring Port" message, the mbuf pool creation log and the pool lookup error message. Print "any" instead.
--- app/test-pmd/cmdline.c | 6 ++++-- app/test-pmd/config.c | 18 ++++++++++++------ app/test-pmd/testpmd.c | 28 ++++++++++++++++++---------- app/test-pmd/testpmd.h | 11 +++++++++++ 4 files changed, 45 insertions(+), 18 deletions(-)
app/test-pmd/cmdline.c +4 -2
@@ -3078,9 +3078,11 @@ cmd_setup_rxtx_queue_parsed( mp =mbuf_pool_find_first (socket_id);if (mp ==NULL ) { +char buf[16 ]; +fprintf (stderr, -"Failed to setup RX queue: No mempool allocation on the socket %d\n" , - rxring_numa[res->portid]); +"Failed to setup RX queue: No mempool allocation on the socket %s\n" , +socket_id_str (socket_id, buf,sizeof (buf)));return ; } ret = rx_queue_setup(res->portid,
app/test-pmd/config.c +12 -6
@@ -808,6 +808,7 @@ port_infos_display(portid_t port_id)static const char *info_border ="*********************" ;uint16_t mtu;char name[RTE_ETH_NAME_MAX_LEN ]; +char buf[16 ];int ret;char fw_version[ETHDEV_FWVERS_LEN ];uint32_t lanes;
@@ -841,7 +842,8 @@ port_infos_display(portid_t port_id)if (rte_dev_devargs (dev_info.device) &&rte_dev_devargs (dev_info.device)->args)printf ("\nDevargs: %s" ,rte_dev_devargs (dev_info.device)->args); -printf ("\nConnect to socket: %u" , port->socket_id); +printf ("\nConnect to socket: %s" , +socket_id_str (port->socket_id, buf,sizeof (buf)));if (port_numa[port_id] !=NUMA_NO_CONFIG ) { mp =mbuf_pool_find (port_numa[port_id],0 );
@@ -849,7 +851,8 @@ port_infos_display(portid_t port_id)printf ("\nmemory allocation on the socket: %d" , port_numa[port_id]); }else -printf ("\nmemory allocation on the socket: %u" ,port->socket_id); +printf ("\nmemory allocation on the socket: %s" , +socket_id_str (port->socket_id, buf,sizeof (buf)));printf ("\nLink status: %s\n" , (link.link_status) ? ("up" ) : ("down" ));printf ("Link speed: %s\n" ,rte_eth_link_speed_to_str (link.link_speed));
@@ -5642,6 +5645,7 @@ pkt_fwd_config_display(struct fwd_config *cfg)struct fwd_stream *fs;lcoreid_t lc_id;streamid_t sm_id; +char buf[16 ]; printf("%s%s%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " "NUMA support %s, MP allocation mode: %s\n",
@@ -5664,12 +5668,14 @@ pkt_fwd_config_display(struct fwd_config *cfg) fwd_lcores[lc_id]->stream_nb);for (sm_id =0 ; sm_id < fwd_lcores[lc_id]->stream_nb; sm_id++) { fs = fwd_streams[fwd_lcores[lc_id]->stream_idx + sm_id]; -printf ("\n RX P=%d/Q=%d (socket %u) -> TX " -"P=%d/Q=%d (socket %u) " , +printf ("\n RX P=%d/Q=%d (socket %s) -> " , fs->rx_port, fs->rx_queue, - ports[fs->rx_port].socket_id, +socket_id_str (ports[fs->rx_port].socket_id, + buf,sizeof (buf))); +printf ("TX P=%d/Q=%d (socket %s) " , fs->tx_port, fs->tx_queue, - ports[fs->tx_port].socket_id); +socket_id_str (ports[fs->tx_port].socket_id, + buf,sizeof (buf)));print_ethaddr ("peer=" , &peer_eth_addrs[fs->peer_addr]); }
app/test-pmd/testpmd.c +18 -10
@@ -1263,6 +1263,7 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, unsigned int socket_id, uint16_t size_idx) { char pool_name[RTE_MEMPOOL_NAMESIZE]; +char sock_str[16 ];struct rte_mempool *rte_mp =NULL ;#ifndef RTE_EXEC_ENV_WINDOWS uint32_t mb_size;
@@ -1274,14 +1275,17 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, rte_mp =rte_mempool_lookup (pool_name);if (rte_mp ==NULL )rte_exit (EXIT_FAILURE , -"Get mbuf pool for socket %u failed: %s\n" , - socket_id,rte_strerror (rte_errno)); +"Get mbuf pool for socket %s failed: %s\n" , +socket_id_str (socket_id, sock_str, +sizeof (sock_str)), +rte_strerror (rte_errno));return rte_mp; }TESTPMD_LOG (INFO , -"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n" , - pool_name, nb_mbuf, mbuf_seg_size, socket_id); +"create a new mbuf pool <%s>: n=%u, size=%u, socket=%s\n" , + pool_name, nb_mbuf, mbuf_seg_size, +socket_id_str (socket_id, sock_str,sizeof (sock_str)));switch (mp_alloc_type) {case MP_ALLOC_NATIVE :
@@ -1366,8 +1370,9 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,#endif if (rte_mp ==NULL ) {rte_exit (EXIT_FAILURE , -"Creation of mbuf pool for socket %u failed: %s\n" , - socket_id,rte_strerror (rte_errno)); +"Creation of mbuf pool for socket %s failed: %s\n" , +socket_id_str (socket_id, sock_str,sizeof (sock_str)), +rte_strerror (rte_errno)); }else if (verbose_level >0 ) {rte_mempool_dump (stdout, rte_mp); }
@@ -2973,6 +2978,7 @@ int start_port(portid_t pid) {int diag; +char buf[16 ];portid_t pi;portid_t p_pi =RTE_MAX_ETHPORTS ;portid_t pl[RTE_MAX_ETHPORTS ];
@@ -3027,8 +3033,9 @@ start_port(portid_t pid) } }configure_rxtx_dump_callbacks (0 ); -printf ("Configuring Port %d (socket %u)\n" , pi, - port->socket_id); +printf ("Configuring Port %d (socket %s)\n" , pi, +socket_id_str (port->socket_id, + buf,sizeof (buf)));if (nb_hairpinq >0 &&rte_eth_dev_hairpin_capability_get (pi, &cap)) { fprintf(stderr,
@@ -3156,8 +3163,9 @@ start_port(portid_t pid) (port->socket_id);if (mp ==NULL ) {fprintf (stderr, -"Failed to setup RX queue: No mempool allocation on the socket %d\n" , - port->socket_id); +"Failed to setup RX queue: No mempool allocation on the socket %s\n" , +socket_id_str (port->socket_id, + buf,sizeof (buf)));return -1 ; } diag = rx_queue_setup(pi, qi,
app/test-pmd/testpmd.h +11 -0
@@ -873,6 +873,17 @@ current_fwd_lcore(void)void parse_fwd_portlist (const char *port); +/* Format a socket id for display, SOCKET_ID_ANY as "any". */ +static inline const char * +socket_id_str (unsigned int socket_id,char *buf,size_t size) +{ +if (socket_id == (unsigned int )SOCKET_ID_ANY ) +snprintf (buf, size,"any" ); +else +snprintf (buf, size,"%u" , socket_id); +return buf; +} +/* Mbuf Pools */ static inline void mbuf_poolname_build(unsigned int sock_id, char *mp_name,
Signature · 1 line
2.53.0
PATCH 4/4 app/testpmd: report reason when queue setup command fails
In reply to Stephen HemmingerThe interactive "port <id> rxq|txq <id> setup" command printed a bare failure message. rte_eth_rx/tx_queue_setup() can fail with -EBUSY without logging anything, for example when the port is started and the PMD does not support runtime queue setup, leaving the user with no clue:
testpmd> port 0 rxq 0 setup
Failed to setup RX queue
Print the error string:
Failed to setup RX queue: Device or resource busy
--- app/test-pmd/cmdline.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
app/test-pmd/cmdline.c +4 -2
@@ -3092,7 +3092,8 @@ cmd_setup_rxtx_queue_parsed( &port->rxq[res->qid].conf, mp);if (ret) -fprintf (stderr,"Failed to setup RX queue\n" ); +fprintf (stderr,"Failed to setup RX queue: %s\n" , +rte_strerror (-ret)); }else { socket_id = txring_numa[res->portid];if (!numa_support || socket_id ==NUMA_NO_CONFIG )
@@ -3109,7 +3110,8 @@ cmd_setup_rxtx_queue_parsed( socket_id, &port->txq[res->qid].conf);if (ret) -fprintf (stderr,"Failed to setup TX queue\n" ); +fprintf (stderr,"Failed to setup TX queue: %s\n" , +rte_strerror (-ret)); } }
Signature · 1 line
2.53.0
Re: [PATCH 0/4] app/testpmd: fix socket id handling with NUMA disabled
In reply to Stephen HemmingerQuoted 36 lines · Stephen Hemminger
Alternative to the patch "app/testpmd: fix multi-pool Rx setup with --no-NUMA" from Maayan Kashani, fixing the problem at the source
rather than remapping insidembuf_pool_find().Since commit
835fd4893a31("app/testpmd: support cross-NUMA allocations"), mbuf pools are created (and named) onSOCKET_ID_ANY
when NUMA support is disabled, butport->socket_idstill holds 0/--socket-num. Every name-based pool lookup that does not go through the single workaround instart_port()misses. The reported multi-pool Rx failure is one instance; plain "testpmd --no-numa --forward-mode=txonly" also crashes with a NULL mempool dereference, and --socket-num no longer affects pool allocation.Patch 1 restores the invariant that
port->socket_idequals the socket id the pools are created and named with, and removes thestart_port()workaround. It should go to stable (the regression shipped in 25.07 and 25.11); it cherry-picks onto 25.11 with trivial context conflicts and was build- and run-tested there.Patch 2 consolidates the three diverging copies of the port socket id policy into one helper; this also fixes hot-attached ports ignoring --port-numa-config, --no-numa and --socket-num.
Patch 3 prints
SOCKET_ID_ANYas "any" instead of 4294967295.Patch 4 adds the error string to the interactive queue setup command failure message; rte_eth_rx/tx_queue_setup() can return
-EBUSYsilently when the port is started and the PMD lacks runtime queue setup support.Stephen Hemminger (4):
app/testpmd: fix port socket ID with NUMA disabled app/testpmd: consolidate port socket ID computation app/testpmd: display any socket ID as text app/testpmd: report reason when queue setup command fails
Quoted 5 lines
app/test-pmd/cmdline.c | 12 +++-- app/test-pmd/config.c | 18 ++++--- app/test-pmd/testpmd.c | 120 +++++++++++++++++++++-------------------- app/test-pmd/testpmd.h | 11 ++++ 4 files changed, 92 insertions(+), 69 deletions(-)
LGTM.
Series applied, thanks for the cleanups.
Signature · 1 line
David Marchand