bus/vmbus: add device argument to configure if NUMA information on the device should be ignored
PATCH v2 1/1 bus/vmbus: add device argument to configure if NUMA information on the device should be ignored
From: Long Li <[email protected]>
In most cases, netvsc is used with a VF device. The application generally runs with better performance when all the device memory is allocated on VF's NUMA node, as the VF device carries most of the data packets.
But sometimes netvsc may run on a different NUMA node than that of the VF. This patch adds a device argument "numa" to allow the application to configure if this vmbus device (netvsc) should be NUMA aware. The default behavior is NUMA aware. Setting "numa=0" specifies this vmbus device is not NUMA aware.
The device argument is parsed at the time of vmbus device probe to ensure all the following memory allocations are done on the configured NUMA node.
---
Change log
v2: move the code for parsing device arguments to vmbus
doc/guides/nics/netvsc.rst | 8 ++++ drivers/bus/vmbus/bus_vmbus_driver.h | 15 +++++++ drivers/bus/vmbus/linux/vmbus_bus.c | 64 +++++++++++++++++++++++----- drivers/net/netvsc/hn_ethdev.c | 12 ------ 4 files changed, 76 insertions(+), 23 deletions(-)
doc/guides/nics/netvsc.rst +8 -0
@@ -141,3 +141,11 @@ The user can specify below argument in devargs. A non-zero value tells netvsc to attach external buffers to mbuf on receiving packets, thus avoid copying memory. Use of external buffers requires the application is able to read data from external mbuf. + +#. ``numa``: + The numa is used to configure if netvsc driver should be NUMA aware. The + default value is 1 (meaning the netvsc driver is NUMA aware). When used + with a VF device, the VF device's NUMA node may not be the same as netvsc's + NUMA node. The application generally runs with better performance if all the + device queues are allocated on VF's NUMA node, as the VF carries most of the + data packets. Setting this value to 0 makes netvsc driver not NUMA aware.
drivers/bus/vmbus/bus_vmbus_driver.h +15 -0
@@ -17,6 +17,21 @@ extern "C" {struct vmbus_channel ;struct vmbus_mon_page ; +#define NETVSC_ARG_LATENCY "latency" +#define NETVSC_ARG_RXBREAK "rx_copybreak" +#define NETVSC_ARG_TXBREAK "tx_copybreak" +#define NETVSC_ARG_RX_EXTMBUF_ENABLE "rx_extmbuf_enable" +#define NETVSC_ARG_NUMA "numa" + +static const char *const valid_keys[] = { +NETVSC_ARG_LATENCY , +NETVSC_ARG_RXBREAK , +NETVSC_ARG_TXBREAK , +NETVSC_ARG_RX_EXTMBUF_ENABLE , +NETVSC_ARG_NUMA , +NULL +}; +/** Maximum number of VMBUS resources. */ enum hv_uio_map {HV_TXRX_RING_MAP =0 ,
drivers/bus/vmbus/linux/vmbus_bus.c +53 -11
84 lines not on this page · raw message
drivers/net/netvsc/hn_ethdev.c +0 -12
@@ -52,11 +52,6 @@ RTE_ETH_RX_OFFLOAD_VLAN_STRIP | \ RTE_ETH_RX_OFFLOAD_RSS_HASH) -#define NETVSC_ARG_LATENCY "latency" -#define NETVSC_ARG_RXBREAK "rx_copybreak" -#define NETVSC_ARG_TXBREAK "tx_copybreak" -#define NETVSC_ARG_RX_EXTMBUF_ENABLE "rx_extmbuf_enable" -/* The max number of retry when hot adding a VF device */ #define NETVSC_MAX_HOTADD_RETRY 10
@@ -199,13 +194,6 @@ static int hn_parse_args(const struct rte_eth_dev *dev) {struct hn_data *hv = dev->data->dev_private;struct rte_devargs *devargs = dev->device->devargs; -static const char *const valid_keys[] = { -NETVSC_ARG_LATENCY , -NETVSC_ARG_RXBREAK , -NETVSC_ARG_TXBREAK , -NETVSC_ARG_RX_EXTMBUF_ENABLE , -NULL - };struct rte_kvargs *kvlist;int ret;
Signature · 1 line
2.34.1
Quoted 12 lines · On Mon, 18 Aug 2025 17:24:26 -0700 [email protected] wrote:
From: Long Li <[email protected]>
In most cases, netvsc is used with a VF device. The application generally runs with better performance when all the device memory is allocated on VF's NUMA node, as the VF device carries most of the data packets.
But sometimes netvsc may run on a different NUMA node than that of the VF. This patch adds a device argument "numa" to allow the application to configure if this vmbus device (netvsc) should be NUMA aware. The default behavior is NUMA aware. Setting "numa=0" specifies this vmbus device is not NUMA aware.
Quoted 4 lines
The device argument is parsed at the time of vmbus device probe to ensure all the following memory allocations are done on the configured NUMA node.
Signed-off-by Long Li <[email protected]>
Putting all the args in vmbus seems confusing, and not what I meant. Was more thing that ethdev and queues should be on same NUMA node.
If NUMA is better why wouldn't it always be on?
Since netvsc is really a virtual device not sure.
I suspect 99% of users will be using with VF, that should be the default.
Quoted 24 lines
Subject: [EXTERNAL] Re: [PATCH v2] bus/vmbus: add device argument to configure if NUMA information on the device should be ignored
Quoted 5 lines · On Mon, 18 Aug 2025 17:24:26 -0700 [email protected] wrote:
From: Long Li <[email protected]>
In most cases, netvsc is used with a VF device. The application generally runs with better performance when all the device memory is allocated on VF's NUMA node, as the VF device carries most of the data
packets.
Quoted 2 lines
But sometimes netvsc may run on a different NUMA node than that of the
VF.
Quoted 7 lines
This patch adds a device argument "numa" to allow the application to configure if this vmbus device (netvsc) should be NUMA aware. The default behavior is NUMA aware. Setting "numa=0" specifies this vmbus device is not NUMA aware.
The device argument is parsed at the time of vmbus device probe to ensure all the following memory allocations are done on the configured
NUMA node.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Signed-off-by Long Li <[email protected]>Putting all the args in vmbus seems confusing, and not what I meant. Was more thing that ethdev and queues should be on same NUMA node.
If NUMA is better why wouldn't it always be on?
Since netvsc is really a virtual device not sure.
I suspect 99% of users will be using with VF, that should be the default.
Sorry I misread your previous comment on handling this in VMBUS code. I'm still a little confused.
Should I add a "numa_aware" device argument for netvsc, and have VMBUS code to parse this argument before calling the device probe function of netvsc?
Maybe all the vmbus device should default to be not NUMA aware. But I don't know if there are use cases using netvsc without VF and if NUMA makes a difference in those situations.
It's difficult to decide the NUMA node for a VF in advance, since they can be hot added at any time after DPDK starts.