vmxnet3: upgrade to version 6
Subsystems (3)
cover vmxnet3: upgrade to version 6
vmxnet3 emulation has recently added several new features which includes increase in queues supported, remove power of 2 limitation on queues, add RSS for ESP IPv6, etc. This patch series extends the vmxnet3 driver to leverage these new features.
Compatibility is maintained using existing vmxnet3 versioning mechanism as follows:
- new features added to vmxnet3 emulation are associated with new vmxnet3 version viz. vmxnet3 version 6.
- emulation advertises all the versions it supports to the driver.
- during initialization, vmxnet3 driver picks the highest version number supported by both the emulation and the driver and configures emulation to run at that version.
In particular, following changes are introduced:
Patch 1:
This patch introduces utility macros for vmxnet3 version 6 comparison and updates Copyright information.
Patch 2:
This patch adds support to increase maximum Tx/Rx queues from 8 to 32.
Patch 3:
This patch removes the limitation of power of 2 on the queues.
Patch 4:
Uses existing get_rss_hash_opts and set_rss_hash_opts methods to add support for ESP IPv6 RSS.
Patch 5:
This patch reports correct RSS hash type based on the type of RSS performed.
Patch 6:
This patch updates maximum configurable mtu to 9190.
Patch 7:
With all vmxnet3 version 6 changes incorporated in the vmxnet3 driver, with this patch, the driver can configure emulation to run at vmxnet3 version 6.
Ronak Doshi (7):
vmxnet3: prepare for version 6 changes
vmxnet3: add support for 32 Tx/Rx queues
vmxnet3: remove power of 2 limitation on the queues
vmxnet3: add support for ESP IPv6 RSS
vmxnet3: set correct hash type based on rss information
vmxnet3: increase maximum configurable mtu to 9190
vmxnet3: update to version 6
drivers/net/vmxnet3/Makefile | 2 +- drivers/net/vmxnet3/upt1_defs.h | 2 +- drivers/net/vmxnet3/vmxnet3_defs.h | 50 ++++++-- drivers/net/vmxnet3/vmxnet3_drv.c | 221 ++++++++++++++++++++++++---------- drivers/net/vmxnet3/vmxnet3_ethtool.c | 22 +++- drivers/net/vmxnet3/vmxnet3_int.h | 22 +++- 6 files changed, 236 insertions(+), 83 deletions(-)
Signature · 1 line
2.11.0
PATCH 1/7 vmxnet3: prepare for version 6 changes
vmxnet3 is currently at version 4 and this patch initiates the preparation to accommodate changes for upto version 6. Introduced utility macros for vmxnet3 version 6 comparison and update Copyright information.
--- drivers/net/vmxnet3/Makefile | 2 +- drivers/net/vmxnet3/upt1_defs.h | 2 +- drivers/net/vmxnet3/vmxnet3_defs.h | 2 +- drivers/net/vmxnet3/vmxnet3_drv.c | 2 +- drivers/net/vmxnet3/vmxnet3_ethtool.c | 2 +- drivers/net/vmxnet3/vmxnet3_int.h | 8 +++++++- 6 files changed, 12 insertions(+), 6 deletions(-)
drivers/net/vmxnet3/Makefile +1 -1
@@ -2,7 +2,7 @@ # # Linux driver for VMware's vmxnet3 ethernet NIC. # -# Copyright (C) 2007-2020, VMware, Inc. All Rights Reserved. +# Copyright (C) 2007-2021, VMware, Inc. All Rights Reserved. # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the
drivers/net/vmxnet3/upt1_defs.h +1 -1
@@ -1,7 +1,7 @@/* * Linux driver for VMware's vmxnet3 ethernet NIC. * -* Copyright (C) 2008-2020, VMware, Inc. All Rights Reserved. +* Copyright (C) 2008-2021, VMware, Inc. All Rights Reserved. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the
drivers/net/vmxnet3/vmxnet3_defs.h +1 -1
@@ -1,7 +1,7 @@/* * Linux driver for VMware's vmxnet3 ethernet NIC. * -* Copyright (C) 2008-2020, VMware, Inc. All Rights Reserved. +* Copyright (C) 2008-2021, VMware, Inc. All Rights Reserved. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the
drivers/net/vmxnet3/vmxnet3_drv.c +1 -1
@@ -1,7 +1,7 @@/* * Linux driver for VMware's vmxnet3 ethernet NIC. * -* Copyright (C) 2008-2020, VMware, Inc. All Rights Reserved. +* Copyright (C) 2008-2021, VMware, Inc. All Rights Reserved. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the
drivers/net/vmxnet3/vmxnet3_ethtool.c +1 -1
@@ -1,7 +1,7 @@/* * Linux driver for VMware's vmxnet3 ethernet NIC. * -* Copyright (C) 2008-2020, VMware, Inc. All Rights Reserved. +* Copyright (C) 2008-2021, VMware, Inc. All Rights Reserved. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the
drivers/net/vmxnet3/vmxnet3_int.h +7 -1
@@ -1,7 +1,7 @@/* * Linux driver for VMware's vmxnet3 ethernet NIC. * -* Copyright (C) 2008-2020, VMware, Inc. All Rights Reserved. +* Copyright (C) 2008-2021, VMware, Inc. All Rights Reserved. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the
@@ -81,6 +81,8 @@#define VMXNET3_RSS #endif +#define VMXNET3_REV_6 5/* Vmxnet3 Rev. 6 */ +#define VMXNET3_REV_5 4/* Vmxnet3 Rev. 5 */ #define VMXNET3_REV_4 3/* Vmxnet3 Rev. 4 */ #define VMXNET3_REV_3 2/* Vmxnet3 Rev. 3 */ #define VMXNET3_REV_2 1/* Vmxnet3 Rev. 2 */
@@ -421,6 +423,10 @@ struct vmxnet3_adapter { (adapter->version >= VMXNET3_REV_3 + 1)#define VMXNET3_VERSION_GE_4 (adapter) \ (adapter->version >= VMXNET3_REV_4 + 1) +#define VMXNET3_VERSION_GE_5 (adapter) \ + (adapter->version >= VMXNET3_REV_5 + 1) +#define VMXNET3_VERSION_GE_6 (adapter) \ + (adapter->version >= VMXNET3_REV_6 + 1)/* must be a multiple of VMXNET3_RING_SIZE_ALIGN */ #define VMXNET3_DEF_TX_RING_SIZE 512
Signature · 1 line
2.11.0
PATCH 2/7 vmxnet3: add support for 32 Tx/Rx queues
In reply to Ronak DoshiCurrently, vmxnet3 supports maximum of 8 Tx/Rx queues. With increase in number of vcpus on a VM, to achieve better performance and utilize idle vcpus, we need to increase the max number of queues supported.
This patch enhances vmxnet3 to support maximum of 32 Tx/Rx queues. Increasing the Rx queues also increases the probability of distrubuting the traffic from different flows to different queues with RSS.
--- drivers/net/vmxnet3/vmxnet3_defs.h | 31 ++++++- drivers/net/vmxnet3/vmxnet3_drv.c | 170 +++++++++++++++++++++++++------------ drivers/net/vmxnet3/vmxnet3_int.h | 10 ++- 3 files changed, 151 insertions(+), 60 deletions(-)
drivers/net/vmxnet3/vmxnet3_defs.h +30 -1
@@ -98,6 +98,9 @@ enum {VMXNET3_CMD_GET_TXDATA_DESC_SIZE ,VMXNET3_CMD_GET_COALESCE ,VMXNET3_CMD_GET_RSS_FIELDS , +VMXNET3_CMD_GET_RESERVED2 , +VMXNET3_CMD_GET_RESERVED3 , + VMXNET3_CMD_GET_MAX_QUEUES_CONF, };/*
@@ -533,6 +536,13 @@ enum vmxnet3_intr_type {/* addition 1 for events */ #define VMXNET3_MAX_INTRS 25 +/* Version 6 and later will use below macros */ +#define VMXNET3_EXT_MAX_TX_QUEUES 32 +#define VMXNET3_EXT_MAX_RX_QUEUES 32 +/* addition 1 for events */ +#define VMXNET3_EXT_MAX_INTRS 65 +#define VMXNET3_FIRST_SET_INTRS 64 +/* value of intrCtrl */ #define VMXNET3_IC_DISABLE_ALL 0x1/* bit 0 */
@@ -547,6 +557,19 @@ struct Vmxnet3_IntrConf {__le32 reserved[2 ]; }; +struct Vmxnet3_IntrConfExt { +u8 autoMask; +u8 numIntrs;/* # of interrupts */ +u8 eventIntrIdx; +u8 reserved; +__le32 intrCtrl; +__le32 reserved1; +u8 modLevels[VMXNET3_EXT_MAX_INTRS ];/* moderation level for +* each intr +*/ +u8 reserved2[3 ]; +}; +/* one bit per VLAN ID, the size is in the units of u32 */ #define VMXNET3_VFT_SIZE (4096 / (sizeof(u32) * 8))
@@ -719,11 +742,16 @@ struct Vmxnet3_DSDevRead {struct Vmxnet3_VariableLenConfDesc pluginConfDesc; }; +struct Vmxnet3_DSDevReadExt { +/* read-only region for device, read by dev in response to a SET cmd */ +struct Vmxnet3_IntrConfExt intrConfExt; +}; +/* All structures in DriverShared are padded to multiples of 8 bytes */ struct Vmxnet3_DriverShared {__le32 magic;/* make devRead start at 64bit boundaries */ -__le32 pad; +__le32 size;/* size of DriverShared */ struct Vmxnet3_DSDevRead devRead;__le32 ecr;__le32 reserved;
@@ -734,6 +762,7 @@ struct Vmxnet3_DriverShared {* command */ } cu; +struct Vmxnet3_DSDevReadExt devReadExt; };
drivers/net/vmxnet3/vmxnet3_drv.c +114 -56
243 lines not on this page · raw message
drivers/net/vmxnet3/vmxnet3_int.h +7 -3
@@ -303,15 +303,18 @@ struct vmxnet3_rx_queue {struct vmxnet3_rq_driver_stats stats; } __attribute__((__aligned__(SMP_CACHE_BYTES))); -#define VMXNET3_DEVICE_MAX_TX_QUEUES 8 -#define VMXNET3_DEVICE_MAX_RX_QUEUES 8/* Keep this value as a power of 2 */ +#define VMXNET3_DEVICE_MAX_TX_QUEUES 32 +#define VMXNET3_DEVICE_MAX_RX_QUEUES 32/* Keep this value as a power of 2 */ + +#define VMXNET3_DEVICE_DEFAULT_TX_QUEUES 8 +#define VMXNET3_DEVICE_DEFAULT_RX_QUEUES 8/* Keep this value as a power of 2 */ /* Should be less than UPT1_RSS_MAX_IND_TABLE_SIZE */ #define VMXNET3_RSS_IND_TABLE_SIZE (VMXNET3_DEVICE_MAX_RX_QUEUES * 4)#define VMXNET3_LINUX_MAX_MSIX_VECT (VMXNET3_DEVICE_MAX_TX_QUEUES + \ VMXNET3_DEVICE_MAX_RX_QUEUES + 1) -#define VMXNET3_LINUX_MIN_MSIX_VECT 2/* 1 for tx-rx pair and 1 for event */ +#define VMXNET3_LINUX_MIN_MSIX_VECT 3/* 1 for tx, 1 for rx pair and 1 for event */ struct vmxnet3_intr {
@@ -398,6 +401,7 @@ struct vmxnet3_adapter {dma_addr_t adapter_pa;dma_addr_t pm_conf_pa;dma_addr_t rss_conf_pa; +bool queuesExtEnabled; };#define VMXNET3_WRITE_BAR0_REG (adapter, reg, val) \
Signature · 1 line
2.11.0
PATCH 3/7 vmxnet3: remove power of 2 limitation on the queues
In reply to Ronak DoshiWith version 6, vmxnet3 relaxes the restriction on queues to be power of two. This is helpful in cases (Edge VM) where vcpus are less than 8 and device requires more than 4 queues.
--- drivers/net/vmxnet3/vmxnet3_drv.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +5 -3
@@ -3399,7 +3399,6 @@ vmxnet3_probe_device(struct pci_dev *pdev, else#endif num_rx_queues =1 ; - num_rx_queues =rounddown_pow_of_two (num_rx_queues);if (enable_mq) num_tx_queues = min(VMXNET3_DEVICE_MAX_TX_QUEUES,
@@ -3407,7 +3406,6 @@ vmxnet3_probe_device(struct pci_dev *pdev,else num_tx_queues =1 ; - num_tx_queues =rounddown_pow_of_two (num_tx_queues); netdev =alloc_etherdev_mq (sizeof (struct vmxnet3_adapter ),max (num_tx_queues, num_rx_queues));if (!netdev)
@@ -3525,6 +3523,8 @@ vmxnet3_probe_device(struct pci_dev *pdev, } }else { adapter->queuesExtEnabled = false; + num_rx_queues =rounddown_pow_of_two (num_rx_queues); + num_tx_queues =rounddown_pow_of_two (num_tx_queues); adapter->num_rx_queues =min (num_rx_queues,VMXNET3_DEVICE_DEFAULT_RX_QUEUES ); adapter->num_tx_queues = min(num_tx_queues,
@@ -3705,7 +3705,9 @@ vmxnet3_remove_device(struct pci_dev *pdev) else#endif num_rx_queues =1 ; - num_rx_queues =rounddown_pow_of_two (num_rx_queues); +if (!VMXNET3_VERSION_GE_6 (adapter)) { + num_rx_queues =rounddown_pow_of_two (num_rx_queues); + }if (VMXNET3_VERSION_GE_6 (adapter)) {spin_lock_irqsave (&adapter->cmd_lock, flags); VMXNET3_WRITE_BAR1_REG(adapter, VMXNET3_REG_CMD,
Signature · 1 line
2.11.0
PATCH 4/7 vmxnet3: add support for ESP IPv6 RSS
In reply to Ronak DoshiVmxnet3 version 4 added support for ESP RSS. However, only IPv4 was supported. With vmxnet3 version 6, this patch enables RSS for ESP IPv6 packets as well.
--- drivers/net/vmxnet3/vmxnet3_ethtool.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/net/vmxnet3/vmxnet3_ethtool.c +20 -0
@@ -769,6 +769,10 @@ vmxnet3_get_rss_hash_opts(struct vmxnet3_adapter *adapter,case AH_ESP_V6_FLOW :case AH_V6_FLOW :case ESP_V6_FLOW : +if (VMXNET3_VERSION_GE_6 (adapter) && + (rss_fields &VMXNET3_RSS_FIELDS_ESPIP6 )) + info->data |=RXH_L4_B_0_1 |RXH_L4_B_2_3 ; + fallthrough;case SCTP_V6_FLOW :case IPV6_FLOW : info->data |=RXH_IP_SRC |RXH_IP_DST ;
@@ -853,6 +857,22 @@ vmxnet3_set_rss_hash_opt(struct net_device *netdev,case ESP_V6_FLOW :case AH_V6_FLOW :case AH_ESP_V6_FLOW : +if (!VMXNET3_VERSION_GE_6 (adapter)) +return -EOPNOTSUPP ; +if (!(nfc->data &RXH_IP_SRC ) || + !(nfc->data &RXH_IP_DST )) +return -EINVAL ; +switch (nfc->data & (RXH_L4_B_0_1 |RXH_L4_B_2_3 )) { +case 0 : + rss_fields &= ~VMXNET3_RSS_FIELDS_ESPIP6 ; +break ; +case (RXH_L4_B_0_1 |RXH_L4_B_2_3 ): + rss_fields |=VMXNET3_RSS_FIELDS_ESPIP6 ; +break ; +default : +return -EINVAL ; + } +break ;case SCTP_V4_FLOW :case SCTP_V6_FLOW : if (!(nfc->data & RXH_IP_SRC) ||
Signature · 1 line
2.11.0
PATCH 5/7 vmxnet3: set correct hash type based on rss information
In reply to Ronak DoshiAs vmxnet3 supports IP/TCP/UDP RSS, this patch sets appropriate hash type based on the type of RSS performed.
--- drivers/net/vmxnet3/vmxnet3_defs.h | 16 +++++++++------- drivers/net/vmxnet3/vmxnet3_drv.c | 22 ++++++++++++++++++++-- 2 files changed, 29 insertions(+), 9 deletions(-)
drivers/net/vmxnet3/vmxnet3_defs.h +9 -7
@@ -344,13 +344,15 @@ struct Vmxnet3_RxCompDescExt {#define VMXNET3_TXD_EOP_SIZE 1/* value of RxCompDesc.rssType */ -enum { -VMXNET3_RCD_RSS_TYPE_NONE =0 , -VMXNET3_RCD_RSS_TYPE_IPV4 =1 , -VMXNET3_RCD_RSS_TYPE_TCPIPV4 =2 , -VMXNET3_RCD_RSS_TYPE_IPV6 =3 , -VMXNET3_RCD_RSS_TYPE_TCPIPV6 =4 , -}; +#define VMXNET3_RCD_RSS_TYPE_NONE 0 +#define VMXNET3_RCD_RSS_TYPE_IPV4 1 +#define VMXNET3_RCD_RSS_TYPE_TCPIPV4 2 +#define VMXNET3_RCD_RSS_TYPE_IPV6 3 +#define VMXNET3_RCD_RSS_TYPE_TCPIPV6 4 +#define VMXNET3_RCD_RSS_TYPE_UDPIPV4 5 +#define VMXNET3_RCD_RSS_TYPE_UDPIPV6 6 +#define VMXNET3_RCD_RSS_TYPE_ESPIPV4 7 +#define VMXNET3_RCD_RSS_TYPE_ESPIPV6 8/* a union for accessing all cmd/completion descriptors */
drivers/net/vmxnet3/vmxnet3_drv.c +20 -2
@@ -1478,10 +1478,28 @@ vmxnet3_rq_rx_complete(struct vmxnet3_rx_queue *rq,#ifdef VMXNET3_RSS if (rcd->rssType !=VMXNET3_RCD_RSS_TYPE_NONE && - (adapter->netdev->features &NETIF_F_RXHASH )) + (adapter->netdev->features &NETIF_F_RXHASH )) { +enum pkt_hash_types hash_type; + +switch (rcd->rssType) { +case VMXNET3_RCD_RSS_TYPE_IPV4 : +case VMXNET3_RCD_RSS_TYPE_IPV6 : + hash_type =PKT_HASH_TYPE_L3 ; +break ; +case VMXNET3_RCD_RSS_TYPE_TCPIPV4 : +case VMXNET3_RCD_RSS_TYPE_TCPIPV6 : +case VMXNET3_RCD_RSS_TYPE_UDPIPV4 : +case VMXNET3_RCD_RSS_TYPE_UDPIPV6 : + hash_type =PKT_HASH_TYPE_L4 ; +break ; +default : + hash_type =PKT_HASH_TYPE_L3 ; +break ; + }skb_set_hash (ctx->skb,le32_to_cpu (rcd->rssHash), -PKT_HASH_TYPE_L3 ); + hash_type); + }#endif skb_put (ctx->skb, rcd->len);
Signature · 1 line
2.11.0
PATCH 6/7 vmxnet3: increase maximum configurable mtu to 9190
In reply to Ronak DoshiThis patch increases the maximum configurable mtu to 9190 to accommodate jumbo packets of overlay traffic.
--- drivers/net/vmxnet3/vmxnet3_defs.h | 1 + drivers/net/vmxnet3/vmxnet3_drv.c | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/net/vmxnet3/vmxnet3_defs.h +1 -0
@@ -795,6 +795,7 @@ struct Vmxnet3_DriverShared { ((vfTable[vid >> 5] & (1 << (vid & 31))) != 0)#define VMXNET3_MAX_MTU 9000 +#define VMXNET3_V6_MAX_MTU 9190#define VMXNET3_MIN_MTU 60#define VMXNET3_LINK_UP (10000 << 16 | 1)/* 10 Gbps, up */
drivers/net/vmxnet3/vmxnet3_drv.c +5 -2
@@ -3641,9 +3641,12 @@ vmxnet3_probe_device(struct pci_dev *pdev,vmxnet3_set_ethtool_ops (netdev); netdev->watchdog_timeo =5 *HZ ; -/* MTU range: 60 - 9000 */ +/* MTU range: 60 - 9190 */ netdev->min_mtu =VMXNET3_MIN_MTU ; - netdev->max_mtu =VMXNET3_MAX_MTU ; +if (VMXNET3_VERSION_GE_6 (adapter)) + netdev->max_mtu =VMXNET3_V6_MAX_MTU ; +else + netdev->max_mtu =VMXNET3_MAX_MTU ;INIT_WORK (&adapter->work, vmxnet3_reset_work);set_bit (VMXNET3_STATE_BIT_QUIESCED , &adapter->state);
Signature · 1 line
2.11.0
PATCH 7/7 vmxnet3: update to version 6
In reply to Ronak DoshiWith all vmxnet3 version 6 changes incorporated in the vmxnet3 driver, the driver can configure emulation to run at vmxnet3 version 6, provided the emulation advertises support for version 6.
--- drivers/net/vmxnet3/vmxnet3_drv.c | 12 +++++++++++- drivers/net/vmxnet3/vmxnet3_int.h | 4 ++-- 2 files changed, 13 insertions(+), 3 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +11 -1
@@ -3480,7 +3480,17 @@ vmxnet3_probe_device(struct pci_dev *pdev, goto err_alloc_pci; ver =VMXNET3_READ_BAR1_REG (adapter,VMXNET3_REG_VRRS ); -if (ver & (1 <<VMXNET3_REV_4 )) { +if (ver & (1 <<VMXNET3_REV_6 )) { +VMXNET3_WRITE_BAR1_REG (adapter, +VMXNET3_REG_VRRS , +1 <<VMXNET3_REV_6 ); + adapter->version =VMXNET3_REV_6 +1 ; + }else if (ver & (1 <<VMXNET3_REV_5 )) { +VMXNET3_WRITE_BAR1_REG (adapter, +VMXNET3_REG_VRRS , +1 <<VMXNET3_REV_5 ); + adapter->version =VMXNET3_REV_5 +1 ; + }else if (ver & (1 <<VMXNET3_REV_4 )) {VMXNET3_WRITE_BAR1_REG (adapter,VMXNET3_REG_VRRS ,1 <<VMXNET3_REV_4 );
drivers/net/vmxnet3/vmxnet3_int.h +2 -2
@@ -69,12 +69,12 @@/* * Version numbers */ -#define VMXNET3_DRIVER_VERSION_STRING "1.5.0.0-k" +#define VMXNET3_DRIVER_VERSION_STRING "1.6.0.0-k"/* Each byte of this 32-bit integer encodes a version number in * VMXNET3_DRIVER_VERSION_STRING. */ -#define VMXNET3_DRIVER_VERSION_NUM 0x01050000 +#define VMXNET3_DRIVER_VERSION_NUM 0x01060000#if defined(CONFIG_PCI_MSI )/* RSS only makes sense if MSI-X is supported. */
Signature · 1 line
2.11.0