net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)
Dexuan Cui
maintainerPATCH v5 1/1 net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)
Add a VF driver for Microsoft Azure Network Adapter (MANA) that will be available in the future.
--- MAINTAINERS | 4 +- drivers/net/ethernet/Kconfig | 1 + drivers/net/ethernet/Makefile | 1 + drivers/net/ethernet/microsoft/Kconfig | 29 + drivers/net/ethernet/microsoft/Makefile | 5 + drivers/net/ethernet/microsoft/mana/Makefile | 6 + drivers/net/ethernet/microsoft/mana/gdma.h | 728 +++++++ .../net/ethernet/microsoft/mana/gdma_main.c | 1525 +++++++++++++ .../net/ethernet/microsoft/mana/hw_channel.c | 849 ++++++++ .../net/ethernet/microsoft/mana/hw_channel.h | 190 ++ drivers/net/ethernet/microsoft/mana/mana.h | 549 +++++ drivers/net/ethernet/microsoft/mana/mana_en.c | 1919 +++++++++++++++++ .../ethernet/microsoft/mana/mana_ethtool.c | 241 +++ .../net/ethernet/microsoft/mana/shm_channel.c | 298 +++ .../net/ethernet/microsoft/mana/shm_channel.h | 21 + drivers/net/hyperv/netvsc_drv.c | 14 +- 16 files changed, 6377 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/microsoft/Kconfig create mode 100644 drivers/net/ethernet/microsoft/Makefile create mode 100644 drivers/net/ethernet/microsoft/mana/Makefile create mode 100644 drivers/net/ethernet/microsoft/mana/gdma.h create mode 100644 drivers/net/ethernet/microsoft/mana/gdma_main.c create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.c create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.h create mode 100644 drivers/net/ethernet/microsoft/mana/mana.h create mode 100644 drivers/net/ethernet/microsoft/mana/mana_en.c create mode 100644 drivers/net/ethernet/microsoft/mana/mana_ethtool.c create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.c create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.h
Changes in v2:
Removed the module_param(num_queues,...). [Andrew Lunn]
Changed pr_err() to netdev_err() and dev_err(). [Andrew Lunn]
Used reverse X-mas tree style in all the functions. [Andrew Lunn]
Changed "= { 0 };" to "= {};" for struct variables. [Leon Romanovsky]
Addressed 3 build failures on i386, arc and powerpc by making the the driver dependent on X86_64 in Kconfig: so far, the driver is only validated on X86_64 (in the future, we may enable the driver for ARM64).
Also made some cosmetic changes.
Rebased the patch to the latest net-next.
BTW, the support of XDP and BQL haven't been implemented. They are on our TO-DO list.
Changes in v3:
Changed the Kconfig file:
Microsoft Azure Network Device ==> Microsoft Network Devices Drop the default m
validated ==> supported
Fixed some warnings reported by "kernel test robot" (there are
still some warnings against another driver pci-hyperv.c, which I'll fix in a separate patch for the PCI tree).
Changes in v4:
Removed all the "__packed" usages since all the structs are actually naturally aligned. We verified there is no hole n the middle of the structs or at the end of the structs.
Prefixed all the functions with "mana_".
Integrated Shachar Raindel's multi-vPort patches (i.e. create multiple net devices on the single VF PCIe device, if the PF driver reports the support). Added Shachar's Co-developed-by and Signed-off-by.
Also made some cosmetic changes.
Rebased the patch to the latest net-next.
Changes in v5:
Made a small change to drivers/net/hyperv/netvsc_drv.c's get_netvsc_byslot(): now with the multi-vPorts-per-PCI-dev, the netvsc driver needs to check both the serial number and the MAC address to find the matching vPort net device. Note: this change doesn't break any existing VF drivers.
Prefixed all the functions with "mana_". (Previously some were missed). Now there is no "ana" or "ANA" in the code.
Changed the "drv version" to "proto version". We don't have driver version. The correct name in mana_query_device_cfg() should be "proto version".
Fixed the defines and the struct field names accordingly.
Used ethtool_rxfh_indir_default() rather than direct "i % apc->num_queues;"
Handled the error in case netif_set_real_num_tx_queues() fails.
Used register_netdev() since we don't really need to explicitly get and release the RTNL lock.
Changed some WARN_ON()'s to WARN_ON_ONCE()'s. Removed a few unnecessary WARN_ON()'s.
Used "WARN_ON(xxx);" rather than "if (xxx) WARN_ON(1);"
Used "return foo();" rather than "err = foo(); return err;" in several places.
Removed unnecessary checking in mana_set_channels().
Also made some small cosmetic changes.
Rebased the patch to the latest net-next.
MAINTAINERS +3 -1
@@ -8260,11 +8260,12 @@ S: Maintained T: git git://linuxtv.org/media_tree.git F: drivers/media/i2c/hi556.c -Hyper-V CORE AND DRIVERS +Hyper-V/Azure CORE AND DRIVERS M: "K. Y. Srinivasan" <[email protected]> M: Haiyang Zhang <[email protected]> M: Stephen Hemminger <[email protected]> M: Wei Liu <[email protected]> +M: Dexuan Cui <[email protected]> L: [email protected] S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -8281,6 +8282,7 @@ F: drivers/hid/hid-hyperv.c F: drivers/hv/ F: drivers/input/serio/hyperv-keyboard.c F: drivers/iommu/hyperv-iommu.c +F: drivers/net/ethernet/microsoft/ F: drivers/net/hyperv/ F: drivers/pci/controller/pci-hyperv-intf.c F: drivers/pci/controller/pci-hyperv.c
drivers/net/ethernet/Kconfig +1 -0
@@ -82,6 +82,7 @@ source "drivers/net/ethernet/huawei/Kconfig" source "drivers/net/ethernet/i825xx/Kconfig" source "drivers/net/ethernet/ibm/Kconfig" source "drivers/net/ethernet/intel/Kconfig" +source "drivers/net/ethernet/microsoft/Kconfig" source "drivers/net/ethernet/xscale/Kconfig" config JME
drivers/net/ethernet/Makefile +1 -0
@@ -45,6 +45,7 @@ obj-$(CONFIG_NET_VENDOR_HUAWEI) += huawei/ obj-$(CONFIG_NET_VENDOR_IBM) += ibm/ obj-$(CONFIG_NET_VENDOR_INTEL) += intel/ obj-$(CONFIG_NET_VENDOR_I825XX) += i825xx/ +obj-$(CONFIG_NET_VENDOR_MICROSOFT) += microsoft/ obj-$(CONFIG_NET_VENDOR_XSCALE) += xscale/ obj-$(CONFIG_JME) += jme.o obj-$(CONFIG_KORINA) += korina.o
drivers/net/ethernet/microsoft/Kconfig new file +29 -0
@@ -0,0 +1,29 @@ +# +# Microsoft Azure network device configuration +# + +config NET_VENDOR_MICROSOFT + bool "Microsoft Network Devices" + default y + help + If you have a network (Ethernet) device belonging to this class, say Y. + + Note that the answer to this question doesn't directly affect the + kernel: saying N will just cause the configurator to skip the + question about Microsoft network devices. If you say Y, you will be + asked for your specific device in the following question. + +if NET_VENDOR_MICROSOFT + +config MICROSOFT_MANA + tristate "Microsoft Azure Network Adapter (MANA) support" + depends on PCI_MSI && X86_64 + select PCI_HYPERV + help + This driver supports Microsoft Azure Network Adapter (MANA). + So far, the driver is only supported on X86_64. + + To compile this driver as a module, choose M here. + The module will be called mana. + +endif #NET_VENDOR_MICROSOFT
drivers/net/ethernet/microsoft/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# +# Makefile for the Microsoft Azure network device driver. +# + +obj-$(CONFIG_MICROSOFT_MANA) += mana/
drivers/net/ethernet/microsoft/mana/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +# +# Makefile for the Microsoft Azure Network Adapter driver + +obj-$(CONFIG_MICROSOFT_MANA) += mana.o +mana-objs := gdma_main.o shm_channel.o hw_channel.o mana_en.o mana_ethtool.o
drivers/net/ethernet/microsoft/mana/gdma.h new file +728 -0
728 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/gdma_main.c new file +1525 -0
1,525 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/hw_channel.c new file +849 -0
673 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/hw_channel.h new file +190 -0
190 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/mana.h new file +549 -0
549 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/mana_en.c new file +1919 -0
1,919 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/mana_ethtool.c new file +241 -0
241 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/shm_channel.c new file +298 -0
298 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/shm_channel.h new file +21 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.20.1
Part of the diff is not shown here. View the raw message.
Jakub Kicinski
maintainerdrivers/net/ethernet/microsoft/mana/mana_en.c line 1742
Quoted 8 lines · On Mon, 12 Apr 2021 19:35:09 -0700 Dexuan Cui wrote:
drivers/net/ethernet/microsoft/mana/mana_en.c new file +8
-0+ apc->port_st_save = apc->port_is_up; + apc->port_is_up = false; + apc->start_remove = true; + +/* Ensure port state updated before txq state */ +smp_wmb (); + +netif_tx_disable (ndev);
In your napi poll method there is no barrier between port_is_up check and netif_tx_queue_stopped().
Quoted 17 lines
drivers/net/ethernet/microsoft/mana/mana_en.c new file +16
-0+netif_carrier_off (ndev); + +/* No packet can be transmitted now since apc->port_is_up is false. +* There is still a tiny chance that mana_poll_tx_cq() can re-enable +* a txq because it may not timely see apc->port_is_up being cleared +* to false, but it doesn't matter since mana_start_xmit() drops any +* new packets due to apc->port_is_up being false. +* +* Drain all the in-flight TX packets +*/ +for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + +while (atomic_read (&txq->pending_sends) >0 ) +usleep_range (1000 ,2000 ); + }
drivers/net/ethernet/microsoft/mana/mana_en.c
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c new file +10
-0+/* All cleanup actions should stay after rtnl_lock(), otherwise +* other functions may access partially cleaned up data. +*/ +rtnl_lock (); + +mana_detach (ndev); + +unregister_netdevice (ndev); + +rtnl_unlock ();
I find the resource management somewhat strange. Why is mana_attach() and mana_detach() called at probe/remove time, and not when the interface is brought up? Presumably when the user ifdowns the interface there is no point holding the resources? Your open/close methods are rather empty.
drivers/net/ethernet/microsoft/mana/shm_channel.c line 167
Quoted 11 lines
drivers/net/ethernet/microsoft/mana/shm_channel.c new file +11
-0+if ((eq_addr &PAGE_MASK ) != eq_addr) +return -EINVAL ; + +if ((cq_addr &PAGE_MASK ) != cq_addr) +return -EINVAL ; + +if ((rq_addr &PAGE_MASK ) != rq_addr) +return -EINVAL ; + +if ((sq_addr &PAGE_MASK ) != sq_addr) +return -EINVAL ;
PAGE_ALIGNED()
Dexuan Cui
maintainerQuoted 4 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, April 13, 2021 12:03 PMOn Mon, 12 Apr 2021 19:35:09 -0700 Dexuan Cui wrote:
Quoted 11 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+ apc->port_st_save = apc->port_is_up; + apc->port_is_up = false; + apc->start_remove = true; + + /* Ensure port state updated before txq state */ + smp_wmb(); + + netif_tx_disable(ndev);In your napi poll method there is no barrier between
port_is_upcheck andnetif_tx_queue_stopped().
Thanks for spotting this! We'll make the below change:
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -938,16 +938,19 @@ static void mana_poll_tx_cq(struct mana_cq *cq) avail_space =mana_gd_wq_avail_space (gdma_wq);/* Ensure tail updated before checking q stop */ smp_mb (); net_txq = txq->net_txq; txq_stopped =netif_tx_queue_stopped (net_txq); +/* Ensure checking txq_stopped before apc->port_is_up. */ +smp_rmb (); +if (txq_stopped && apc->port_is_up && avail_space >=MAX_TX_WQE_SIZE ) {netif_tx_wake_queue (net_txq); apc->eth_stats.wake_queue++; }if (atomic_sub_return (pkt_transmitted, &txq->pending_sends) <0 )WARN_ON_ONCE (1 ); }
Quoted 17 lines
Quoted 16 lines
(quoted diff) +16
-0+ netif_carrier_off(ndev); + + /* No packet can be transmitted now since apc->port_is_up is false. + * There is still a tiny chance that mana_poll_tx_cq() can re-enable + * a txq because it may not timely see apc->port_is_up being cleared + * to false, but it doesn't matter since mana_start_xmit() drops any + * new packets due to apc->port_is_up being false. + * + * Drain all the in-flight TX packets + */ + for (i = 0; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + + while (atomic_read(&txq->pending_sends) > 0) + usleep_range(1000, 2000); + }
Quoted 16 lines
Review
Quoted 10 lines
(quoted diff) +10
-0+ /* All cleanup actions should stay after rtnl_lock(), otherwise + * other functions may access partially cleaned up data. + */ + rtnl_lock(); + + mana_detach(ndev); + + unregister_netdevice(ndev); + + rtnl_unlock();I find the resource management somewhat strange. Why is
mana_attach()andmana_detach()called at probe/remove time, and not when the interface is brought up? Presumably when the user ifdowns the interface there is no point holding the resources? Your open/close methods are rather empty.
Thanks for the suggestion! Will move the functions to open/close().
Quoted 11 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+ if ((eq_addr & PAGE_MASK) != eq_addr) + return -EINVAL; + + if ((cq_addr & PAGE_MASK) != cq_addr) + return -EINVAL; + + if ((rq_addr & PAGE_MASK) != rq_addr) + return -EINVAL; + + if ((sq_addr & PAGE_MASK) != sq_addr) + return -EINVAL;
Will change to PAGE_ALIGNED().
Thanks,
Dexuan