Skip to contentSkip to conversation

net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

PATCH Superseded Dexuan Cui 3 participants 6 messages Apr 16, 2021

Subsystems (4)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Dexuan Cui

author maintainer
#0 lore raw

PATCH v7 1/1 net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

Add a VF driver for Microsoft Azure Network Adapter (MANA) that will be available in the future.

---
 MAINTAINERS                                   |    4 +-
 drivers/net/ethernet/Kconfig                  |    1 +
 drivers/net/ethernet/Makefile                 |    1 +
 drivers/net/ethernet/microsoft/Kconfig        |   29 +
 drivers/net/ethernet/microsoft/Makefile       |    5 +
 drivers/net/ethernet/microsoft/mana/Makefile  |    6 +
 drivers/net/ethernet/microsoft/mana/gdma.h    |  673 ++++++
 .../net/ethernet/microsoft/mana/gdma_main.c   | 1415 ++++++++++++
 .../net/ethernet/microsoft/mana/hw_channel.c  |  843 ++++++++
 .../net/ethernet/microsoft/mana/hw_channel.h  |  190 ++
 drivers/net/ethernet/microsoft/mana/mana.h    |  533 +++++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 1895 +++++++++++++++++
 .../ethernet/microsoft/mana/mana_ethtool.c    |  250 +++
 .../net/ethernet/microsoft/mana/shm_channel.c |  291 +++
 .../net/ethernet/microsoft/mana/shm_channel.h |   21 +
 drivers/net/hyperv/netvsc_drv.c               |   14 +-
 16 files changed, 6168 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/microsoft/Kconfig
 create mode 100644 drivers/net/ethernet/microsoft/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma_main.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_en.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_ethtool.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.h

Changes in v2:

Removed the module_param(num_queues,...). [Andrew Lunn]
Changed pr_err() to netdev_err() and dev_err(). [Andrew Lunn]
Used reverse X-mas tree style in all the functions. [Andrew Lunn]

Changed "= { 0 };" to "= {};" for struct variables. [Leon Romanovsky]

Addressed 3 build failures on i386, arc and powerpc by making the the driver dependent on X86_64 in Kconfig: so far, the driver is only validated on X86_64 (in the future, we may enable the driver for ARM64).

Also made some cosmetic changes.

Rebased the patch to the latest net-next.

BTW, the support of XDP and BQL haven't been implemented. They are on our TO-DO list.

Changes in v3:
Changed the Kconfig file:
Microsoft Azure Network Device ==> Microsoft Network Devices Drop the default m
validated ==> supported

Fixed some warnings reported by "kernel test robot" (there are
still some warnings against another driver pci-hyperv.c, which I'll fix in a separate patch for the PCI tree).

Changes in v4:
Removed all the "__packed" usages since all the structs are actually naturally aligned. We verified there is no hole n the middle of the structs or at the end of the structs.

Prefixed all the functions with "mana_".

Integrated Shachar Raindel's multi-vPort patches (i.e. create multiple net devices on the single VF PCIe device, if the PF driver reports the support). Added Shachar's Co-developed-by and Signed-off-by.

Also made some cosmetic changes.
Rebased the patch to the latest net-next.

Changes in v5:
Made a small change to drivers/net/hyperv/netvsc_drv.c's get_netvsc_byslot(): now with the multi-vPorts-per-PCI-dev, the netvsc driver needs to check both the serial number and the MAC address to find the matching vPort net device. Note: this change doesn't break any existing VF drivers.

Prefixed all the functions with "mana_". (Previously some were missed). Now there is no "ana" or "ANA" in the code.

Changed the "drv version" to "proto version". We don't have driver version. The correct name in mana_query_device_cfg() should be "proto version".
Fixed the defines and the struct field names accordingly.

Used ethtool_rxfh_indir_default() rather than direct "i % apc->num_queues;"

Handled the error in case netif_set_real_num_tx_queues() fails.

Used register_netdev() since we don't really need to explicitly get and release the RTNL lock.

Changed some WARN_ON()'s to WARN_ON_ONCE()'s. Removed a few unnecessary WARN_ON()'s.
Used "WARN_ON(xxx);" rather than "if (xxx) WARN_ON(1);"

Used "return foo();" rather than "err = foo(); return err;" in several places.

Removed unnecessary checking in mana_set_channels().

Also made some small cosmetic changes.
Rebased the patch to the latest net-next.

Changes in v6:
Used mana_do_attach()/mana_detach() in mana_open()/mana_close(). This way the related resources (e.g. SQ, RQ, etc.) are not allocated until the net interface goes UP.

Fixed IRQ allocation for the multi-vPorts case. Currently each vPort has its own event queues (EQs), and in future we'll share EQs among all the vPorts (a vPort means a net_device).

Added smp_rmb() in mana_poll_tx_cq(): ensure checking txq_stopped before checking apc->port_is_up.

Used PAGE_ALIGNED(), PAGE_ALIGN(), PHYS_PFN() where appropriate.

Removed some code that isn't really used at present, e.g. the below flags are not really used at present:
GDMA_WR_SGL_DIRECT
GDMA_WR_CONSUME_CREDIT
GDMA_WR_FENCE
GDMA_WR_CHECK_SN
so removed the code using the flags, mainly in mana_gd_write_sgl() and mana_gd_write_client_oob().

Removed unused event types in mana_gd_process_eqe(), and also simplified the function a little bit.

Rebased the patch to the latest net-next.

Changes in v7:
Removed enum mana_attach_caller {MANA_OPEN, MANA_ATTACH,MANA_PROBE}. Split mana_do_attach() and mana_detech() into different functions:
mana_init_port(), mana_alloc_queues(), mana_dealloc_queues, so now mana_open(), mana_close(), mana_probe() and mana_probe_port(),etc. are more readable.

Fixed a race condition in mana_hwc_send_request(): moved mana_hwc_put_msg_index() from mana_hwc_rx_event_handler() to the end of mana_hwc_send_request(), otherwise the msg index is released too early and a new request's response may overwrite the previous req's response buffer which may not have been used yet.

Dropped the __GFP_ZERO flag when calling dma_alloc_coherent().

Removed some unnecessary empty lines.

Fixed the 'queue' memory leak issue in a few places if mana_gd_alloc_memory() fails.

Used ether_addr_copy() rather than memcpy().

Used netdev_rss_key_fill() rather than get_random_bytes().

Removed some dead code in mana_send_request().

Rebased the patch to the latest net-next.

MAINTAINERS +3 -1
index 795b9941c151..3b75bc7c51f8 100644
@@ -8260,11 +8260,12 @@ S:	Maintained
 T:	git git://linuxtv.org/media_tree.git
 F:	drivers/media/i2c/hi556.c
 
-Hyper-V CORE AND DRIVERS
+Hyper-V/Azure CORE AND DRIVERS
 M:	"K. Y. Srinivasan" <[email protected]>
 M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
+M:	Dexuan Cui <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -8281,6 +8282,7 @@ F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
 F:	drivers/iommu/hyperv-iommu.c
+F:	drivers/net/ethernet/microsoft/
 F:	drivers/net/hyperv/
 F:	drivers/pci/controller/pci-hyperv-intf.c
 F:	drivers/pci/controller/pci-hyperv.c
drivers/net/ethernet/Kconfig +1 -0
index 4b85f2b74872..d46460c5b44d 100644
@@ -82,6 +82,7 @@ source "drivers/net/ethernet/huawei/Kconfig"
 source "drivers/net/ethernet/i825xx/Kconfig"
 source "drivers/net/ethernet/ibm/Kconfig"
 source "drivers/net/ethernet/intel/Kconfig"
+source "drivers/net/ethernet/microsoft/Kconfig"
 source "drivers/net/ethernet/xscale/Kconfig"
 
 config JME
drivers/net/ethernet/Makefile +1 -0
index 9394493e8187..cb3f9084a21b 100644
@@ -45,6 +45,7 @@ obj-$(CONFIG_NET_VENDOR_HUAWEI) += huawei/
 obj-$(CONFIG_NET_VENDOR_IBM) += ibm/
 obj-$(CONFIG_NET_VENDOR_INTEL) += intel/
 obj-$(CONFIG_NET_VENDOR_I825XX) += i825xx/
+obj-$(CONFIG_NET_VENDOR_MICROSOFT) += microsoft/
 obj-$(CONFIG_NET_VENDOR_XSCALE) += xscale/
 obj-$(CONFIG_JME) += jme.o
 obj-$(CONFIG_KORINA) += korina.o
drivers/net/ethernet/microsoft/Kconfig new file +29 -0
new file mode 100644
index 000000000000..e1ac0a5d808d
@@ -0,0 +1,29 @@
+#
+# Microsoft Azure network device configuration
+#
+
+config NET_VENDOR_MICROSOFT
+	bool "Microsoft Network Devices"
+	default y
+	help
+	  If you have a network (Ethernet) device belonging to this class, say Y.
+
+	  Note that the answer to this question doesn't directly affect the
+	  kernel: saying N will just cause the configurator to skip the
+	  question about Microsoft network devices. If you say Y, you will be
+	  asked for your specific device in the following question.
+
+if NET_VENDOR_MICROSOFT
+
+config MICROSOFT_MANA
+	tristate "Microsoft Azure Network Adapter (MANA) support"
+	depends on PCI_MSI && X86_64
+	select PCI_HYPERV
+	help
+	  This driver supports Microsoft Azure Network Adapter (MANA).
+	  So far, the driver is only supported on X86_64.
+
+	  To compile this driver as a module, choose M here.
+	  The module will be called mana.
+
+endif #NET_VENDOR_MICROSOFT
drivers/net/ethernet/microsoft/Makefile new file +5 -0
new file mode 100644
index 000000000000..d2ddc218135f
@@ -0,0 +1,5 @@
+#
+# Makefile for the Microsoft Azure network device driver.
+#
+
+obj-$(CONFIG_MICROSOFT_MANA) += mana/
drivers/net/ethernet/microsoft/mana/Makefile new file +6 -0
new file mode 100644
index 000000000000..0edd5bb685f3
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#
+# Makefile for the Microsoft Azure Network Adapter driver
+
+obj-$(CONFIG_MICROSOFT_MANA) += mana.o
+mana-objs := gdma_main.o shm_channel.o hw_channel.o mana_en.o mana_ethtool.o
drivers/net/ethernet/microsoft/mana/gdma.h new file +673 -0

673 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/gdma_main.c new file +1415 -0

1,415 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/hw_channel.c new file +843 -0

838 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/hw_channel.h new file +190 -0

190 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana.h new file +533 -0

533 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana_en.c new file +1895 -0

1,895 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana_ethtool.c new file +250 -0

250 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/shm_channel.c new file +291 -0

291 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/shm_channel.h new file +21 -0
new file mode 100644
index 000000000000..5199b41497ff

Lines of this file are not shown. View the raw message.

drivers/net/hyperv/netvsc_drv.c +12 -2
index 7349a70af083..f682a5572d84 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 22 lines · On Thu, 15 Apr 2021 23:07:05 -0700 Dexuan Cui <[email protected]> wrote:
drivers/net/hyperv/netvsc_drv.c +5 -2
index 7349a70af083..f682a5572d84 100644
@@ -2297,6 +2297,7 @@ static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev)
 {
 	struct device *parent = vf_netdev->dev.parent;
 	struct net_device_context *ndev_ctx;
+	struct net_device *ndev;
 	struct pci_dev *pdev;
 	u32 serial;
 
@@ -2319,8 +2320,17 @@ static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev)
 		if (!ndev_ctx->vf_alloc)
 			continue;
 
-		if (ndev_ctx->vf_serial == serial)
-			return hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ndev_ctx->vf_serial != serial)
+			continue;
+
+		ndev = hv_get_drvdata(ndev_ctx->device_ctx);

drivers/net/hyperv/netvsc_drv.c static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev) · line 2336

Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +7 -0
+		if (ndev->addr_len != vf_netdev->addr_len ||
+		    memcmp(ndev->perm_addr, vf_netdev->perm_addr,
+			   ndev->addr_len) != 0)
+			continue;
+
+		return ndev;
+
 	}
 
 	netdev_notice(vf_netdev,

This probably should be a separate patch.
I think it is trying to address the case of VF discovery in Hyper-V/Azure where the reported VF from Hypervisor is bogus or confused.

maintainer
#2 lore raw
Quoted 53 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Friday, April 16, 2021 12:40 PM
To: Dexuan Cui <[email protected]>
Cc: [email protected]; [email protected]; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; Wei Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Shachar Raindel <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v7 net-next] net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

Review

Quoted 1 line · On Thu, 15 Apr 2021 23:07:05 -0700 Dexuan Cui <[email protected]> wrote:
(quoted diff) +0 -0

b/drivers/net/hyperv/netvsc_drv.c

drivers/net/hyperv/netvsc_drv.c static struct net_device *get_netvsc_byslot(const

Quoted 4 lines

index 7349a70af083..f682a5572d84 100644

drivers/net/hyperv/netvsc_drv.c +0 -0
@@ -2297,6 +2297,7 @@ static struct net_device *get_netvsc_byslot(const

struct net_device *vf_netdev)

drivers/net/hyperv/netvsc_drv.c static struct net_device

Quoted 8 lines
drivers/net/hyperv/netvsc_drv.c +1 -0
 {
 	struct device *parent = vf_netdev->dev.parent;
 	struct net_device_context *ndev_ctx;
+	struct net_device *ndev;
 	struct pci_dev *pdev;
 	u32 serial;
 
@@ -2319,8 +2320,17 @@ static struct net_device

*get_netvsc_byslot(const struct net_device *vf_netdev)

Quoted 18 lines
drivers/net/hyperv/netvsc_drv.c +11 -2
 		if (!ndev_ctx->vf_alloc)
 			continue;
 
-		if (ndev_ctx->vf_serial == serial)
-			return hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ndev_ctx->vf_serial != serial)
+			continue;
+
+		ndev = hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ndev->addr_len != vf_netdev->addr_len ||
+		    memcmp(ndev->perm_addr, vf_netdev->perm_addr,
+			   ndev->addr_len) != 0)
+			continue;
+
+		return ndev;
+
 	}
 
Quoted 7 lines

drivers/net/hyperv/netvsc_drv.c static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev) · line 2336

Quoted 1 line
drivers/net/hyperv/netvsc_drv.c +0 -0
 	netdev_notice(vf_netdev,

This probably should be a separate patch.
I think it is trying to address the case of VF discovery in Hyper-V/Azure where the reported
VF from Hypervisor is bogus or confused.

This is for the Multi vPorts feature of MANA driver, which allows one VF to create multiple vPorts (NICs). They have the same PCI device and same VF serial number, but different MACs.

So we put the change in one patch to avoid distro vendors missing this change when backporting the MANA driver.

Thanks,
- Haiyang

author maintainer
#3 lore raw
Quoted 36 lines

From: Haiyang Zhang <[email protected]>
Sent: Friday, April 16, 2021 10:11 AM

Quoted 22 lines

From: Stephen Hemminger <[email protected]>

Review static struct net_device

Quoted 2 lines

...

(quoted diff) +0 -0
@@ -2319,8 +2320,17 @@ static struct net_device

*get_netvsc_byslot(const struct net_device *vf_netdev)

Quoted 18 lines
drivers/net/hyperv/netvsc_drv.c +11 -2
 		if (!ndev_ctx->vf_alloc)
 			continue;
 
-		if (ndev_ctx->vf_serial == serial)
-			return hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ndev_ctx->vf_serial != serial)
+			continue;
+
+		ndev = hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ndev->addr_len != vf_netdev->addr_len ||
+		    memcmp(ndev->perm_addr, vf_netdev->perm_addr,
+			   ndev->addr_len) != 0)
+			continue;
+
+		return ndev;
+
 	}
 
Quoted 7 lines

drivers/net/hyperv/netvsc_drv.c static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev) · line 2336

Quoted 1 line
drivers/net/hyperv/netvsc_drv.c +0 -0
 	netdev_notice(vf_netdev,

This probably should be a separate patch.
I think it is trying to address the case of VF discovery in Hyper-V/Azure where the reported
VF from Hypervisor is bogus or confused.

This is for the Multi vPorts feature of MANA driver, which allows one VF to create multiple vPorts (NICs). They have the same PCI device and same VF serial number, but different MACs.

Quoted 5 lines

So we put the change in one patch to avoid distro vendors missing this change when backporting the MANA driver.

Thanks,
- Haiyang

The netvsc change should come together in the same patch with this VF driver, otherwise the multi-vPorts functionality doesn't work properly.

The netvsc change should not break any other existing VF drivers, because Hyper-V NIC SR-IOV implementation requires the the NetVSC network interface and the VF network interface should have the same MAC address, otherwise things won't work.

Thanks,
Dexuan

#4 lore raw
Quoted 19 lines · On Fri, 16 Apr 2021 17:58:45 +0000 Dexuan Cui <[email protected]> wrote:
Quoted 10 lines
Quoted 5 lines

This probably should be a separate patch.
I think it is trying to address the case of VF discovery in Hyper-V/Azure where the reported
VF from Hypervisor is bogus or confused.

This is for the Multi vPorts feature of MANA driver, which allows one VF to create multiple vPorts (NICs). They have the same PCI device and same VF serial number, but different MACs.

Quoted 5 lines

So we put the change in one patch to avoid distro vendors missing this change when backporting the MANA driver.

Thanks,
- Haiyang

The netvsc change should come together in the same patch with this VF driver, otherwise the multi-vPorts functionality doesn't work properly.

Quoted 7 lines

The netvsc change should not break any other existing VF drivers, because Hyper-V NIC SR-IOV implementation requires the the NetVSC network interface and the VF network interface should have the same MAC address, otherwise things won't work.

Thanks,
Dexuan

Distro vendors should be able to handle a patch series. Don't see why this could not be two patch series.

author maintainer
#5 lore raw
Quoted 31 lines

From: Stephen Hemminger <[email protected]>
Sent: Friday, April 16, 2021 11:09 AM
...

Quoted 3 lines · On Fri, 16 Apr 2021 17:58:45 +0000 Dexuan Cui <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines

This probably should be a separate patch.
I think it is trying to address the case of VF discovery in Hyper-V/Azure

where

Quoted 21 lines
Quoted 7 lines
Quoted 2 lines

the reported
VF from Hypervisor is bogus or confused.

This is for the Multi vPorts feature of MANA driver, which allows one VF to create multiple vPorts (NICs). They have the same PCI device and same VF serial number, but different MACs.

Quoted 5 lines

So we put the change in one patch to avoid distro vendors missing this change when backporting the MANA driver.

Thanks,
- Haiyang

The netvsc change should come together in the same patch with this VF driver, otherwise the multi-vPorts functionality doesn't work properly.

The netvsc change should not break any other existing VF drivers, because Hyper-V NIC SR-IOV implementation requires the the NetVSC network interface and the VF network interface should have the same MAC address, otherwise things won't work.

Quoted 5 lines
Quoted 2 lines

Thanks,
Dexuan

Distro vendors should be able to handle a patch series. Don't see why this could not be two patch series.

Ok. Will split this into 2 patches (the first one is the netvsc change, and the second is the Linux VF driver) and post v8 shortly.