Skip to contentSkip to conversation

net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

PATCH Superseded Dexuan Cui 6 participants 11 messages Apr 8, 2021 – Apr 9, 2021

Subsystems (4)

v3 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Dexuan Cui

author maintainer
#0 lore raw

PATCH v3 1/1 net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

Add a VF driver for Microsoft Azure Network Adapter (MANA) that will be available in the future.

---
 MAINTAINERS                                   |    4 +-
 drivers/net/ethernet/Kconfig                  |    1 +
 drivers/net/ethernet/Makefile                 |    1 +
 drivers/net/ethernet/microsoft/Kconfig        |   29 +
 drivers/net/ethernet/microsoft/Makefile       |    5 +
 drivers/net/ethernet/microsoft/mana/Makefile  |    6 +
 drivers/net/ethernet/microsoft/mana/gdma.h    |  728 +++++++
 .../net/ethernet/microsoft/mana/gdma_main.c   | 1515 ++++++++++++++
 .../net/ethernet/microsoft/mana/hw_channel.c  |  859 ++++++++
 .../net/ethernet/microsoft/mana/hw_channel.h  |  186 ++
 drivers/net/ethernet/microsoft/mana/mana.h    |  531 +++++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 1827 +++++++++++++++++
 .../ethernet/microsoft/mana/mana_ethtool.c    |  278 +++
 .../net/ethernet/microsoft/mana/shm_channel.c |  292 +++
 .../net/ethernet/microsoft/mana/shm_channel.h |   21 +
 15 files changed, 6282 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/microsoft/Kconfig
 create mode 100644 drivers/net/ethernet/microsoft/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma_main.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_en.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_ethtool.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.h

Changes in v2:

Removed the module_param(num_queues,...). [Andrew Lunn]
Changed pr_err() to netdev_err() and dev_err(). [Andrew Lunn]
Used reverse X-mas tree style in all the functions. [Andrew Lunn]

Changed "= { 0 };" to "= {};" for struct variables. [Leon Romanovsky]

Addressed 3 build failures on i386, arc and powerpc by making the the driver dependent on X86_64 in Kconfig: so far, the driver is only validated on X86_64 (in the future, we may enable the driver for ARM64).

Also made some cosmetic changes.

Rebased the patch to the latest net-next.

BTW, the support of XDP and BQL haven't been implemented. They are on our TO-DO list.

Changes in v3:
Changed the Kconfig file:
Microsoft Azure Network Device ==> Microsoft Network Devices Drop the default m
validated ==> supported

Fixed some warnings reported by "kernel test robot" (there are
still some warnings against another driver pci-hyperv.c, which I'll fix in a separate patch for the PCI tree).

MAINTAINERS +3 -1
index 3ea9539821b5..6c5bcdcab9d3 100644
@@ -8261,11 +8261,12 @@ S:	Maintained
 T:	git git://linuxtv.org/media_tree.git
 F:	drivers/media/i2c/hi556.c
 
-Hyper-V CORE AND DRIVERS
+Hyper-V/Azure CORE AND DRIVERS
 M:	"K. Y. Srinivasan" <[email protected]>
 M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
+M:	Dexuan Cui <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -8282,6 +8283,7 @@ F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
 F:	drivers/iommu/hyperv-iommu.c
+F:	drivers/net/ethernet/microsoft/
 F:	drivers/net/hyperv/
 F:	drivers/pci/controller/pci-hyperv-intf.c
 F:	drivers/pci/controller/pci-hyperv.c
drivers/net/ethernet/Kconfig +1 -0
index 4b85f2b74872..d46460c5b44d 100644
@@ -82,6 +82,7 @@ source "drivers/net/ethernet/huawei/Kconfig"
 source "drivers/net/ethernet/i825xx/Kconfig"
 source "drivers/net/ethernet/ibm/Kconfig"
 source "drivers/net/ethernet/intel/Kconfig"
+source "drivers/net/ethernet/microsoft/Kconfig"
 source "drivers/net/ethernet/xscale/Kconfig"
 
 config JME
drivers/net/ethernet/Makefile +1 -0
index 9394493e8187..cb3f9084a21b 100644
@@ -45,6 +45,7 @@ obj-$(CONFIG_NET_VENDOR_HUAWEI) += huawei/
 obj-$(CONFIG_NET_VENDOR_IBM) += ibm/
 obj-$(CONFIG_NET_VENDOR_INTEL) += intel/
 obj-$(CONFIG_NET_VENDOR_I825XX) += i825xx/
+obj-$(CONFIG_NET_VENDOR_MICROSOFT) += microsoft/
 obj-$(CONFIG_NET_VENDOR_XSCALE) += xscale/
 obj-$(CONFIG_JME) += jme.o
 obj-$(CONFIG_KORINA) += korina.o
drivers/net/ethernet/microsoft/Kconfig new file +29 -0
new file mode 100644
index 000000000000..e1ac0a5d808d
@@ -0,0 +1,29 @@
+#
+# Microsoft Azure network device configuration
+#
+
+config NET_VENDOR_MICROSOFT
+	bool "Microsoft Network Devices"
+	default y
+	help
+	  If you have a network (Ethernet) device belonging to this class, say Y.
+
+	  Note that the answer to this question doesn't directly affect the
+	  kernel: saying N will just cause the configurator to skip the
+	  question about Microsoft network devices. If you say Y, you will be
+	  asked for your specific device in the following question.
+
+if NET_VENDOR_MICROSOFT
+
+config MICROSOFT_MANA
+	tristate "Microsoft Azure Network Adapter (MANA) support"
+	depends on PCI_MSI && X86_64
+	select PCI_HYPERV
+	help
+	  This driver supports Microsoft Azure Network Adapter (MANA).
+	  So far, the driver is only supported on X86_64.
+
+	  To compile this driver as a module, choose M here.
+	  The module will be called mana.
+
+endif #NET_VENDOR_MICROSOFT
drivers/net/ethernet/microsoft/Makefile new file +5 -0
new file mode 100644
index 000000000000..d2ddc218135f
@@ -0,0 +1,5 @@
+#
+# Makefile for the Microsoft Azure network device driver.
+#
+
+obj-$(CONFIG_MICROSOFT_MANA) += mana/
drivers/net/ethernet/microsoft/mana/Makefile new file +6 -0
new file mode 100644
index 000000000000..0edd5bb685f3
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#
+# Makefile for the Microsoft Azure Network Adapter driver
+
+obj-$(CONFIG_MICROSOFT_MANA) += mana.o
+mana-objs := gdma_main.o shm_channel.o hw_channel.o mana_en.o mana_ethtool.o
drivers/net/ethernet/microsoft/mana/gdma.h new file +728 -0

728 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/gdma_main.c new file +1515 -0

1,515 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/hw_channel.c new file +859 -0

683 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/hw_channel.h new file +186 -0

186 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana.h new file +531 -0

531 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana_en.c new file +1827 -0

1,827 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/mana_ethtool.c new file +278 -0

278 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/shm_channel.c new file +292 -0

292 lines not on this page · raw message

drivers/net/ethernet/microsoft/mana/shm_channel.h new file +21 -0
new file mode 100644
index 000000000000..aa55837239d7

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 27 lines · On Thu, 8 Apr 2021 15:58:40 -0700 Dexuan Cui <[email protected]> wrote:

Add a VF driver for Microsoft Azure Network Adapter (MANA) that will be available in the future.

---
 MAINTAINERS                                   |    4 +-
 drivers/net/ethernet/Kconfig                  |    1 +
 drivers/net/ethernet/Makefile                 |    1 +
 drivers/net/ethernet/microsoft/Kconfig        |   29 +
 drivers/net/ethernet/microsoft/Makefile       |    5 +
 drivers/net/ethernet/microsoft/mana/Makefile  |    6 +
 drivers/net/ethernet/microsoft/mana/gdma.h    |  728 +++++++
 .../net/ethernet/microsoft/mana/gdma_main.c   | 1515 ++++++++++++++
 .../net/ethernet/microsoft/mana/hw_channel.c  |  859 ++++++++
 .../net/ethernet/microsoft/mana/hw_channel.h  |  186 ++
 drivers/net/ethernet/microsoft/mana/mana.h    |  531 +++++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 1827 +++++++++++++++++
 .../ethernet/microsoft/mana/mana_ethtool.c    |  278 +++
 .../net/ethernet/microsoft/mana/shm_channel.c |  292 +++
 .../net/ethernet/microsoft/mana/shm_channel.h |   21 +
 15 files changed, 6282 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/microsoft/Kconfig
 create mode 100644 drivers/net/ethernet/microsoft/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/Makefile
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma.h
Quoted 8 lines
 create mode 100644 drivers/net/ethernet/microsoft/mana/gdma_main.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/hw_channel.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana.h
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_en.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_ethtool.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.c
 create mode 100644 drivers/net/ethernet/microsoft/mana/shm_channel.h

Linux kernel doesn't do namespaces in the code, so every new driver needs to worry about global symbols clashing

Please make sure there are not name conflicts with other drivers around variable or functions name "gdma_". Noticed there is one driver in staging using similar names (drivers/staging/ralink-gdma/ralink-gdma.c). Granted that is in the staging doghouse so probably not important but might show up as a name conflict with something like the randconfig testing.

#2 lore raw
Quoted 2 lines

Linux kernel doesn't do namespaces in the code, so every new driver needs to worry about global symbols clashing

This driver is called mana, yet the code uses ana. It would be good to resolve this inconsistency as well. Ideally, you want to prefix everything with ana_ or mana_, depending on what you choose, so we have a clean namespace.

Andrew

author maintainer
#3 lore raw
Quoted 7 lines

From: Andrew Lunn <[email protected]>
Sent: Thursday, April 8, 2021 5:30 PM
To: Stephen Hemminger <[email protected]>
...

Quoted 2 lines

Linux kernel doesn't do namespaces in the code, so every new driver needs to worry about global symbols clashing

Quoted 6 lines

This driver is called mana, yet the code uses ana. It would be good to resolve this inconsistency as well. Ideally, you want to prefix everything with ana_ or mana_, depending on what you choose, so we have a clean namespace.

Andrew

Thanks for the suggestion! Let me think about this and work out a solution.

maintainer
#4 lore raw
In reply to Dexuan Cui

From: Dexuan Cui <[email protected]>
Date: Thu, 8 Apr 2021 15:58:40 -0700

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +8 -0
+struct gdma_msg_hdr {
+	u32 hdr_type;
+	u32 msg_type;
+	u16 msg_version;
+	u16 hwc_msg_id;
+	u32 msg_size;
+} __packed;
+

drivers/net/ethernet/microsoft/mana/gdma.h line 119

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +10 -0
+struct gdma_dev_id {
+	union {
+		struct {
+			u16 type;
+			u16 instance;
+		};
+
+		u32 as_uint32;
+	};
+} __packed;

Please don't use __packed unless absolutely necessary. It generates suboptimal code (byte at a time accesses etc.) and for many of these you don't even need it.

Thank you.

author maintainer
#5 lore raw
Quoted 17 lines

From: David Miller <[email protected]>
Sent: Thursday, April 8, 2021 4:46 PM
...

Quoted 14 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +14 -0
+struct gdma_msg_hdr {
+	u32 hdr_type;
+	u32 msg_type;
+	u16 msg_version;
+	u16 hwc_msg_id;
+	u32 msg_size;
+} __packed;
+
+struct gdma_dev_id {
+	union {
+		struct {
+			u16 type;
+			u16 instance;
+		};
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/gdma.h line 119

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +4 -0
+
+		u32 as_uint32;
+	};
+} __packed;

Please don't use __packed unless absolutely necessary. It generates suboptimal code (byte at a time
accesses etc.) and for many of these you don't even need it.

In the driver code, all the structs/unions marked by __packed are used to talk with the hardware, so I think __packed is necessary here?

Do you think if it's better if we remove all the __packed, and add static_assert(sizeof(struct XXX) == YYY) instead? e.g.

@@ -105,7 +105,8 @@ struct gdma_msg_hdr {
        u16 msg_version;
        u16 hwc_msg_id;
        u32 msg_size;

-} __packed;
+};
+static_assert(sizeof(struct gdma_msg_hdr) == 16);

 struct gdma_dev_id {
        union {

Now I'm trying to figure out how other NIC drivers define structs/unions.

Thanks,
Dexuan

maintainer
#6 lore raw

From: Dexuan Cui <[email protected]>
Date: Fri, 9 Apr 2021 00:24:51 +0000

Quoted 26 lines
Quoted 17 lines

From: David Miller <[email protected]>
Sent: Thursday, April 8, 2021 4:46 PM
...

Quoted 14 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +14 -0
+struct gdma_msg_hdr {
+	u32 hdr_type;
+	u32 msg_type;
+	u16 msg_version;
+	u16 hwc_msg_id;
+	u32 msg_size;
+} __packed;
+
+struct gdma_dev_id {
+	union {
+		struct {
+			u16 type;
+			u16 instance;
+		};
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/gdma.h line 119

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +4 -0
+
+		u32 as_uint32;
+	};
+} __packed;

Please don't use __packed unless absolutely necessary. It generates suboptimal code (byte at a time
accesses etc.) and for many of these you don't even need it.

Quoted 2 lines

In the driver code, all the structs/unions marked by __packed are used to talk with the hardware, so I think __packed is necessary here?

It actually isan't in many cases, check with and without the __packed directive and see if anything chasnges.

Review struct gdma_msg_hdr { · line 109

Quoted 10 lines

Do you think if it's better if we remove all the __packed, and add static_assert(sizeof(struct XXX) == YYY) instead? e.g.

(quoted diff) +2 -1
@@ -105,7 +105,8 @@ struct gdma_msg_hdr {
        u16 msg_version;
        u16 hwc_msg_id;
        u32 msg_size;
-} __packed;
+};
+static_assert(sizeof(struct gdma_msg_hdr) == 16);

This won't make sure the structure member offsets are what you expect.

I think you'll have to go through the structures one-by-one by hand to figure out which ones really require the __packed attribute and which do not.

Thank you.

author maintainer
#7 lore raw
Quoted 8 lines

From: David Miller <[email protected]>
Sent: Thursday, April 8, 2021 5:41 PM

Quoted 3 lines

...
In the driver code, all the structs/unions marked by __packed are used to talk with the hardware, so I think __packed is necessary here?

It actually isan't in many cases, check with and without the __packed directive and see if anything chasnges.

Will do.

Quoted 13 lines

Review struct gdma_msg_hdr { · line 109

Quoted 10 lines

Do you think if it's better if we remove all the __packed, and add static_assert(sizeof(struct XXX) == YYY) instead? e.g.

(quoted diff) +2 -1
@@ -105,7 +105,8 @@ struct gdma_msg_hdr {
        u16 msg_version;
        u16 hwc_msg_id;
        u32 msg_size;
-} __packed;
+};
+static_assert(sizeof(struct gdma_msg_hdr) == 16);

This won't make sure the structure member offsets are what you expect.

Quoted 2 lines

I think you'll have to go through the structures one-by-one by hand to figure out which ones really require the __packed attribute and which do not.

Got it. Let me see if I can remove all the __packed.

maintainer
#8 lore raw
In reply to David Miller
Quoted 64 lines

-----Original Message-----
From: David Miller <[email protected]>
Sent: Thursday, April 8, 2021 8:41 PM
To: Dexuan Cui <[email protected]>
Cc: [email protected]; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; Wei Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH v3 net-next] net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)

From: Dexuan Cui <[email protected]>
Date: Fri, 9 Apr 2021 00:24:51 +0000

Quoted 26 lines
Quoted 17 lines

From: David Miller <[email protected]>
Sent: Thursday, April 8, 2021 4:46 PM
...

Quoted 14 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +14 -0
+struct gdma_msg_hdr {
+	u32 hdr_type;
+	u32 msg_type;
+	u16 msg_version;
+	u16 hwc_msg_id;
+	u32 msg_size;
+} __packed;
+
+struct gdma_dev_id {
+	union {
+		struct {
+			u16 type;
+			u16 instance;
+		};
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/gdma.h line 119

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/gdma.h new file +4 -0
+
+		u32 as_uint32;
+	};
+} __packed;

Please don't use __packed unless absolutely necessary. It generates suboptimal code (byte at a time
accesses etc.) and for many of these you don't even need it.

Quoted 2 lines

In the driver code, all the structs/unions marked by __packed are used to talk with the hardware, so I think __packed is necessary here?

It actually isan't in many cases, check with and without the __packed directive
and see if anything chasnges.

Review struct gdma_msg_hdr { · line 109

Quoted 10 lines

Do you think if it's better if we remove all the __packed, and add static_assert(sizeof(struct XXX) == YYY) instead? e.g.

(quoted diff) +2 -1
@@ -105,7 +105,8 @@ struct gdma_msg_hdr {
        u16 msg_version;
        u16 hwc_msg_id;
        u32 msg_size;
-} __packed;
+};
+static_assert(sizeof(struct gdma_msg_hdr) == 16);

This won't make sure the structure member offsets are what you expect.

Quoted 2 lines

I think you'll have to go through the structures one-by-one by hand to figure out which ones really require the __packed attribute and which do not.

For the structs containing variables with the same sizes, or already size aligned variables, we knew the __packed has no effect. And for these structs, it doesn't cause performance impact either, correct?

But in the future, if different sized variables are added, the __packed may become necessary again. To prevent anyone accidently forget to add __packed when adding new variables to these structs, can we keep the __packed for all messages going through the "wire"?

Thanks,
- Haiyang

#9 lore raw

For the structs containing variables with the same sizes, or already size aligned

Quoted 7 lines

variables, we knew the __packed has no effect. And for these structs, it doesn't cause performance impact either, correct?

But in the future, if different sized variables are added, the __packed may become necessary again. To prevent anyone accidently forget to add __packed when adding new variables to these structs, can we keep the __packed for all messages going through the "wire"?

It should not be a problem because anybody adding new variables should know packed is not liked in the kernel and will take care.

If you want to be paranoid add a BUILD_BUG_ON(size(struct foo) != 42);

Andrew

#10 lore raw
In reply to Dexuan Cui
Quoted 5 lines · Dexuan Cui

Changes in v3:
Changed the Kconfig file:
Microsoft Azure Network Device ==> Microsoft Network Devices Drop the default m
validated ==> supported

Hi,
I am satisfied with the Kconfig changes.

thanks.

Signature · 1 line
~Randy