Skip to contentSkip to conversation

Add Mellanox BlueField Gigabit Ethernet driver

PATCH Superseded David Thompson 4 participants 8 messages May 28, 2021 – Jun 1, 2021

Subsystems (2)

v5 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 Add Mellanox BlueField Gigabit Ethernet driver

This patch adds build and driver logic for the "mlxbf_gige" Ethernet driver from Mellanox Technologies. The second generation BlueField SoC from Mellanox supports an out-of-band GigaBit Ethernet management port to the Arm subsystem. This driver supports TCP/IP network connectivity for that port, and provides back-end routines to handle basic ethtool requests.

The driver interfaces to the Gigabit Ethernet block of BlueField SoC via MMIO accesses to registers, which contain control information or pointers describing transmit and receive resources. There is a single transmit queue, and the port supports transmit ring sizes of 4 to 256 entries. There is a single receive queue, and the port supports receive ring sizes of 32 to 32K entries. The transmit and receive rings are allocated from DMA coherent memory. There is a 16-bit producer and consumer index per ring to denote software ownership and hardware ownership, respectively.

The main driver logic such as probe(), remove(), and netdev ops are in "mlxbf_gige_main.c". Logic in "mlxbf_gige_rx.c" and "mlxbf_gige_tx.c" handles the packet processing for receive and transmit respectively.

The logic in "mlxbf_gige_ethtool.c" supports the handling of some basic ethtool requests: get driver info, get ring parameters, get registers, and get statistics.

The logic in "mlxbf_gige_mdio.c" is the driver controlling the Mellanox BlueField hardware that interacts with a PHY device via MDIO/MDC pins. This driver does the following:
- At driver probe time, it configures several BlueField MDIO parameters such as sample rate, full drive, voltage and MDC
- It defines functions to read and write MDIO registers and registers the MDIO bus.
- It defines the phy interrupt handler reporting a link up/down status change
- This driver's probe is invoked from the main driver logic while the phy interrupt handler is registered in ndo_open.

Driver limitations
- Only supports 1Gbps speed
- Only supports GMII protocol
- Supports maximum packet size of 2KB
- Does not support scatter-gather buffering

Testing
- Successful build of kernel for ARM64, ARM32, X86_64
- Tested ARM64 build on FastModels & Palladium
- Tested ARM64 build on several Mellanox boards that are built with the BlueField-2 SoC. The testing includes coverage in the areas of networking (e.g. ping, iperf, ifconfig, route), file transfers (e.g. SCP), and various ethtool options relevant to this driver.

v4 -> v5
Created a separate interrupt controller for the GPIO PHY interrupt and as a result, the GIGE driver no longer depends on GPIO driver Updated the logic in mlxbf_gige_adjust_link() to store the negotiated pause settings into the driver's private settings.
Modified logic to only change enable bit in RX_DMA register Changed logic to only map and unmap the actual length of the TX SKB, instead using the default size.
Added better error handling to open() method
Modified receive packet logic to use polarity bit to signify ownership (software vs. hardware) of the RX CQE slot
v3 -> v4
Main driver module broken out into rx, tx, intr, and ethtool modules Removed some GPIO PHY interrupt logic; moved to GPIO_MLXBF2 driver
v2 -> v3
Added logic to handle PHY link up/down interrupts
Use streaming DMA mapping for packet buffers
Changed logic to use standard iopoll methods
Changed PHY logic to not allow C45 transactions
Enhanced the error handling in open() method
Enhanced start_xmit() method to use xmit_more mechanism Added support for ndo_get_stats64
Removed standard stats from "ethtool -S" output
v1 -> v2:
Fixed all warnings raised by "make C=1" and "make W=1"
a) Changed logic in mlxbf_gige_rx_deinit() and mlxbf_gige_tx_deinit() to initialize relevant pointers as NULL, not 0
b) Change mlxbf_gige_get_mac_rx_filter() to return void, as this function's return status is not used by caller
c) Fixed type definition of "buff" in mlxbf_gige_get_regs()

---
 drivers/net/ethernet/mellanox/Kconfig         |   1 +
 drivers/net/ethernet/mellanox/Makefile        |   1 +
 .../net/ethernet/mellanox/mlxbf_gige/Kconfig  |  13 +
 .../net/ethernet/mellanox/mlxbf_gige/Makefile |  11 +
 .../ethernet/mellanox/mlxbf_gige/mlxbf_gige.h | 193 ++++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_ethtool.c  | 140 ++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_gpio.c     | 208 ++++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_intr.c     | 143 ++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_main.c     | 459 ++++++++++++++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_mdio.c     | 187 +++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_regs.h     |  78 +++
 .../mellanox/mlxbf_gige/mlxbf_gige_rx.c       | 318 ++++++++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_tx.c       | 284 +++++++++++
 13 files changed, 2036 insertions(+)
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/Kconfig
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/Makefile
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_ethtool.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_gpio.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_intr.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_mdio.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_regs.h
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_rx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
drivers/net/ethernet/mellanox/Kconfig +1 -0
index ff6613a5cdd3..b4f66eb9ddb9 100644
@@ -22,5 +22,6 @@ source "drivers/net/ethernet/mellanox/mlx4/Kconfig"
 source "drivers/net/ethernet/mellanox/mlx5/core/Kconfig"
 source "drivers/net/ethernet/mellanox/mlxsw/Kconfig"
 source "drivers/net/ethernet/mellanox/mlxfw/Kconfig"
+source "drivers/net/ethernet/mellanox/mlxbf_gige/Kconfig"
 
 endif # NET_VENDOR_MELLANOX
drivers/net/ethernet/mellanox/Makefile +1 -0
index 79773ac331ee..d4b5f547a727 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_MLX4_CORE) += mlx4/
 obj-$(CONFIG_MLX5_CORE) += mlx5/core/
 obj-$(CONFIG_MLXSW_CORE) += mlxsw/
 obj-$(CONFIG_MLXFW) += mlxfw/
+obj-$(CONFIG_MLXBF_GIGE) += mlxbf_gige/
drivers/net/ethernet/mellanox/mlxbf_gige/Kconfig new file +13 -0
new file mode 100644
index 000000000000..08a4487932fe
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+#
+# Mellanox GigE driver configuration
+#
+
+config MLXBF_GIGE
+	tristate "Mellanox Technologies BlueField Gigabit Ethernet support"
+	depends on (ARM64 || COMPILE_TEST) && ACPI
+	select PHYLIB
+	help
+	  The second generation BlueField SoC from Mellanox Technologies
+	  supports an out-of-band Gigabit Ethernet management port to the
+	  Arm subsystem.
drivers/net/ethernet/mellanox/mlxbf_gige/Makefile new file +11 -0
new file mode 100644
index 000000000000..e57c1375f236
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+obj-$(CONFIG_MLXBF_GIGE) += mlxbf_gige.o
+
+mlxbf_gige-y := mlxbf_gige_ethtool.o \
+		mlxbf_gige_gpio.o \
+		mlxbf_gige_intr.o \
+		mlxbf_gige_main.o \
+		mlxbf_gige_mdio.o \
+		mlxbf_gige_rx.o   \
+		mlxbf_gige_tx.o
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h new file +193 -0

193 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_ethtool.c new file +140 -0

140 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_gpio.c new file +208 -0

208 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_intr.c new file +143 -0

143 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +459 -0

459 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_mdio.c new file +187 -0

187 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_regs.h new file +78 -0

78 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_rx.c new file +318 -0

318 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c new file +284 -0

284 lines not on this page · raw message

Signature · 1 line
2.30.1
maintainer
#1 lore raw
Quoted 7 lines · On Fri, 28 May 2021 15:37:19 -0400 David Thompson wrote:

This patch adds build and driver logic for the "mlxbf_gige" Ethernet driver from Mellanox Technologies. The second generation BlueField SoC from Mellanox supports an out-of-band GigaBit Ethernet management port to the Arm subsystem. This driver supports TCP/IP network connectivity for that port, and provides back-end routines to handle basic ethtool requests.

Please address the 32 bit build warnings.

#2 lore raw
Quoted 9 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Friday, May 28, 2021 6:15 PM
To: David Thompson <[email protected]>
Cc: [email protected]; [email protected]; Liming Sun <[email protected]>; Asmaa Mnebhi <[email protected]>
Subject: Re: [PATCH net-next v5] Add Mellanox BlueField Gigabit Ethernet driver

On Fri, 28 May 2021 15:37:19 -0400 David Thompson wrote:

Quoted 8 lines
Quoted 6 lines

This patch adds build and driver logic for the "mlxbf_gige" Ethernet driver from Mellanox Technologies. The second generation BlueField SoC from Mellanox supports an out-of-band GigaBit Ethernet management port to the Arm subsystem. This driver supports TCP/IP network connectivity for that port, and provides back-end routines to handle basic ethtool requests.

Please address the 32 bit build warnings.

My apologies Jakub. Prior versions of this patch did compile for ARM32.

I will change the code accordingly so that ARM32 build succeeds.

Regards, Dave

#3 lore raw
In reply to David Thompson
Quoted 9 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +9 -0
+static void mlxbf_gige_adjust_link(struct net_device *netdev)
+{
+	struct mlxbf_gige *priv = netdev_priv(netdev);
+	struct phy_device *phydev = netdev->phydev;
+
+	if (phydev->link) {
+		priv->rx_pause = phydev->pause;
+		priv->tx_pause = phydev->pause;
+	}

...

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c line 392

Quoted 2 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +2 -0
+	/* MAC supports symmetric flow control */
+	phy_support_sym_pause(phydev);

What i don't see anywhere is you acting on the results of the pause negotiation. It could be, mlxbf_gige_adjust_link() tells you the peer does not support pause, and you need to disable it in this MAC as well. It is a negotiation, after all.

Andrew

#4 lore raw

Hi Andrew,

Thank you for your feedback.

Our Hardware only supports a single configuration: symmetric flow control, nothing else. This is why we have to enforce this, and there is not adjustment to do. The only reason we set priv->rx_pause and priv->tx_pause here is for ethtool purposes.

Thanks.
Asmaa
-----Original Message-----
From: Andrew Lunn <[email protected]>
Sent: Friday, May 28, 2021 8:22 PM
To: David Thompson <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Liming Sun <[email protected]>; Asmaa Mnebhi <[email protected]>
Subject: Re: [PATCH net-next v5] Add Mellanox BlueField Gigabit Ethernet driver

Quoted 8 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +8 -0
+static void mlxbf_gige_adjust_link (struct net_device *netdev) {
+	struct mlxbf_gige *priv = netdev_priv(netdev);
+	struct phy_device *phydev = netdev->phydev;
+
+	if (phydev->link) {
+		priv->rx_pause = phydev->pause;
+		priv->tx_pause = phydev->pause;
+	}

...

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c line 392

Quoted 2 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +2 -0
+	/* MAC supports symmetric flow control */
+	phy_support_sym_pause(phydev);

What i don't see anywhere is you acting on the results of the pause negotiation. It could be, mlxbf_gige_adjust_link() tells you the peer does not support pause, and you need to disable it in this MAC as well. It is a negotiation, after all.

Andrew

#5 lore raw

Please do not top post.

Quoted 20 lines

Thanks.
Asmaa
-----Original Message-----
From: Andrew Lunn <[email protected]>
Sent: Friday, May 28, 2021 8:22 PM
To: David Thompson <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Liming Sun <[email protected]>; Asmaa Mnebhi <[email protected]>
Subject: Re: [PATCH net-next v5] Add Mellanox BlueField Gigabit Ethernet driver

Quoted 8 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +8 -0
+static void mlxbf_gige_adjust_link (struct net_device *netdev) {
+	struct mlxbf_gige *priv = netdev_priv(netdev);
+	struct phy_device *phydev = netdev->phydev;
+
+	if (phydev->link) {
+		priv->rx_pause = phydev->pause;
+		priv->tx_pause = phydev->pause;
+	}

...

Quoted 8 lines

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c line 392

Quoted 2 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +2 -0
+	/* MAC supports symmetric flow control */
+	phy_support_sym_pause(phydev);

What i don't see anywhere is you acting on the results of the pause negotiation. It could be, mlxbf_gige_adjust_link() tells you the peer does not support pause, and you need to disable it in this MAC as well. It is a negotiation, after all.

From what you are saying, this is all wrong. You don't negotiate at all. So don't report negotiated values in ethtool, just report the fixed values, and do not set autoneg in ethtool_pauseparam because you have not negotiated it.

You also should not be calling phy_support_sym_pause(), which means, i support negotiated pause up to and including symmetric pause. You might also need to clear the pause bits from phydev->advertising.

Andrew

#6 lore raw

-----Original Message-----
From: Andrew Lunn <[email protected]>
Sent: Tuesday, June 1, 2021 9:20 AM
To: Asmaa Mnebhi <[email protected]>
Cc: David Thompson <[email protected]>; [email protected]; [email protected]; [email protected]; Liming Sun <[email protected]>
Subject: Re: [PATCH net-next v5] Add Mellanox BlueField Gigabit Ethernet driver

Please do not top post.

Quoted 22 lines

Thanks.
Asmaa
-----Original Message-----
From: Andrew Lunn <[email protected]>
Sent: Friday, May 28, 2021 8:22 PM
To: David Thompson <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Liming Sun <[email protected]>; Asmaa Mnebhi <[email protected]>
Subject: Re: [PATCH net-next v5] Add Mellanox BlueField Gigabit Ethernet driver

Quoted 8 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +8 -0
+static void mlxbf_gige_adjust_link (struct net_device *netdev) {
+	struct mlxbf_gige *priv = netdev_priv(netdev);
+	struct phy_device *phydev = netdev->phydev;
+
+	if (phydev->link) {
+		priv->rx_pause = phydev->pause;
+		priv->tx_pause = phydev->pause;
+	}

...

Quoted 8 lines

drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c line 392

Quoted 2 lines
drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c new file +2 -0
+	/* MAC supports symmetric flow control */
+	phy_support_sym_pause(phydev);

What i don't see anywhere is you acting on the results of the pause negotiation. It could be, mlxbf_gige_adjust_link() tells you the peer does not support pause, and you need to disable it in this MAC as well. It is a negotiation, after all.

From what you are saying, this is all wrong. You don't negotiate at all. So don't report negotiated values in ethtool, just report the fixed values, and do not set autoneg in ethtool_pauseparam because you have not negotiated it.

You also should not be calling phy_support_sym_pause(), which means, i support negotiated pause up to and including symmetric pause. You might also need to clear the pause bits from phydev->advertising.

Asmaa>> Sounds good! We will update this in our next patch.
I guess I misunderstood. I thought 1G speed always requires autonegotiation. And phy_support_sym_pause would display that the only pause supported by this MAC is symmetric. But what you are saying is that we don't really "negotiate" the pause since our MAC HW supports only symmetric pause? /**
* phy_support_sym_pause - Enable support of symmetrical pause
* @phydev: target phy_device struct
*
* Description: Called by the MAC to indicate is supports symmetrical
* Pause, but not asym pause.
*/
void phy_support_sym_pause(struct phy_device *phydev)

#7 lore raw
Quoted 5 lines

I guess I misunderstood. I thought 1G speed always requires autonegotiation. And phy_support_sym_pause would display that the only pause supported by this MAC is symmetric. But what you are saying is that we don't really "negotiate" the pause since our MAC HW supports only symmetric pause?

The point of negotiation is that the link peer can say it does not support pause, because it is missing the needed hardware, or because of a policy decision. Pause can be bad, particularly if used away from the leaf nodes of the network. If the negotiation says pause should not be used, you really should not use it.

Now, you hardware seems to be very limited. You cannot turn pause off. It is hard coded on. So you cannot negotiate this feature. You need to report that in ethtool. And you are also going to have to manually configure pause in the link peer. Since it cannot be negotiated, the peer is not going to use pause. Meaning by default, pause in your MAC is pointless. You need to manually configure pause in the link peer, by telling it to also not to negotiate, and perform fixed symmetric pause.

Hopefully for the next generation of the hardware, you can fix this bad design decision.

Andrew