Skip to contentSkip to conversation

mlx4: make sure to always set the port type

PATCH In review Jakub Kicinski 3 participants 11 messages Sep 4, 2020 – Sep 7, 2020

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 mlx4: make sure to always set the port type

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +11 -0
index 258c7a96f269..70cf24ba71e4 100644
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);
+	else if (!IS_ENABLED(CONFIG_MLX4_INFINIBAND) &&
+		 dev->caps.port_type[port] == MLX4_PORT_TYPE_IB)
+		devlink_port_type_ib_set(&info->devlink_port, NULL);
+
 	info->dev = dev;
 	info->port = port;
 	if (!mlx4_is_slave(dev)) {
Signature · 1 line
2.26.2
#1 lore raw
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);
                                                               ^^^^^

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

Thanks

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +4 -1
+	else if (!IS_ENABLED(CONFIG_MLX4_INFINIBAND) &&
+		 dev->caps.port_type[port] == MLX4_PORT_TYPE_IB)
+		devlink_port_type_ib_set(&info->devlink_port, NULL);
+
 	info->dev = dev;
 	info->port = port;
 	if (!mlx4_is_slave(dev)) {
--

2.26.2

author maintainer
#2 lore raw
Quoted 35 lines · On Sun, 6 Sep 2020 10:27:59 +0300 Leon Romanovsky wrote:
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);  
                                                               ^^^^^
Quoted 3 lines

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

Damn, good catch. It's not supposed to be required. I'll patch devlink.

Quoted 1 line

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

#3 lore raw
Quoted 36 lines · Sun, Sep 06, 2020 at 06:33:05PM CEST, [email protected] wrote:
Quoted 35 lines · On Sun, 6 Sep 2020 10:27:59 +0300 Leon Romanovsky wrote:
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);  
                                                               ^^^^^
Quoted 5 lines
Quoted 3 lines

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

Damn, good catch. It's not supposed to be required. I'll patch devlink.

When you set the port type to ethernet, you should have the net_device instance. Why wouldn't you?

Quoted 5 lines
Quoted 2 lines

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

#4 lore raw
Quoted 37 lines · Jiri Pirko
Quoted 36 lines · Sun, Sep 06, 2020 at 06:33:05PM CEST, [email protected] wrote:
Quoted 35 lines · On Sun, 6 Sep 2020 10:27:59 +0300 Leon Romanovsky wrote:
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);
                                                               ^^^^^
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

Damn, good catch. It's not supposed to be required. I'll patch devlink.

When you set the port type to ethernet, you should have the net_device instance. Why wouldn't you?

It is how mlx4 is implemented, see mlx4_dev_cap() function:
588 for (i = 1; i <= dev->caps.num_ports; ++i) {
589 dev->caps.port_type[i] = MLX4_PORT_TYPE_NONE;
....

The port type is being set to IB or ETH without relation to net_device, fixing it will require very major code rewrite for the stable driver that in maintenance mode.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

I would say that such a situation is corner case during the driver init and not an end of the world to see double netlink message.

Thanks

#5 lore raw
Quoted 47 lines · Mon, Sep 07, 2020 at 08:48:30AM CEST, [email protected] wrote:
Quoted 37 lines · Jiri Pirko
Quoted 36 lines · Sun, Sep 06, 2020 at 06:33:05PM CEST, [email protected] wrote:
Quoted 35 lines · On Sun, 6 Sep 2020 10:27:59 +0300 Leon Romanovsky wrote:
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);
                                                               ^^^^^
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

Damn, good catch. It's not supposed to be required. I'll patch devlink.

When you set the port type to ethernet, you should have the net_device instance. Why wouldn't you?

Quoted 8 lines

It is how mlx4 is implemented, see mlx4_dev_cap() function:
588 for (i = 1; i <= dev->caps.num_ports; ++i) {
589 dev->caps.port_type[i] = MLX4_PORT_TYPE_NONE;
....

The port type is being set to IB or ETH without relation to net_device, fixing it will require very major code rewrite for the stable driver that in maintenance mode.

Because the eth driver is not loaded, I see. The purpose of the WARN in devlink_port_type_eth_set is to prevent drivers from registering particular port without netdev/ibdev. That is what was repeatedly happening in the past as the driver developers didn't know they need to do it or were just lazy to do so.

I wonder if there is any possibility to do both...

Quoted 13 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

I would say that such a situation is corner case during the driver init and not an end of the world to see double netlink message.

Thanks

#6 lore raw
Quoted 57 lines · Jiri Pirko
Quoted 47 lines · Mon, Sep 07, 2020 at 08:48:30AM CEST, [email protected] wrote:
Quoted 37 lines · Jiri Pirko
Quoted 36 lines · Sun, Sep 06, 2020 at 06:33:05PM CEST, [email protected] wrote:
Quoted 35 lines · On Sun, 6 Sep 2020 10:27:59 +0300 Leon Romanovsky wrote:
Quoted 22 lines · Jakub Kicinski

Even tho mlx4_core registers the devlink ports, it's mlx4_en and mlx4_ib which set their type. In situations where one of the two is not built yet the machine has ports of given type we see the devlink warning from devlink_port_type_warn() trigger.

Having ports of a type not supported by the kernel may seem surprising, but it does occur in practice - when the unsupported port is not plugged in to a switch anyway users are more than happy not to see it (and potentially allocate any resources to it).

Set the type in mlx4_core if type-specific driver is not built.

---
 drivers/net/ethernet/mellanox/mlx4/main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx4/main.c +0 -0
index 258c7a96f269..70cf24ba71e4 100644

drivers/net/ethernet/mellanox/mlx4/main.c static int mlx4_init_port_info(struct mlx4_dev *dev, int port) · line 3040

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/main.c +7 -0
@@ -3031,6 +3031,17 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
 	if (err)
 		return err;
 
+	/* Ethernet and IB drivers will normally set the port type,
+	 * but if they are not built set the type now to prevent
+	 * devlink_port_type_warn() from firing.
+	 */
+	if (!IS_ENABLED(CONFIG_MLX4_EN) &&
+	    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+		devlink_port_type_eth_set(&info->devlink_port, NULL);
                                                               ^^^^^
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Won't it crash in devlink_port_type_eth_set()?
The first line there dereferences pointer.
7612 const struct net_device_ops *ops = netdev->netdev_ops;

Damn, good catch. It's not supposed to be required. I'll patch devlink.

When you set the port type to ethernet, you should have the net_device instance. Why wouldn't you?

Quoted 8 lines

It is how mlx4 is implemented, see mlx4_dev_cap() function:
588 for (i = 1; i <= dev->caps.num_ports; ++i) {
589 dev->caps.port_type[i] = MLX4_PORT_TYPE_NONE;
....

The port type is being set to IB or ETH without relation to net_device, fixing it will require very major code rewrite for the stable driver that in maintenance mode.

Quoted 7 lines

Because the eth driver is not loaded, I see. The purpose of the WARN in devlink_port_type_eth_set is to prevent drivers from registering particular port without netdev/ibdev. That is what was repeatedly happening in the past as the driver developers didn't know they need to do it or were just lazy to do so.

I wonder if there is any possibility to do both...

It is hard to say, hope that Jakub will take a look.

Thanks

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

I would say that such a situation is corner case during the driver init and not an end of the world to see double netlink message.

Thanks

author maintainer
#9 lore raw
In reply to Leon Romanovsky
Quoted 7 lines · On Mon, 7 Sep 2020 09:48:30 +0300 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

I would say that such a situation is corner case during the driver init and not an end of the world to see double netlink message.

Could you spell out your reasoning here? Are you concerned about out-of-tree drivers?

I don't see how adding IS_ENABLED() to the condition outweighs the benefit of not having duplicated netlink notifications.

#8 lore raw
In reply to Jakub Kicinski
Quoted 20 lines · Mon, Sep 07, 2020 at 06:34:01PM CEST, [email protected] wrote:
Quoted 11 lines · On Mon, 7 Sep 2020 09:19:39 +0200 Jiri Pirko wrote:
Quoted 3 lines

The port type is being set to IB or ETH without relation to net_device, fixing it will require very major code rewrite for the stable driver that in maintenance mode.

Because the eth driver is not loaded, I see. The purpose of the WARN in devlink_port_type_eth_set is to prevent drivers from registering particular port without netdev/ibdev. That is what was repeatedly happening in the past as the driver developers didn't know they need to do it or were just lazy to do so.

I wonder if there is any possibility to do both...

I think we have two options in this case:
- set type to eth without the netdev
- selectively mute the warning

I think the former is better, because we still want to see what the port type is. Perhaps we should add a:

Quoted 6 lines

dev_warn("devlink port type set without software interface

           reference, device type not supported by the kernel?");

That way people won't just pass NULL out of laziness, hopefully.

WDYT?

Okay. That sounds probably like the best option we have.

author maintainer
#7 lore raw
In reply to Jiri Pirko
Quoted 11 lines · On Mon, 7 Sep 2020 09:19:39 +0200 Jiri Pirko wrote:
Quoted 3 lines

The port type is being set to IB or ETH without relation to net_device, fixing it will require very major code rewrite for the stable driver that in maintenance mode.

Because the eth driver is not loaded, I see. The purpose of the WARN in devlink_port_type_eth_set is to prevent drivers from registering particular port without netdev/ibdev. That is what was repeatedly happening in the past as the driver developers didn't know they need to do it or were just lazy to do so.

I wonder if there is any possibility to do both...

I think we have two options in this case:
- set type to eth without the netdev
- selectively mute the warning

I think the former is better, because we still want to see what the port type is. Perhaps we should add a:

dev_warn("devlink port type set without software interface

            reference, device type not supported by the kernel?");

That way people won't just pass NULL out of laziness, hopefully.

WDYT?

#10 lore raw
In reply to Jakub Kicinski
Quoted 11 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 7 Sep 2020 09:48:30 +0300 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

And can we call to devlink_port_type_*_set() without IS_ENABLED() check?

It'll generate two netlink notifications - not the end of the world but also doesn't feel super clean.

I would say that such a situation is corner case during the driver init and not an end of the world to see double netlink message.

Could you spell out your reasoning here? Are you concerned about out-of-tree drivers?

Nothing fancy, I just didn't see users who compiled mlx4_core and used it without eth/ib.

The corner case is because this double netlink can be seen only during driver reload and only if port type wasn't set.

Quoted 3 lines

I don't see how adding IS_ENABLED() to the condition outweighs the benefit of not having duplicated netlink notifications.

Readability?

Anyway, it doesn't matter.

Thanks