Skip to contentSkip to conversation

RDMA/mana_ib: Set correct device into ib

PATCH Applied Konstantin Taranov 4 participants 4 messages Jul 11, 2024 – Jul 14, 2024

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 RDMA/mana_ib: Set correct device into ib

From: Konstantin Taranov <[email protected]>

Add mana_get_primary_netdev_rcu helper to get a primary netdevice for a given port. When mana is used with netvsc, the VF netdev is controlled by an upper netvsc device. In a baremetal case, the VF netdev is the primary device.

Use the mana_get_primary_netdev_rcu() helper in the mana_ib to get the correct device for querying network states.

Fixes 8b184e4f1c32 ("RDMA/mana_ib: Enable RoCE on port 1")

---
I would appreciate if I could get Acks on it from:
* netvsc maintainers (e.g., Haiyang)
* net maintainers (e.g., Jakub, David, Eric, Paolo)

v1->v2:
Leon Romanovsky asked to make a helper in the net/mana and get acks from net maintainers.
v2->v3:
Added warn on rcu lock not held.
Use the word "primary" instead of "master"
Merged two commits into one and submitted to rdma-next

 drivers/infiniband/hw/mana/device.c           | 16 ++++++++--------
 drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++++
 include/net/mana/mana.h                       |  2 ++
 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/infiniband/hw/mana/device.c +8 -8
index b07a8e2e838f..7ac01918ef7c 100644
@@ -56,7 +56,7 @@ static int mana_ib_probe(struct auxiliary_device *adev,
 {
 	struct mana_adev *madev = container_of(adev, struct mana_adev, adev);
 	struct gdma_dev *mdev = madev->mdev;
-	struct net_device *upper_ndev;
+	struct net_device *ndev;
 	struct mana_context *mc;
 	struct mana_ib_dev *dev;
 	u8 mac_addr[ETH_ALEN];
@@ -84,17 +84,17 @@ static int mana_ib_probe(struct auxiliary_device *adev,
 	dev->ib_dev.num_comp_vectors = mdev->gdma_context->max_num_queues;
 	dev->ib_dev.dev.parent = mdev->gdma_context->dev;
 
-	rcu_read_lock(); /* required to get upper dev */
-	upper_ndev = netdev_master_upper_dev_get_rcu(mc->ports[0]);
-	if (!upper_ndev) {
+	rcu_read_lock(); /* required to get primary netdev */
+	ndev = mana_get_primary_netdev_rcu(mc, 0);
+	if (!ndev) {
 		rcu_read_unlock();
 		ret = -ENODEV;
-		ibdev_err(&dev->ib_dev, "Failed to get master netdev");
+		ibdev_err(&dev->ib_dev, "Failed to get netdev for IB port 1");
 		goto free_ib_device;
 	}
-	ether_addr_copy(mac_addr, upper_ndev->dev_addr);
-	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, upper_ndev->dev_addr);
-	ret = ib_device_set_netdev(&dev->ib_dev, upper_ndev, 1);
+	ether_addr_copy(mac_addr, ndev->dev_addr);
+	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, ndev->dev_addr);
+	ret = ib_device_set_netdev(&dev->ib_dev, ndev, 1);
 	rcu_read_unlock();
 	if (ret) {
 		ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret);
drivers/net/ethernet/microsoft/mana/mana_en.c +19 -0
index b89ad4afd66e..68c2bea2c022 100644
@@ -3007,3 +3007,22 @@ void mana_remove(struct gdma_dev *gd, bool suspending)
 	gd->gdma_context = NULL;
 	kfree(ac);
 }
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index)
+{
+	struct net_device *ndev;
+
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+			 "Taking primary netdev without holding the RCU read lock");
+	if (port_index >= ac->num_ports)
+		return NULL;
+
+	/* When mana is used in netvsc, the upper netdevice should be returned. */
+	if (ac->ports[port_index]->flags & IFF_SLAVE)
+		ndev = netdev_master_upper_dev_get_rcu(ac->ports[port_index]);
+	else
+		ndev = ac->ports[port_index];
+
+	return ndev;
+}
+EXPORT_SYMBOL_NS(mana_get_primary_netdev_rcu, NET_MANA);
include/net/mana/mana.h +2 -0
index 59823901b74f..f9b4b0dcb69f 100644
@@ -797,4 +797,6 @@ void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
 int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
 		   u32 doorbell_pg_id);
 void mana_uncfg_vport(struct mana_port_context *apc);
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index);
 #endif /* _MANA_H */
Signature · 1 line
2.43.0
Reviewed-by on v3 1/1
#1 lore raw
Quoted 8 lines

Subject: [PATCH rdma-next v3 1/1] RDMA/mana_ib: Set correct device into ib

From: Konstantin Taranov <[email protected]>

Add mana_get_primary_netdev_rcu helper to get a primary netdevice for a given port. When mana is used with netvsc, the VF netdev is controlled by an upper netvsc device. In a baremetal case, the VF netdev is the primary device.

Quoted 5 lines

Use the mana_get_primary_netdev_rcu() helper in the mana_ib to get the correct device for querying network states.

Fixes 8b184e4f1c32 ("RDMA/mana_ib: Enable RoCE on port 1")
Quoted 106 lines

---
I would appreciate if I could get Acks on it from:
* netvsc maintainers (e.g., Haiyang)
* net maintainers (e.g., Jakub, David, Eric, Paolo)

v1->v2:
Leon Romanovsky asked to make a helper in the net/mana and get acks from net maintainers.
v2->v3:
Added warn on rcu lock not held.
Use the word "primary" instead of "master"
Merged two commits into one and submitted to rdma-next

 drivers/infiniband/hw/mana/device.c           | 16 ++++++++--------
 drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++++
 include/net/mana/mana.h                       |  2 ++
 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/infiniband/hw/mana/device.c +8 -8
index b07a8e2e838f..7ac01918ef7c 100644
@@ -56,7 +56,7 @@ static int mana_ib_probe(struct auxiliary_device *adev,  {
 	struct mana_adev *madev = container_of(adev, struct mana_adev,
 adev);
 	struct gdma_dev *mdev = madev->mdev;
-	struct net_device *upper_ndev;
+	struct net_device *ndev;
 	struct mana_context *mc;
 	struct mana_ib_dev *dev;
 	u8 mac_addr[ETH_ALEN];
@@ -84,17 +84,17 @@ static int mana_ib_probe(struct auxiliary_device *adev,
 	dev->ib_dev.num_comp_vectors = mdev->gdma_context-
 >max_num_queues;
 	dev->ib_dev.dev.parent = mdev->gdma_context->dev;
 
-	rcu_read_lock(); /* required to get upper dev */
-	upper_ndev = netdev_master_upper_dev_get_rcu(mc->ports[0]);
-	if (!upper_ndev) {
+	rcu_read_lock(); /* required to get primary netdev */
+	ndev = mana_get_primary_netdev_rcu(mc, 0);
+	if (!ndev) {
 		rcu_read_unlock();
 		ret = -ENODEV;
-		ibdev_err(&dev->ib_dev, "Failed to get master netdev");
+		ibdev_err(&dev->ib_dev, "Failed to get netdev for IB port 1");
 		goto free_ib_device;
 	}
-	ether_addr_copy(mac_addr, upper_ndev->dev_addr);
-	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, upper_ndev-
 >dev_addr);
-	ret = ib_device_set_netdev(&dev->ib_dev, upper_ndev, 1);
+	ether_addr_copy(mac_addr, ndev->dev_addr);
+	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, ndev->dev_addr);
+	ret = ib_device_set_netdev(&dev->ib_dev, ndev, 1);
 	rcu_read_unlock();
drivers/infiniband/hw/mana/device.c +0 -0
 	if (ret) {
 		ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret);
drivers/net/ethernet/microsoft/mana/mana_en.c +7 -0
index b89ad4afd66e..68c2bea2c022 100644
@@ -3007,3 +3007,22 @@ void mana_remove(struct gdma_dev *gd, bool
 suspending)
 	gd->gdma_context = NULL;
 	kfree(ac);
 }
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac,
+u32 port_index) {
+	struct net_device *ndev;
+
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+			 "Taking primary netdev without holding the RCU read

lock");

drivers/net/ethernet/microsoft/mana/mana_en.c +4 -0
+	if (port_index >= ac->num_ports)
+		return NULL;
+
+	/* When mana is used in netvsc, the upper netdevice should be returned.

*/

drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
+	if (ac->ports[port_index]->flags & IFF_SLAVE)
+		ndev = netdev_master_upper_dev_get_rcu(ac-
Quoted 1 line

ports[port_index]);

drivers/net/ethernet/microsoft/mana/mana_en.c +6 -0
+	else
+		ndev = ac->ports[port_index];
+
+	return ndev;
+}
+EXPORT_SYMBOL_NS(mana_get_primary_netdev_rcu, NET_MANA);
include/net/mana/mana.h index renamed from include/net/mana/mana.h +0 -0

59823901b74f..f9b4b0dcb69f 100644

include/net/mana/mana.h +3 -1
@@ -797,4 +797,6 @@ void mana_destroy_wq_obj(struct mana_port_context
 *apc, u32 wq_type,  int mana_cfg_vport(struct mana_port_context *apc, u32
 protection_dom_id,
 		   u32 doorbell_pg_id);
 void mana_uncfg_vport(struct mana_port_context *apc);
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac,
+u32 port_index);
 #endif /* _MANA_H */
--

2.43.0

Reviewed-by on v3 1/1
#2 lore raw
In reply to Konstantin Taranov
Quoted 74 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Add mana_get_primary_netdev_rcu helper to get a primary netdevice for a given port. When mana is used with netvsc, the VF netdev is controlled by an upper netvsc device. In a baremetal case, the VF netdev is the primary device.

Use the mana_get_primary_netdev_rcu() helper in the mana_ib to get the correct device for querying network states.

Fixes 8b184e4f1c32 ("RDMA/mana_ib: Enable RoCE on port 1")

---
I would appreciate if I could get Acks on it from:
* netvsc maintainers (e.g., Haiyang)
* net maintainers (e.g., Jakub, David, Eric, Paolo)

v1->v2:
Leon Romanovsky asked to make a helper in the net/mana and get acks from net maintainers.
v2->v3:
Added warn on rcu lock not held.
Use the word "primary" instead of "master"
Merged two commits into one and submitted to rdma-next

  drivers/infiniband/hw/mana/device.c           | 16 ++++++++--------
  drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++++
  include/net/mana/mana.h                       |  2 ++
  3 files changed, 29 insertions(+), 8 deletions(-)
drivers/infiniband/hw/mana/device.c +8 -8
index b07a8e2e838f..7ac01918ef7c 100644
@@ -56,7 +56,7 @@ static int mana_ib_probe(struct auxiliary_device *adev,
  {
  	struct mana_adev *madev = container_of(adev, struct mana_adev, adev);
  	struct gdma_dev *mdev = madev->mdev;
-	struct net_device *upper_ndev;
+	struct net_device *ndev;
  	struct mana_context *mc;
  	struct mana_ib_dev *dev;
  	u8 mac_addr[ETH_ALEN];
@@ -84,17 +84,17 @@ static int mana_ib_probe(struct auxiliary_device *adev,
  	dev->ib_dev.num_comp_vectors = mdev->gdma_context->max_num_queues;
  	dev->ib_dev.dev.parent = mdev->gdma_context->dev;
 
-	rcu_read_lock(); /* required to get upper dev */
-	upper_ndev = netdev_master_upper_dev_get_rcu(mc->ports[0]);
-	if (!upper_ndev) {
+	rcu_read_lock(); /* required to get primary netdev */
+	ndev = mana_get_primary_netdev_rcu(mc, 0);
+	if (!ndev) {
  		rcu_read_unlock();
  		ret = -ENODEV;
-		ibdev_err(&dev->ib_dev, "Failed to get master netdev");
+		ibdev_err(&dev->ib_dev, "Failed to get netdev for IB port 1");
  		goto free_ib_device;
  	}
-	ether_addr_copy(mac_addr, upper_ndev->dev_addr);
-	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, upper_ndev->dev_addr);
-	ret = ib_device_set_netdev(&dev->ib_dev, upper_ndev, 1);
+	ether_addr_copy(mac_addr, ndev->dev_addr);
+	addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, ndev->dev_addr);
+	ret = ib_device_set_netdev(&dev->ib_dev, ndev, 1);
  	rcu_read_unlock();
  	if (ret) {
  		ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret);
drivers/net/ethernet/microsoft/mana/mana_en.c +0 -0
index b89ad4afd66e..68c2bea2c022 100644

drivers/net/ethernet/microsoft/mana/mana_en.c void mana_remove(struct gdma_dev *gd, bool suspending) · line 3016

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +7 -0
@@ -3007,3 +3007,22 @@ void mana_remove(struct gdma_dev *gd, bool suspending)
  	gd->gdma_context = NULL;
  	kfree(ac);
  }
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index)
+{
+	struct net_device *ndev;
+
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+			 "Taking primary netdev without holding the RCU read lock");

Thanks a lot.

Zhu Yanjun

Quoted 23 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +12 -0
+	if (port_index >= ac->num_ports)
+		return NULL;
+
+	/* When mana is used in netvsc, the upper netdevice should be returned. */
+	if (ac->ports[port_index]->flags & IFF_SLAVE)
+		ndev = netdev_master_upper_dev_get_rcu(ac->ports[port_index]);
+	else
+		ndev = ac->ports[port_index];
+
+	return ndev;
+}
+EXPORT_SYMBOL_NS(mana_get_primary_netdev_rcu, NET_MANA);
include/net/mana/mana.h +2 -0
index 59823901b74f..f9b4b0dcb69f 100644
@@ -797,4 +797,6 @@ void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
  int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
  		   u32 doorbell_pg_id);
  void mana_uncfg_vport(struct mana_port_context *apc);
+
+struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index);
  #endif /* _MANA_H */
#3 lore raw
In reply to Konstantin Taranov
Quoted 10 lines · Konstantin Taranov

Add mana_get_primary_netdev_rcu helper to get a primary netdevice for a given port. When mana is used with netvsc, the VF netdev is controlled by an upper netvsc device. In a baremetal case, the VF netdev is the primary device.

Use the mana_get_primary_netdev_rcu() helper in the mana_ib to get the correct device for querying network states.

[...]

Applied, thanks!

[1/1] RDMA/mana_ib: Set correct device into ib

      https://git.kernel.org/rdma/rdma/c/1df03a4b44146c

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>