Add support for active width and speed for RoCE

PATCH In review Leon Romanovsky 2 participants 6 messages Apr 20, 2017 – Apr 24, 2017 2 messages not in the archive

Subsystems (2)

InfiniBand Mellanox MLX5 IB driver

lore

Share & tools

lore

Leon Romanovsky

author

2017-04-20 17:53 UTC #0 lore raw

cover Add support for active width and speed for RoCE

Hi Doug,

This series from Noa replaces the current hard-coded active speed and width reported for RoCE in the mlx5 driver.

Two preliminary commits were needed for this change:
1. As a part of proper translation of the ib_port_spped enum, the IB_SPEED_HDR enum entry was added.
2. Set mlx5_query_roce_port's return value to void, since in case of an error, the properties reported back to the user are zeroed.

It is based on k.o/for-4.12 branch.

Thanks,
Noa and Leon.

Noa Osherovich (3):
IB/core: Add HDR speed enum
IB/mlx5: Set mlx5_query_roce_port's return value to void IB/mlx5: Add support for active_width and active_speed in RoCE

 drivers/infiniband/core/sysfs.c   |  4 ++
 drivers/infiniband/hw/mlx5/main.c | 89 +++++++++++++++++++++++++++++++++------
 include/rdma/ib_verbs.h           |  3 +-
 3 files changed, 82 insertions(+), 14 deletions(-)

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170420175333.7082-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

Leon Romanovsky

author

2017-04-20 17:53 UTC #2 lore raw

PATCH 1/3 IB/core: Add HDR speed enum

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add high data rate speed to the ib_port_speed enumeration.

Signed-off-by Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Eran Ben Elisha <eranbe-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/sysfs.c | 4 ++++
 include/rdma/ib_verbs.h         | 3 ++-
 2 files changed, 6 insertions(+), 1 deletion(-)

drivers/infiniband/core/sysfs.c +4 -0

index daadf3130c9f..b6317f220f7d 100644
@@ -253,6 +253,10 @@ static ssize_t rate_show(struct ib_port *p, struct port_attribute *unused,
 		speed = " EDR";
 		rate = 250;
 		break;
+	case IB_SPEED_HDR:
+		speed = " HDR";
+		rate = 500;
+		break;
 	case IB_SPEED_SDR:
 	default:		/* default to SDR for invalid rates */
 		rate = 25;

include/rdma/ib_verbs.h +2 -1

index 3a8e05894e9b..796a38609e1a 100644
@@ -431,7 +431,8 @@ enum ib_port_speed {
 	IB_SPEED_QDR	= 4,
 	IB_SPEED_FDR10	= 8,
 	IB_SPEED_FDR	= 16,
-	IB_SPEED_EDR	= 32
+	IB_SPEED_EDR	= 32,
+	IB_SPEED_HDR	= 64
 };
 
 /**

Signature · 6 lines

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Leon Romanovsky

author

2017-04-20 17:53 UTC #3 lore raw

PATCH 2/3 IB/mlx5: Set mlx5_query_roce_port's return value to void

In reply to a message not in the archive

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

In case of an error, the properties reported to user are zeroed out, so no need for a return value.

Signed-off-by Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/main.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

drivers/infiniband/hw/mlx5/main.c +10 ~~-10~~

index 4dc0a8785fe0..6ba8bbc5c74a 100644
@@ -165,15 +165,20 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device,
 	return ndev;
 }
 
-static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
-				struct ib_port_attr *props)
+static void mlx5_query_port_roce(struct ib_device *device, u8 port_num,
+				 struct ib_port_attr *props)
 {
 	struct mlx5_ib_dev *dev = to_mdev(device);
 	struct net_device *ndev, *upper;
 	enum ib_mtu ndev_ib_mtu;
 	u16 qkey_viol_cntr;
 
-	/* props being zeroed by the caller, avoid zeroing it here */
+	/* Getting netdev before filling out props so in case of an error it
+	 * will still be zeroed out.
+	 */
+	ndev = mlx5_ib_get_netdev(device, port_num);
+	if (!ndev)
+		return;
 
 	props->port_cap_flags  |= IB_PORT_CM_SUP;
 	props->port_cap_flags  |= IB_PORT_IP_BASED_GIDS;

@@ -189,10 +194,6 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 	mlx5_query_nic_vport_qkey_viol_cntr(dev->mdev, &qkey_viol_cntr);
 	props->qkey_viol_cntr = qkey_viol_cntr;
 
-	ndev = mlx5_ib_get_netdev(device, port_num);
-	if (!ndev)
-		return 0;
-
 	if (mlx5_lag_is_active(dev->mdev)) {
 		rcu_read_lock();
 		upper = netdev_master_upper_dev_get_rcu(ndev);

@@ -217,8 +218,6 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 
 	props->active_width	= IB_WIDTH_4X;  /* TODO */
 	props->active_speed	= IB_SPEED_QDR; /* TODO */
-
-	return 0;
 }
 
 static void ib_gid_to_mlx5_roce_addr(const union ib_gid *gid,

@@ -924,7 +923,8 @@ int mlx5_ib_query_port(struct ib_device *ibdev, u8 port,
 		return mlx5_query_hca_port(ibdev, port, props);
 
 	case MLX5_VPORT_ACCESS_METHOD_NIC:
-		return mlx5_query_port_roce(ibdev, port, props);
+		mlx5_query_port_roce(ibdev, port, props);
+		return 0;
 
 	default:
 		return -EINVAL;

Signature · 6 lines

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170420175333.7082-3-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

Leon Romanovsky

author

2017-04-21 02:30 UTC #5 lore raw

Re: [PATCH rdma-next 2/3] IB/mlx5: Set mlx5_query_roce_port's return value to void

Quoted 27 lines · Leon Romanovsky

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>
In case of an error, the properties reported to user are zeroed out, so no need for a return value.
Signed-off-by Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/hw/mlx5/main.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/main.c +2 -2
index 4dc0a8785fe0..6ba8bbc5c74a 100644
@@ -165,15 +165,20 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device,
 	return ndev;
 }
 
-static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
-				struct ib_port_attr *props)
+static void mlx5_query_port_roce(struct ib_device *device, u8 port_num,
+				 struct ib_port_attr *props)
 {
 	struct mlx5_ib_dev *dev = to_mdev(device);
 	struct net_device *ndev, *upper;
 	enum ib_mtu ndev_ib_mtu;

drivers/infiniband/hw/mlx5/main.c static struct net_device *mlx5_ib_get_netdev(struct ib_device *device, · line 182

Quoted 10 lines

drivers/infiniband/hw/mlx5/main.c +6 -1

 	u16 qkey_viol_cntr;
 
-	/* props being zeroed by the caller, avoid zeroing it here */
+	/* Getting netdev before filling out props so in case of an error it
+	 * will still be zeroed out.
+	 */
+	ndev = mlx5_ib_get_netdev(device, port_num);
+	if (!ndev)
+		return;

It breaks ib_write_bw when netdevice is not bounded to ib device. Need to get rid of this chunk. I'll resend.

Quoted 39 lines

drivers/infiniband/hw/mlx5/main.c +2 -8

 	props->port_cap_flags  |= IB_PORT_CM_SUP;
 	props->port_cap_flags  |= IB_PORT_IP_BASED_GIDS;

@@ -189,10 +194,6 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 	mlx5_query_nic_vport_qkey_viol_cntr(dev->mdev, &qkey_viol_cntr);
 	props->qkey_viol_cntr = qkey_viol_cntr;
 
-	ndev = mlx5_ib_get_netdev(device, port_num);
-	if (!ndev)
-		return 0;
-
 	if (mlx5_lag_is_active(dev->mdev)) {
 		rcu_read_lock();
 		upper = netdev_master_upper_dev_get_rcu(ndev);

@@ -217,8 +218,6 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 
 	props->active_width	= IB_WIDTH_4X;  /* TODO */
 	props->active_speed	= IB_SPEED_QDR; /* TODO */
-
-	return 0;
 }
 
 static void ib_gid_to_mlx5_roce_addr(const union ib_gid *gid,

@@ -924,7 +923,8 @@ int mlx5_ib_query_port(struct ib_device *ibdev, u8 port,
 		return mlx5_query_hca_port(ibdev, port, props);
 
 	case MLX5_VPORT_ACCESS_METHOD_NIC:
-		return mlx5_query_port_roce(ibdev, port, props);
+		mlx5_query_port_roce(ibdev, port, props);
+		return 0;
 
 	default:
 		return -EINVAL;
--

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Leon Romanovsky

author

2017-04-20 17:53 UTC #6 lore raw

PATCH 3/3 IB/mlx5: Add support for active_width and active_speed in RoCE

In reply to a message not in the archive

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add missing calculation and translation of active_width and active_speed for RoCE.

Fixes 3f89a643eb295 ('IB/mlx5: Extend query_device/port to ...')

Signed-off-by Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Eran Ben Elisha <eranbe-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/main.c | 73 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 68 insertions(+), 5 deletions(-)

drivers/infiniband/hw/mlx5/main.c +68 -5

index 6ba8bbc5c74a..0eaf769a34f5 100644
@@ -165,21 +165,87 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device,
 	return ndev;
 }
 
+static int translate_eth_proto_oper(u32 eth_proto_oper, u8 *active_speed,
+				    u8 *active_width)
+{
+	switch (eth_proto_oper) {
+	case MLX5E_PROT_MASK(MLX5E_1000BASE_CX_SGMII):
+	case MLX5E_PROT_MASK(MLX5E_1000BASE_KX):
+	case MLX5E_PROT_MASK(MLX5E_100BASE_TX):
+	case MLX5E_PROT_MASK(MLX5E_1000BASE_T):
+		*active_width = IB_WIDTH_1X;
+		*active_speed = IB_SPEED_SDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_T):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_CX4):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_KX4):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_KR):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_CR):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_SR):
+	case MLX5E_PROT_MASK(MLX5E_10GBASE_ER):
+		*active_width = IB_WIDTH_1X;
+		*active_speed = IB_SPEED_QDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_25GBASE_CR):
+	case MLX5E_PROT_MASK(MLX5E_25GBASE_KR):
+	case MLX5E_PROT_MASK(MLX5E_25GBASE_SR):
+		*active_width = IB_WIDTH_1X;
+		*active_speed = IB_SPEED_EDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_40GBASE_CR4):
+	case MLX5E_PROT_MASK(MLX5E_40GBASE_KR4):
+	case MLX5E_PROT_MASK(MLX5E_40GBASE_SR4):
+	case MLX5E_PROT_MASK(MLX5E_40GBASE_LR4):
+		*active_width = IB_WIDTH_4X;
+		*active_speed = IB_SPEED_QDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_50GBASE_CR2):
+	case MLX5E_PROT_MASK(MLX5E_50GBASE_KR2):
+	case MLX5E_PROT_MASK(MLX5E_50GBASE_SR2):
+		*active_width = IB_WIDTH_1X;
+		*active_speed = IB_SPEED_HDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_56GBASE_R4):
+		*active_width = IB_WIDTH_4X;
+		*active_speed = IB_SPEED_FDR;
+		break;
+	case MLX5E_PROT_MASK(MLX5E_100GBASE_CR4):
+	case MLX5E_PROT_MASK(MLX5E_100GBASE_SR4):
+	case MLX5E_PROT_MASK(MLX5E_100GBASE_KR4):
+	case MLX5E_PROT_MASK(MLX5E_100GBASE_LR4):
+		*active_width = IB_WIDTH_4X;
+		*active_speed = IB_SPEED_EDR;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static void mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 				 struct ib_port_attr *props)
 {
 	struct mlx5_ib_dev *dev = to_mdev(device);
+	struct mlx5_core_dev *mdev = dev->mdev;
 	struct net_device *ndev, *upper;
 	enum ib_mtu ndev_ib_mtu;
 	u16 qkey_viol_cntr;
+	u32 eth_prot_oper;
 
-	/* Getting netdev before filling out props so in case of an error it
-	 * will still be zeroed out.
+	/* Possible bad flows are checked before filling out props so in case
+	 * of an error it will still be zeroed out.
 	 */
 	ndev = mlx5_ib_get_netdev(device, port_num);
 	if (!ndev)
 		return;
 
+	if (mlx5_query_port_eth_proto_oper(mdev, &eth_prot_oper, port_num))
+		return;
+
+	translate_eth_proto_oper(eth_prot_oper, &props->active_speed,
+				 &props->active_width);
+
 	props->port_cap_flags  |= IB_PORT_CM_SUP;
 	props->port_cap_flags  |= IB_PORT_IP_BASED_GIDS;

@@ -215,9 +281,6 @@ static void mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 	dev_put(ndev);
 
 	props->active_mtu	= min(props->max_mtu, ndev_ib_mtu);
-
-	props->active_width	= IB_WIDTH_4X;  /* TODO */
-	props->active_speed	= IB_SPEED_QDR; /* TODO */
 }
 
 static void ib_gid_to_mlx5_roce_addr(const union ib_gid *gid,

Signature · 6 lines

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Doug Ledford

2017-04-24 16:30 UTC #7 lore raw

In reply to a message not in the archive

Quoted 21 lines · Leon Romanovsky

Hi Doug,
This series from Noa replaces the current hard-coded active speed and width reported for RoCE in the mlx5 driver.
Two preliminary commits were needed for this change:
1. As a part of proper translation of the ib_port_spped enum, the IB_SPEED_HDR enum entry was added.
2. Set mlx5_query_roce_port's return value to void, since in case of an error, the properties reported back to the user are zeroed.
It is based on k.o/for-4.12 branch.
Thanks,
Noa and Leon.
Noa Osherovich (3):
IB/core: Add HDR speed enum
IB/mlx5: Set mlx5_query_roce_port's return value to void IB/mlx5: Add support for active_width and active_speed in RoCE

Quoted 5 lines

drivers/infiniband/core/sysfs.c | 4 ++
drivers/infiniband/hw/mlx5/main.c | 89
+++++++++++++++++++++++++++++++++------
include/rdma/ib_verbs.h | 3 +-
 3 files changed, 82 insertions(+), 14 deletions(-)

I fixed up the second patch so it wouldn't break things any more, then series applied.

Signature · 9 lines

Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
   
Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html