Skip to contentSkip to conversation

RDMA/siw: Fix setting active_{speed, width} attributes

PATH for-next Kamal Heib 3 participants 8 messages Feb 13, 2020 – Feb 17, 2020

Subsystems (2)
lore
Share & tools
lore

Kamal Heib

author
#0 lore raw

Make sure to set the active_{speed, width} attributes to avoid reporting the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")
---
 drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw_verbs.c +4 -3
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev, u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
 	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 
 int siw_get_port_immutable(struct ib_device *base_dev, u8 port,
Signature · 1 line
2.21.1
#1 lore raw

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 49 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>, "Kamal Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw_verbs.c +3 -2
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev,
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 
Quoted 5 lines

int siw_get_port_immutable(struct ib_device *base_dev, u8 port,

Signature · 1 line
2.21.1

Hi Kamal,
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (iw_cxgb4), but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Another remark: It has been siw folklore to name integer return values 'rv', and not 'rc'. I never liked 'return code'. It's a value in principle, sometimes it's interpreted as a code though, as in your case.

Many thanks!
Bernard.

author
#2 lore raw
Quoted 51 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 49 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>, "Kamal Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw_verbs.c +3 -2
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev,
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 
Quoted 6 lines
Quoted 5 lines

int siw_get_port_immutable(struct ib_device *base_dev, u8 port,

Signature · 1 line
2.21.1

Hi Kamal,

Hi Bernard,

Quoted 4 lines

Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

Sure :)

Quoted 10 lines

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (iw_cxgb4), but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

Quoted 7 lines

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying net device that used for both iw_cxgb4 and siw?

Quoted 6 lines

Another remark: It has been siw folklore to name integer return values 'rv', and not 'rc'. I never liked 'return code'. It's a value in principle, sometimes it's interpreted as a code though, as in your case.

Sure, I'll fix it in v2.

Quoted 4 lines

Many thanks!
Bernard.

Thanks,
Kamal

#3 lore raw

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 94 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 7 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 5 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,

"Kamal

Quoted 14 lines
Quoted 14 lines

Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 6 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
Quoted 8 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device *base_dev,

Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2 -2
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev,

u8

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 
Quoted 6 lines
Quoted 5 lines

int siw_get_port_immutable(struct ib_device *base_dev, u8 port,

Signature · 1 line
2.21.1

Hi Kamal,

Hi Bernard,

Quoted 4 lines

Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

Sure :)

Quoted 15 lines

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (iw_cxgb4), but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Quoted 7 lines
Quoted 2 lines

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying
net device that used for both iw_cxgb4 and siw?

H Kamal,

Sure! Speed looks correct, and its also what I get at maximum:

[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: None
Speed: 40000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes

author
#4 lore raw
Quoted 125 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 94 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 7 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 5 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,

"Kamal

Quoted 14 lines
Quoted 14 lines

Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 6 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
Quoted 8 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device *base_dev,

Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2 -2
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev,

u8

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 
Quoted 6 lines
Quoted 5 lines

int siw_get_port_immutable(struct ib_device *base_dev, u8 port,

Signature · 1 line
2.21.1

Hi Kamal,

Hi Bernard,

Quoted 4 lines

Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

Sure :)

Quoted 15 lines

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (iw_cxgb4), but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Quoted 7 lines
Quoted 2 lines

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying
net device that used for both iw_cxgb4 and siw?

H Kamal,

Sure! Speed looks correct, and its also what I get at maximum:

[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/s
Duplex: Full

Quoted 9 lines

Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes

Hi Bernard,

Well, this is the expected value for 40GbE, Because the reported value is 4X aggregation of FDR10. For more info please the table of speeds under [1].

[1] - https://en.wikipedia.org/wiki/InfiniBand

Thanks,
Kamal

#5 lore raw

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 157 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 48 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 46 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 7 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 5 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,

"Kamal

Quoted 14 lines
Quoted 14 lines

Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 6 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2 -2
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 10 lines
Quoted 10 lines

u8

Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 

int siw_get_port_immutable(struct ib_device *base_dev, u8

port,

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Signature · 1 line
2.21.1

Hi Kamal,

Hi Bernard,

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

Sure :)

Quoted 1 line

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4

driver).

Quoted 1 line
Quoted 1 line
Quoted 1 line

Works in principle, but reported numbers are off. I am not

saying

Quoted 1 line
Quoted 1 line
Quoted 1 line

I would get right numbers when using Chelsio HW iWarp

(iw_cxgb4),

Quoted 40 lines
Quoted 12 lines
Quoted 12 lines

but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Quoted 7 lines
Quoted 2 lines

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying
net device that used for both iw_cxgb4 and siw?

H Kamal,

Sure! Speed looks correct, and its also what I get at maximum:

[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/s

Quoted 8 lines

Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup

rx_err tx_err

Quoted 2 lines

Link detected: yes

Hi Bernard,

Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table of speeds under [1].

Quoted 9 lines

[1] -
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org
_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1PP 1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0jk YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=

Thanks,
Kamal

Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by ibv_devinfio (and somehow strange what the iw_cxgb4 makes out of it).

Many thanks for the info,
Bernard.

#6 lore raw
Quoted 165 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 163 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 48 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 46 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 7 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 5 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,

"Kamal

Quoted 14 lines
Quoted 14 lines

Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 6 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2 -2
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 10 lines
Quoted 10 lines

u8

Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 

int siw_get_port_immutable(struct ib_device *base_dev, u8

port,

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Signature · 1 line
2.21.1

Hi Kamal,

Hi Bernard,

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.

Sure :)

Quoted 1 line

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4

driver).

Quoted 1 line
Quoted 1 line
Quoted 1 line

Works in principle, but reported numbers are off. I am not

saying

Quoted 1 line
Quoted 1 line
Quoted 1 line

I would get right numbers when using Chelsio HW iWarp

(iw_cxgb4),

Quoted 40 lines
Quoted 12 lines
Quoted 12 lines

but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Quoted 7 lines
Quoted 2 lines

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying
net device that used for both iw_cxgb4 and siw?

H Kamal,

Sure! Speed looks correct, and its also what I get at maximum:

[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/s

Quoted 8 lines

Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup

rx_err tx_err

Quoted 2 lines

Link detected: yes

Hi Bernard,

Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table of speeds under [1].

[1] -
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org
_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1PP 1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0jk YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=

Quoted 6 lines
Quoted 3 lines

Thanks,
Kamal

Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by ibv_devinfio (and somehow strange what the iw_cxgb4 makes out of it).

Hi Bernard,
Are you running siw and iw_cxgb4 on the same port or different ports? if not same, Is the port running iw_cxgb4 connected with 100G cable? If 40G cable is conencted to iw_cxgb4 as well then its reporting wrong.

Thanks.

Quoted 4 lines

Many thanks for the info,
Bernard.

#7 lore raw

-----"Potnuri Bharat Teja" <[email protected]> wrote: -----

Quoted 192 lines

To: "Bernard Metzler" <[email protected]>
From: "Potnuri Bharat Teja" <[email protected]>
Date: 02/17/2020 03:51PM
Cc: "Kamal Heib" <[email protected]>, "[email protected]" <[email protected]>, "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 17 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 15 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Quoted 6 lines · Bernard Metzler

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 4 lines

To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe"

<[email protected]>,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

"Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

On Thu, Feb 13, 2020 at 01:59:30PM +0000, Bernard Metzler

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----"Kamal Heib" <[email protected]> wrote: -----

Quoted 5 lines

To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler"

<[email protected]>,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

"Kamal

Quoted 5 lines
Quoted 5 lines

Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributes

Make sure to set the active_{speed, width} attributes to

avoid

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 9 lines

reporting
the same values regardless of the underlying device.

Fixes 303ae1cdfdf7 ("rdma/siw: application interface")

---
drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 6 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index 73485d0da907..b1aaec912edb 100644
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
 u8 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	int rc;
 
 	memset(attr, 0, sizeof(*attr));
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device

Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2 -2
-	attr->active_speed = 2;
-	attr->active_width = 2;
+	rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
+			 &attr->active_width);
 	attr->gid_tbl_len = 1;
 	attr->max_msg_sz = -1;
	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device

*base_dev,

Quoted 10 lines
Quoted 10 lines

u8

Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
 port,
 	 * attr->subnet_timeout = 0;
 	 * attr->init_type_repy = 0;
 	 */
-	return 0;
+	return rc;
 }
 

int siw_get_port_immutable(struct ib_device *base_dev, u8

port,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Signature · 1 line
2.21.1
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Hi Kamal,

Hi Bernard,

Quoted 1 line

Many thanks for looking after this! So there definitely seem

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

be applications which are taking care of those values. So,

good

Quoted 60 lines
Quoted 60 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

to get my obvious laziness fixed.

Sure :)

Quoted 1 line

I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4

driver).

Quoted 1 line
Quoted 1 line
Quoted 1 line

Works in principle, but reported numbers are off. I am not

saying

Quoted 1 line
Quoted 1 line
Quoted 1 line

I would get right numbers when using Chelsio HW iWarp

(iw_cxgb4),

Quoted 40 lines
Quoted 12 lines
Quoted 12 lines

but it's closer to reality (using ibv_devinfo <ibname> -vv)

iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)

siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)

Quoted 7 lines
Quoted 2 lines

Any idea how we can improve that, maybe coming even close to reality (40Gbs)?

Could you please share the output of ethtool <if_name> for the underlying
net device that used for both iw_cxgb4 and siw?

H Kamal,

Sure! Speed looks correct, and its also what I get at maximum:

[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/s

Quoted 8 lines

Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup

rx_err tx_err

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Link detected: yes

Hi Bernard,

Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table of

speeds

Quoted 5 lines
Quoted 3 lines

under [1].

[1] -

https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.or

g

Quoted 2 lines

_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1P

P

Quoted 2 lines

1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0j

k

Quoted 5 lines
Quoted 5 lines

YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=

Thanks,
Kamal

Quoted 9 lines
Quoted 3 lines

Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by ibv_devinfio (and somehow strange what the iw_cxgb4 makes out of it).

Hi Bernard,
Are you running siw and iw_cxgb4 on the same port or different ports? if not same, Is the port running iw_cxgb4 connected with 100G cable? If 40G cable is conencted to iw_cxgb4 as well then its reporting wrong.

This is a single port 40Gbs setup. All my fault:
For iw_cxgb4, I looked at the reported values for the second port, which is not physically connected but reported PORT_DOWN and 4 x 25Gbs.
The connected port 1 has 4 x 10Gbs for both siw and iw_cxgb4. Sorry about the confusion!

So its all good.

Thanks
Bernard.