Skip to contentSkip to conversation

RDMA/iw_cxgb4: disable delayed ack by default

PATCH In review Potnuri Bharat Teja 2 participants 10 messages Sep 9, 2020 – Sep 15, 2020

Subsystems (2)
lore
Share & tools
lore

Potnuri Bharat Teja

author maintainer
#0 lore raw

PATCH 1/1 RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +2 -2
index 1f288c73ccfc..8769e7aa097f 100644
@@ -77,9 +77,9 @@ static int enable_ecn;
 module_param(enable_ecn, int, 0644);
 MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");
 
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
Signature · 1 line
2.24.0
author maintainer
#1 lore raw

Missed rdma-next subject prefix. Resending.

-----Original Message-----
From: Potnuri Bharat Teja <[email protected]>
Sent: Wednesday, September 9, 2020 7:17 PM
To: [email protected]; [email protected]
Cc: [email protected]; Potnuri Bharat Teja <[email protected]>
Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +2 -3
index 1f288c73ccfc..8769e7aa097f 100644
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");
 
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
--

2.24.0

#2 lore raw
Quoted 21 lines · Potnuri Bharat Teja

Missed rdma-next subject prefix. Resending.

-----Original Message-----
From: Potnuri Bharat Teja <[email protected]>
Sent: Wednesday, September 9, 2020 7:17 PM
To: [email protected]; [email protected]
Cc: [email protected]; Potnuri Bharat Teja <[email protected]>
Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 6 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Thanks

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +0 -1
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
--

2.24.0

author maintainer
#3 lore raw
Quoted 15 lines
Quoted 15 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
Quoted 7 lines

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

Thanks,
Bharat.

Quoted 8 lines

Thanks

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +0 -1
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
--

2.24.0

#4 lore raw
Quoted 15 lines · Potnuri Bharat Teja
Quoted 15 lines
Quoted 15 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
Quoted 8 lines
Quoted 7 lines

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma-hw-modules.rules#L12

Thanks

Quoted 11 lines

Thanks,
Bharat.

Quoted 8 lines

Thanks

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +0 -1
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
--

2.24.0

author maintainer
#5 lore raw
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Potnuri Bharat Teja
Quoted 22 lines
Quoted 14 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 6 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Quoted 6 lines
Quoted 1 line

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

Sorry I didnt get it right earlier. Yes now if user wants delayed ack mode, user can issue "modprobe iw_cxgb4 dack_mode=1" or change it dynamically via /sys/module/<>/paramters.
Chelsio adapters for better performance in most cases needs delayed ack mode to be disabled. This change somehow got missed upstream all this while. In fewer explicit cases, user can enable delayed ack mode where it is needed.

Quoted 15 lines

https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma-hw-modules.rules#L12

Thanks

Quoted 11 lines

Thanks,
Bharat.

Quoted 8 lines

Thanks

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +0 -1
 uint c4iw_max_read_depth = 32;
 module_param(c4iw_max_read_depth, int, 0644);
--

2.24.0

#6 lore raw
Quoted 30 lines · Potnuri Bharat Teja
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Potnuri Bharat Teja
Quoted 22 lines
Quoted 14 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 6 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Quoted 6 lines
Quoted 1 line

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

Quoted 6 lines

Sorry I didnt get it right earlier. Yes now if user wants delayed ack mode, user can issue "modprobe iw_cxgb4 dack_mode=1" or change it dynamically via /sys/module/<>/paramters.
Chelsio adapters for better performance in most cases needs delayed ack mode to be disabled. This change somehow got missed upstream all this while. In fewer explicit cases, user can enable delayed ack mode where it is needed.

So why doesn't rdma-core have dack_mode=0 in kernel-boot scripts?

Thanks

author maintainer
#7 lore raw
Quoted 31 lines · Leon Romanovsky
Quoted 30 lines · Potnuri Bharat Teja
Quoted 29 lines · Leon Romanovsky
Quoted 15 lines · Potnuri Bharat Teja
Quoted 15 lines
Quoted 15 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
Quoted 8 lines
Quoted 7 lines

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 5 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

Quoted 9 lines
Quoted 6 lines

Sorry I didnt get it right earlier. Yes now if user wants delayed ack mode, user can issue "modprobe iw_cxgb4 dack_mode=1" or change it dynamically via /sys/module/<>/paramters.
Chelsio adapters for better performance in most cases needs delayed ack mode to be disabled. This change somehow got missed upstream all this while. In fewer explicit cases, user can enable delayed ack mode where it is needed.

So why doesn't rdma-core have dack_mode=0 in kernel-boot scripts?

Do you mean why dont I simply change rdma-hw-modules.rules instead?
I could do that if this is a user mode only setting. But this is needed for kernel mode too where in some cases rdma-core may not be installed.

Thanks.

#8 lore raw
Quoted 39 lines · Potnuri Bharat Teja
Quoted 38 lines · Leon Romanovsky
Quoted 36 lines · Potnuri Bharat Teja
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Potnuri Bharat Teja
Quoted 22 lines
Quoted 14 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 6 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Quoted 6 lines
Quoted 1 line

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

Sorry I didnt get it right earlier. Yes now if user wants delayed ack mode, user can issue "modprobe iw_cxgb4 dack_mode=1" or change it dynamically via /sys/module/<>/paramters.
Chelsio adapters for better performance in most cases needs delayed ack mode to be disabled. This change somehow got missed upstream all this while. In fewer explicit cases, user can enable delayed ack mode where it is needed.

Quoted 5 lines
Quoted 2 lines

So why doesn't rdma-core have dack_mode=0 in kernel-boot scripts?

Do you mean why dont I simply change rdma-hw-modules.rules instead?
I could do that if this is a user mode only setting. But this is needed for kernel mode too where in some cases rdma-core may not be installed.

No, I'm asking why if "dack_mode=0" is better and preferable way to operate, it was never set in rdma-core.

I'm not saying that your kernel patch is wrong, just trying to get sense on implications for the rdma-core.

Thanks

Quoted 3 lines

Thanks.

author maintainer
#9 lore raw
Quoted 40 lines · Leon Romanovsky
Quoted 39 lines · Potnuri Bharat Teja
Quoted 38 lines · Leon Romanovsky
Quoted 36 lines · Potnuri Bharat Teja
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Potnuri Bharat Teja
Quoted 22 lines
Quoted 14 lines

Subject: [PATCH] RDMA/iw_cxgb4: disable delayed ack by default

Receive side delayed ack mode is needed only for certain area networks/ connections. Therefore disable it by default.

---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +0 -0
index 1f288c73ccfc..8769e7aa097f 100644

drivers/infiniband/hw/cxgb4/cm.c static int enable_ecn; module_param(enable_ecn, int, 0644); MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)"); · line 80

Quoted 6 lines
drivers/infiniband/hw/cxgb4/cm.c +2 -2
@@ -77,9 +77,9 @@ static int enable_ecn;  module_param(enable_ecn, int, 0644);  MODULE_PARM_DESC(enable_ecn, "Enable ECN (default=0/disabled)");
 
-static int dack_mode = 1;
+static int dack_mode;
 module_param(dack_mode, int, 0644);
-MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=1)");
+MODULE_PARM_DESC(dack_mode, "Delayed ack mode (default=0)");

Are you sure that this doesn't break user scripts?

Quoted 6 lines
Quoted 1 line

Yes, I am sure. This does not interfere with user/kernel RDMA functionalities.

How is it possible?
Before this change user that did "modprobe iw_cxgb4" had delayed mode enabled, after this he will need to issue "modprobe iw_cxgb4 dack_mode=1"

Sorry I didnt get it right earlier. Yes now if user wants delayed ack mode, user can issue "modprobe iw_cxgb4 dack_mode=1" or change it dynamically via /sys/module/<>/paramters.
Chelsio adapters for better performance in most cases needs delayed ack mode to be disabled. This change somehow got missed upstream all this while. In fewer explicit cases, user can enable delayed ack mode where it is needed.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

So why doesn't rdma-core have dack_mode=0 in kernel-boot scripts?

Do you mean why dont I simply change rdma-hw-modules.rules instead?
I could do that if this is a user mode only setting. But this is needed for kernel mode too where in some cases rdma-core may not be installed.

No, I'm asking why if "dack_mode=0" is better and preferable way to operate, it was never set in rdma-core.

Found this only recently that it was wrongly enabled by default which affects severely the performance of some of our adapters with low HW resources for larger IO sizes. I could have mentioned this in commit description but I thought it's a trivial change code wise.

Quoted 3 lines

I'm not saying that your kernel patch is wrong, just trying to get sense on implications for the rdma-core.

ok. yes rdma-hw-modules.rules provides a cleaner way. I am still used to setting module params via /sys/module/<>/parameters/*.

Thanks for the review.