Skip to contentSkip to conversation

RDMA/bnxt_re: Congestion control settings using debugfs hook

PATCH Applied Selvin Xavier 4 participants 11 messages Jan 19, 2025 – Feb 8, 2025

Subsystems (2)
lore
Share & tools
lore

Selvin Xavier

author maintainer
#0 lore raw

PATCH v2 1/1 RDMA/bnxt_re: Congestion control settings using debugfs hook

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Currently supporting only GEN 0 version of the parameters. Reading these files queries the firmware and report the values currently programmed. Writing to the files sends commands that update the congestion control settings

---
v1 -> v2:
Addressed Leon's comments
- rename debugfs file "g" to "run_avg_weight_g"
- Fix the indentation errors
- Remove the unnecessary error message during the read entry point
- Fix the return value

 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/debugfs.c | 212 +++++++++++++++++++++++++++++++-
 drivers/infiniband/hw/bnxt_re/debugfs.h |  15 +++
 3 files changed, 228 insertions(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
index b91a85a..b33b04e 100644
@@ -232,6 +232,8 @@ struct bnxt_re_dev {
 	unsigned long			event_bitmap;
 	struct bnxt_qplib_cc_param	cc_param;
 	struct workqueue_struct		*dcb_wq;
+	struct dentry                   *cc_config;
+	struct bnxt_re_dbg_cc_config_params *cc_config_params;
 };
 
 #define to_bnxt_re_dev(ptr, member)	\
drivers/infiniband/hw/bnxt_re/debugfs.c +211 -1

236 lines not on this page · raw message

drivers/infiniband/hw/bnxt_re/debugfs.h +15 -0
index cd3be0a9..3374097 100644
@@ -18,4 +18,19 @@ void bnxt_re_debugfs_rem_pdev(struct bnxt_re_dev *rdev);
 void bnxt_re_register_debugfs(void);
 void bnxt_re_unregister_debugfs(void);
 
+#define CC_CONFIG_GEN_EXT(x, y)	(((x) << 16) | (y))
+#define CC_CONFIG_GEN0_EXT0  CC_CONFIG_GEN_EXT(0, 0)
+
+#define BNXT_RE_CC_PARAM_GEN0	__ffs(CMDQ_MODIFY_ROCE_CC_MODIFY_MASK_INACTIVITY_CP)
+
+struct bnxt_re_cc_param {
+	struct bnxt_re_dev *rdev;
+	struct dentry *dentry;
+	u32 offset;
+	u8 cc_gen;
+};
+
+struct bnxt_re_dbg_cc_config_params {
+	struct bnxt_re_cc_param        gen0_parms[BNXT_RE_CC_PARAM_GEN0];
+};
 #endif
Signature · 1 line
2.5.5
#1 lore raw
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Should something like this be in debugfs though?

bnxt_qplib_modify_cc() is just sending a firmware command, seems like this should belong to fwctl?

Additionally there may be interest in some common way to control CC for RDMA..

Jason

author maintainer
#2 lore raw
Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Should something like this be in debugfs though?

Since these are Broadcom specific parameters, i thought its better to be under debugfs. Also I took the reference of a similar implementation in mlx5.

Quoted 3 lines

bnxt_qplib_modify_cc() is just sending a firmware command, seems like this should belong to fwctl?

Agree. We can move to this model once fwctl is accepted. For now, it is important for us to support our customers with an immediate solution. Customers are asking for this support.

Quoted 3 lines

Additionally there may be interest in some common way to control CC for RDMA..

Do you think there are common parameters for multiple vendors here? I think enable/disable is an option.

Quoted 2 lines

Jason

#3 lore raw
Quoted 11 lines · Selvin Xavier
Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Should something like this be in debugfs though?

Since these are Broadcom specific parameters, i thought its better to be under debugfs. Also I took the reference of a similar implementation in mlx5.

debugfs is disabled in a lot of deployments, it is a big part of why we are doing fwctl. If you know it works for you cases, debugfs is pretty open ended..

Quoted 5 lines
Quoted 2 lines

bnxt_qplib_modify_cc() is just sending a firmware command, seems like this should belong to fwctl?

Agree. We can move to this model once fwctl is accepted. For now, it is important for us to support our customers with an immediate solution. Customers are asking for this support.

Well, fwctl can be accepted when you guys come through with an implementation :)

Quoted 5 lines
Quoted 2 lines

Additionally there may be interest in some common way to control CC for RDMA..

Do you think there are common parameters for multiple vendors here? I think enable/disable is an option.

I haven't seen much commonality here, every site seems to have their own totally different stuff right now.

Jason

author maintainer
#4 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 7 lines · Selvin Xavier
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Should something like this be in debugfs though?

Since these are Broadcom specific parameters, i thought its better to be under debugfs. Also I took the reference of a similar implementation in mlx5.

debugfs is disabled in a lot of deployments, it is a big part of why we are doing fwctl. If you know it works for you cases, debugfs is pretty open ended..

The main use case for this debugfs support is for evaluation customers and the tuning for their network. So debugfs should be okay.

Quoted 20 lines
Quoted 6 lines
Quoted 3 lines

bnxt_qplib_modify_cc() is just sending a firmware command, seems like this should belong to fwctl?

Agree. We can move to this model once fwctl is accepted. For now, it is important for us to support our customers with an immediate solution. Customers are asking for this support.

Well, fwctl can be accepted when you guys come through with an implementation :)

Quoted 5 lines
Quoted 2 lines

Additionally there may be interest in some common way to control CC for RDMA..

Do you think there are common parameters for multiple vendors here? I think enable/disable is an option.

I haven't seen much commonality here, every site seems to have their own totally different stuff right now.

Jason

#5 lore raw
Quoted 15 lines · Selvin Xavier
Quoted 14 lines · Jason Gunthorpe
Quoted 11 lines · Selvin Xavier
Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Should something like this be in debugfs though?

Since these are Broadcom specific parameters, i thought its better to be under debugfs. Also I took the reference of a similar implementation in mlx5.

Quoted 5 lines
Quoted 3 lines

debugfs is disabled in a lot of deployments, it is a big part of why we are doing fwctl. If you know it works for you cases, debugfs is pretty open ended..

The main use case for this debugfs support is for evaluation customers and the tuning for their network. So debugfs should be okay.

In my experience it makes no difference, if the customer is using secure boot then they are always using secure boot expect in small lab systems perhaps.

Are you certain this is useful not just "should be okay" ?

Jason

author maintainer
#6 lore raw
Quoted 20 lines · Jason Gunthorpe
Quoted 18 lines · Selvin Xavier
Quoted 17 lines · Jason Gunthorpe
Quoted 11 lines · Selvin Xavier
Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Should something like this be in debugfs though?

Since these are Broadcom specific parameters, i thought its better to be under debugfs. Also I took the reference of a similar implementation in mlx5.

debugfs is disabled in a lot of deployments, it is a big part of why we are doing fwctl. If you know it works for you cases, debugfs is pretty open ended..

Quoted 8 lines
Quoted 2 lines

The main use case for this debugfs support is for evaluation customers and the tuning for their network. So debugfs should be okay.

In my experience it makes no difference, if the customer is using secure boot then they are always using secure boot expect in small lab systems perhaps.

Are you certain this is useful not just "should be okay" ?

Yes. It is useful based on the request we have seen from our customers.

Quoted 2 lines

Jason

#7 lore raw
In reply to Selvin Xavier
Quoted 10 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Currently supporting only GEN 0 version of the parameters. Reading these files queries the firmware and report the values currently programmed. Writing to the files sends commands that update the congestion control settings

[...]

Applied, thanks!

[1/1] RDMA/bnxt_re: Congestion control settings using debugfs hook

      https://git.kernel.org/rdma/rdma/c/a3c71713d9548f

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
#8 lore raw
In reply to Selvin Xavier
Quoted 18 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Currently supporting only GEN 0 version of the parameters. Reading these files queries the firmware and report the values currently programmed. Writing to the files sends commands that update the congestion control settings

---
v1 -> v2:
Addressed Leon's comments
- rename debugfs file "g" to "run_avg_weight_g"
- Fix the indentation errors
- Remove the unnecessary error message during the read entry point
- Fix the return value

Quoted 4 lines
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/debugfs.c | 212 +++++++++++++++++++++++++++++++-
 drivers/infiniband/hw/bnxt_re/debugfs.h |  15 +++
 3 files changed, 228 insertions(+), 1 deletion(-)

We don't have strong policy of what can/can't be in debugfs, so let's take it.

Thanks

#9 lore raw
In reply to Selvin Xavier
Quoted 23 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Currently supporting only GEN 0 version of the parameters. Reading these files queries the firmware and report the values currently programmed. Writing to the files sends commands that update the congestion control settings

---
v1 -> v2:
Addressed Leon's comments
- rename debugfs file "g" to "run_avg_weight_g"
- Fix the indentation errors
- Remove the unnecessary error message during the read entry point
- Fix the return value

 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/debugfs.c | 212 +++++++++++++++++++++++++++++++-
 drivers/infiniband/hw/bnxt_re/debugfs.h |  15 +++
 3 files changed, 228 insertions(+), 1 deletion(-)

...

Quoted 17 lines
drivers/infiniband/hw/bnxt_re/debugfs.c +10 -0
+static const struct file_operations bnxt_re_cc_config_ops = {
+	.owner = THIS_MODULE,
+	.open = simple_open,
+	.read = bnxt_re_cc_config_get,
+	.write = bnxt_re_cc_config_set,
+};
+
 void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev)
 {
 	struct pci_dev *pdev = rdev->en_dev->pdev;
+	struct bnxt_re_dbg_cc_config_params *cc_params;
+	int i;
 
 	rdev->dbg_root = debugfs_create_dir(dev_name(&pdev->dev), bnxt_re_debugfs_root);
 
 	rdev->qp_debugfs = debugfs_create_dir("QPs", rdev->dbg_root);
+	rdev->cc_config = debugfs_create_dir("cc_config", rdev->dbg_root);

drivers/infiniband/hw/bnxt_re/debugfs.c void bnxt_re_debug_rem_qpinfo(struct bnxt_re_dev *rdev, struct bnxt_re_qp *qp) · line 325

Quoted 10 lines
drivers/infiniband/hw/bnxt_re/debugfs.c +10 -0
+
+	rdev->cc_config_params = kzalloc(sizeof(*cc_params), GFP_KERNEL);
+
+	for (i = 0; i < BNXT_RE_CC_PARAM_GEN0; i++) {
+		struct bnxt_re_cc_param *tmp_params = &rdev->cc_config_params->gen0_parms[i];
+
+		tmp_params->rdev = rdev;
+		tmp_params->offset = i;
+		tmp_params->cc_gen = CC_CONFIG_GEN0_EXT0;
+		tmp_params->dentry = debugfs_create_file(bnxt_re_cc_gen0_name[i], 0400,

Write operation doesn't seem to work?

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/debugfs.c +3 -0
+							 rdev->cc_config, tmp_params,
+							 &bnxt_re_cc_config_ops);
+	}
 }
author maintainer
#10 lore raw
Quoted 44 lines · Chengchang Tang
Quoted 23 lines · Selvin Xavier

Implements routines to set and get different settings of the congestion control. This will enable the users to modify the settings according to their network.

Currently supporting only GEN 0 version of the parameters. Reading these files queries the firmware and report the values currently programmed. Writing to the files sends commands that update the congestion control settings

---
v1 -> v2:
Addressed Leon's comments
- rename debugfs file "g" to "run_avg_weight_g"
- Fix the indentation errors
- Remove the unnecessary error message during the read entry point
- Fix the return value

 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/debugfs.c | 212 +++++++++++++++++++++++++++++++-
 drivers/infiniband/hw/bnxt_re/debugfs.h |  15 +++
 3 files changed, 228 insertions(+), 1 deletion(-)

...

Quoted 15 lines
drivers/infiniband/hw/bnxt_re/debugfs.c +9 -0
+static const struct file_operations bnxt_re_cc_config_ops = {
+     .owner = THIS_MODULE,
+     .open = simple_open,
+     .read = bnxt_re_cc_config_get,
+     .write = bnxt_re_cc_config_set,
+};
+
 void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev)
 {
      struct pci_dev *pdev = rdev->en_dev->pdev;
+     struct bnxt_re_dbg_cc_config_params *cc_params;
+     int i;
 
      rdev->dbg_root = debugfs_create_dir(dev_name(&pdev->dev), bnxt_re_debugfs_root);
Quoted 14 lines

drivers/infiniband/hw/bnxt_re/debugfs.c void bnxt_re_debug_rem_qpinfo(struct bnxt_re_dev *rdev, struct bnxt_re_qp *qp) · line 325

Quoted 12 lines
      rdev->qp_debugfs = debugfs_create_dir("QPs", rdev->dbg_root);
drivers/infiniband/hw/bnxt_re/debugfs.c +11 -0
+     rdev->cc_config = debugfs_create_dir("cc_config", rdev->dbg_root);
+
+     rdev->cc_config_params = kzalloc(sizeof(*cc_params), GFP_KERNEL);
+
+     for (i = 0; i < BNXT_RE_CC_PARAM_GEN0; i++) {
+             struct bnxt_re_cc_param *tmp_params = &rdev->cc_config_params->gen0_parms[i];
+
+             tmp_params->rdev = rdev;
+             tmp_params->offset = i;
+             tmp_params->cc_gen = CC_CONFIG_GEN0_EXT0;
+             tmp_params->dentry = debugfs_create_file(bnxt_re_cc_gen0_name[i], 0400,

Write operation doesn't seem to work?

right. I made a mistake in some code cleanup done just before posting this patch.
I have a bug fix patch for this, will post it. Thanks.

Quoted 7 lines
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/debugfs.c +3 -0
+                                                      rdev->cc_config, tmp_params,
+                                                      &bnxt_re_cc_config_ops);
+     }
 }