Skip to contentSkip to conversation

RDMA/bnxt_re: Bug fixes for 6.14

PATCH Applied Selvin Xavier 5 participants 18 messages Feb 4, 2025 – Feb 26, 2025

Subsystems (2)
lore
Share & tools
lore

Selvin Xavier

author maintainer
#0 lore raw

cover RDMA/bnxt_re: Bug fixes for 6.14

Includes some of the critical fixes found while testing the 6.14 branch. Please review and apply.

Thanks,
Selvin

Kalesh AP (3):
RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier RDMA/bnxt_re: Add sanity checks on rdev validity RDMA/bnxt_re: Fix issue in the unload path

Selvin Xavier (1):
RDMA/bnxt_re: Fix the statistics for Gen P7 VF

 drivers/infiniband/hw/bnxt_re/bnxt_re.h     |  1 -
 drivers/infiniband/hw/bnxt_re/hw_counters.c |  4 ++--
 drivers/infiniband/hw/bnxt_re/main.c        | 22 +++++++++++-----------
 drivers/infiniband/hw/bnxt_re/qplib_res.h   |  8 ++++++++
 4 files changed, 21 insertions(+), 14 deletions(-)
Signature · 1 line
2.5.5
author maintainer
#1 lore raw

PATCH 1/4 RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier

From: Kalesh AP <[email protected]>

In the bnxt_re_async_notifier() callback, the way driver retrieves rdev pointer is wrong. The rdev pointer should be parsed from adev pointer as while registering with the L2 for ULP, driver uses the aux device pointer for the handle.

Fixes 7fea32784068 ("RDMA/bnxt_re: Add Async event handling support")
---
 drivers/infiniband/hw/bnxt_re/main.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/main.c +6 -1
index e9e4da4..c4c3d67 100644
@@ -396,11 +396,16 @@ static void bnxt_re_dcb_wq_task(struct work_struct *work)
 
 static void bnxt_re_async_notifier(void *handle, struct hwrm_async_event_cmpl *cmpl)
 {
-	struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle;
+	struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle);
 	struct bnxt_re_dcb_work *dcb_work;
+	struct bnxt_re_dev *rdev;
 	u32 data1, data2;
 	u16 event_id;
 
+	rdev = en_info->rdev;
+	if (!rdev)
+		return;
+
 	event_id = le16_to_cpu(cmpl->event_id);
 	data1 = le32_to_cpu(cmpl->event_data1);
 	data2 = le32_to_cpu(cmpl->event_data2);
Signature · 1 line
2.5.5
author maintainer
#2 lore raw

PATCH 2/4 RDMA/bnxt_re: Add sanity checks on rdev validity

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +5 -0
index c4c3d67..89ac5c2 100644
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
 	int indx;
 
 	rdev = en_info->rdev;
+	if (!rdev)
+		return;
 	rcfw = &rdev->rcfw;
 
 	if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
 	int indx, rc;
 
 	rdev = en_info->rdev;
+	if (!rdev)
+		return;
 	msix_ent = rdev->nqr->msix_entries;
 	rcfw = &rdev->rcfw;
 	if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
 	ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
 		   __func__, en_dev->en_state);
 	bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+	bnxt_re_update_en_info_rdev(NULL, en_info, adev);
 	mutex_unlock(&bnxt_re_mutex);
 
 	return 0;
Signature · 1 line
2.5.5
#3 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

Quoted 13 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
 	int indx;
 
 	rdev = en_info->rdev;
+	if (!rdev)
+		return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
 	rcfw = &rdev->rcfw;
 
 	if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
 	int indx, rc;
 
 	rdev = en_info->rdev;
+	if (!rdev)
+		return;
 	msix_ent = rdev->nqr->msix_entries;
 	rcfw = &rdev->rcfw;
 	if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
 	ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
 		   __func__, en_dev->en_state);
 	bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+	bnxt_re_update_en_info_rdev(NULL, en_info, adev);
 	mutex_unlock(&bnxt_re_mutex);
 
 	return 0;
Signature · 1 line
2.5.5
#4 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

We are trying to address the NULL pointer dereference issue here.

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 49 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -1
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
      return 0;
--

2.5.5

Signature · 2 lines
Regards,
Kalesh AP
#5 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 23 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

Quoted 8 lines

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Quoted 2 lines

We are trying to address the NULL pointer dereference issue here.

You are hiding bugs and not fixing them.

Quoted 61 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 49 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -1
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
      return 0;
--

2.5.5

Signature · 2 lines
Regards,
Kalesh AP
#6 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Hi Leon,

Quoted 29 lines · Leon Romanovsky
Quoted 27 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

Quoted 7 lines
Quoted 4 lines

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

Quoted 2 lines

BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.

Quoted 5 lines
Quoted 3 lines

We are trying to address the NULL pointer dereference issue here.

You are hiding bugs and not fixing them.

Yes, but this change is critical for the current design of the driver.

Quoted 64 lines
Quoted 62 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0 -1
      return 0;
--

2.5.5

--
Regards,
Kalesh AP

Signature · 2 lines
Regards,
Kalesh AP
#7 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 37 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 34 lines · Leon Romanovsky
Quoted 31 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Quoted 7 lines
Quoted 2 lines

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

This is wrong.

Quoted 11 lines
Quoted 2 lines

BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.

Quoted 5 lines
Quoted 3 lines

We are trying to address the NULL pointer dereference issue here.

You are hiding bugs and not fixing them.

Yes, but this change is critical for the current design of the driver.

Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/

Thanks

Quoted 69 lines
Quoted 64 lines
Quoted 62 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0 -1
      return 0;
--

2.5.5

--
Regards,
Kalesh AP

Signature · 2 lines
Regards,
Kalesh AP
#8 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Hi Leon,

Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

Quoted 8 lines

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

Quoted 6 lines
Quoted 4 lines

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

This is wrong.

We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking auxdrv->suspend callback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.

Quoted 4 lines
Quoted 3 lines

We are trying to address the NULL pointer dereference issue here.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

You are hiding bugs and not fixing them.

Yes, but this change is critical for the current design of the driver.

Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/

Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.

Quoted 74 lines

Thanks

Quoted 69 lines
Quoted 52 lines
Quoted 52 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0 -1
      return 0;
--

2.5.5

--
Regards,
Kalesh AP

--
Regards,
Kalesh AP

Signature · 2 lines
Regards,
Kalesh AP
author maintainer
#9 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 69 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

Quoted 8 lines

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

Quoted 6 lines
Quoted 4 lines

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

This is wrong.

We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking auxdrv->suspend callback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.

Quoted 5 lines
Quoted 3 lines

We are trying to address the NULL pointer dereference issue here.

You are hiding bugs and not fixing them.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Yes, but this change is critical for the current design of the driver.

Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/

Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.

Hi Leon,
Is it okay to merge this change along with the other fixes in the
series? Its important
for the current driver design.

Thanks,

Quoted 79 lines
Quoted 74 lines

Thanks

Quoted 69 lines
Quoted 52 lines
Quoted 52 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0 -1
      return 0;
--

2.5.5

--
Regards,
Kalesh AP

--
Regards,
Kalesh AP

--
Regards,
Kalesh AP

#10 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 77 lines · Selvin Xavier
Quoted 75 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.

Quoted 6 lines

Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.

The main idea of auxiliary bus is to remove the need to implement driver specific ops.

There is no race condition here.

Let me explain the scenario.

Quoted 8 lines

User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.

After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

Quoted 6 lines
Quoted 4 lines

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

This is wrong.

We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking auxdrv->suspend callback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.

Quoted 16 lines
Quoted 7 lines
Quoted 3 lines

BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.

Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.

Quoted 2 lines
Quoted 2 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

We are trying to address the NULL pointer dereference issue here.

You are hiding bugs and not fixing them.

Yes, but this change is critical for the current design of the driver.

Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/

Quoted 6 lines
Quoted 2 lines

Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.

Hi Leon,
Is it okay to merge this change along with the other fixes in the
series? Its important
for the current driver design.

Yes, because it is current situation, i will merge it tomorrow.

Thanks

Quoted 85 lines

Thanks,

Quoted 76 lines
Quoted 74 lines

Thanks

Quoted 69 lines
Quoted 52 lines
Quoted 52 lines

The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.

Hope this clarifies your question.

Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")
---
 drivers/infiniband/hw/bnxt_re/main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +0 -0
index c4c3d67..89ac5c2 100644

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)
      int indx;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;

This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.

Thanks

Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3 -0
      rcfw = &rdev->rcfw;
 
      if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
      int indx, rc;
 
      rdev = en_info->rdev;
+     if (!rdev)
+             return;
      msix_ent = rdev->nqr->msix_entries;
      rcfw = &rdev->rcfw;
      if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)
      ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx",
                 __func__, en_dev->en_state);
      bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev);
+     bnxt_re_update_en_info_rdev(NULL, en_info, adev);
      mutex_unlock(&bnxt_re_mutex);
 
Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0 -1
      return 0;
--

2.5.5

--
Regards,
Kalesh AP

--
Regards,
Kalesh AP

Quoted 3 lines

--
Regards,
Kalesh AP

Attachments:
* smime.p7s

#11 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

In reply to Kalesh Anakkur Purayil
Quoted 7 lines · Kalesh Anakkur Purayil
Quoted 2 lines

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

What is the difference between your suspend and reloading the driver??

Do you keep the ib device registered during suspend and all uverbs contexts open? How does that work???

Jason

#12 lore raw

Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity

Quoted 10 lines · Jason Gunthorpe
Quoted 7 lines · Kalesh Anakkur Purayil
Quoted 2 lines

Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?

That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.

Quoted 4 lines

What is the difference between your suspend and reloading the driver??

Do you keep the ib device registered during suspend and all uverbs contexts open? How does that work???

We unregister the IB device in both cases.
The only difference is in the suspend path, the communication channel with the Ethernet driver is still active. As a result, the ethernet driver may invoke ULP callbacks even though the IB device is removed.

As I mentioned in my last response to Leon, we are working on to improve this design to remove the auxiliary device completely instead of invoking auxdrv->suspend.

Quoted 2 lines

Jason

Signature · 2 lines
Regards,
Kalesh AP
author maintainer
#13 lore raw

PATCH 3/4 RDMA/bnxt_re: Fix issue in the unload path

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.

Fixes d3b15fcc4201 ("RDMA/bnxt_re: Remove deliver net device event")
---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |  1 -
 drivers/infiniband/hw/bnxt_re/main.c    | 10 ----------
 2 files changed, 11 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +0 -1
index b91a85a..3721446 100644
@@ -187,7 +187,6 @@ struct bnxt_re_dev {
 #define BNXT_RE_FLAG_ISSUE_ROCE_STATS          29
 	struct net_device		*netdev;
 	struct auxiliary_device         *adev;
-	struct notifier_block		nb;
 	unsigned int			version, major, minor;
 	struct bnxt_qplib_chip_ctx	*chip_ctx;
 	struct bnxt_en_dev		*en_dev;
drivers/infiniband/hw/bnxt_re/main.c +0 -10
index 89ac5c2..a94c8c5 100644
@@ -1359,7 +1359,6 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct auxiliary_device *adev,
 		return NULL;
 	}
 	/* Default values */
-	rdev->nb.notifier_call = NULL;
 	rdev->netdev = en_dev->net;
 	rdev->en_dev = en_dev;
 	rdev->adev = adev;
@@ -2354,15 +2353,6 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type)
 static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, u8 op_type,
 				  struct auxiliary_device *aux_dev)
 {
-	if (rdev->nb.notifier_call) {
-		unregister_netdevice_notifier(&rdev->nb);
-		rdev->nb.notifier_call = NULL;
-	} else {
-		/* If notifier is null, we should have already done a
-		 * clean up before coming here.
-		 */
-		return;
-	}
 	bnxt_re_setup_cc(rdev, false);
 	ib_unregister_device(&rdev->ibdev);
 	bnxt_re_dev_uninit(rdev, op_type);
Signature · 1 line
2.5.5
maintainer
#14 lore raw

Re: [PATCH rdma-rc 3/4] RDMA/bnxt_re: Fix issue in the unload path

Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.

Please don't use vauge generic commit titles:

RDMA/bnxt_re: Fix issue in the unload path

Be specific and exact, there are several cases like this in this series and I'm actively wondering if Linus will object to this for rc patches.

RDMA/bnxt_re: Fix missed call to ib_unregister_device()

Since rdev->nb.notifier_call is always NULL the actual work in bnxt_re_remove_device() was skipped

Jason

author maintainer
#15 lore raw

Re: [PATCH rdma-rc 3/4] RDMA/bnxt_re: Fix issue in the unload path

Quoted 13 lines · Jason Gunthorpe
Quoted 6 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.

Please don't use vauge generic commit titles:

RDMA/bnxt_re: Fix issue in the unload path

Quoted 8 lines

Be specific and exact, there are several cases like this in this series and I'm actively wondering if Linus will object to this for rc patches.

RDMA/bnxt_re: Fix missed call to ib_unregister_device()

Since rdev->nb.notifier_call is always NULL the actual work in bnxt_re_remove_device() was skipped

Sure. will try to be specific in the commit messages in future. thanks

Quoted 2 lines

Jason

author maintainer
#16 lore raw

PATCH 4/4 RDMA/bnxt_re: Fix the statistics for Gen P7 VF

In reply to Selvin Xavier

Gen P7 VF support the extended stats and is prevented by a VF check. Fix the check to issue the FW command for GenP7 VFs also.

Fixes 1801d87b3598 ("RDMA/bnxt_re: Support new 5760X P7 devices")
---
 drivers/infiniband/hw/bnxt_re/hw_counters.c | 4 ++--
 drivers/infiniband/hw/bnxt_re/qplib_res.h   | 8 ++++++++
 2 files changed, 10 insertions(+), 2 deletions(-)
drivers/infiniband/hw/bnxt_re/hw_counters.c +2 -2
index 3ac47f4..f039aef 100644
@@ -348,8 +348,8 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev,
 			goto done;
 		}
 		bnxt_re_copy_err_stats(rdev, stats, err_s);
-		if (_is_ext_stats_supported(rdev->dev_attr->dev_cap_flags) &&
-		    !rdev->is_virtfn) {
+		if (bnxt_ext_stats_supported(rdev->chip_ctx, rdev->dev_attr->dev_cap_flags,
+					     rdev->is_virtfn)) {
 			rc = bnxt_re_get_ext_stat(rdev, stats);
 			if (rc) {
 				clear_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS,
drivers/infiniband/hw/bnxt_re/qplib_res.h +8 -0
index be5d907..7119902 100644
@@ -547,6 +547,14 @@ static inline bool _is_ext_stats_supported(u16 dev_cap_flags)
 		CREQ_QUERY_FUNC_RESP_SB_EXT_STATS;
 }
 
+static inline int bnxt_ext_stats_supported(struct bnxt_qplib_chip_ctx *ctx,
+					   u16 flags, bool virtfn)
+{
+	/* ext stats supported if cap flag is set AND is a PF OR a Thor2 VF */
+	return (_is_ext_stats_supported(flags) &&
+		((virtfn && bnxt_qplib_is_chip_gen_p7(ctx)) || (!virtfn)));
+}
+
 static inline bool _is_hw_retx_supported(u16 dev_cap_flags)
 {
 	return dev_cap_flags &
Signature · 1 line
2.5.5
#17 lore raw
In reply to Selvin Xavier
Quoted 6 lines · Selvin Xavier

Includes some of the critical fixes found while testing the 6.14 branch. Please review and apply.

Thanks,
Selvin

Quoted 6 lines

Kalesh AP (3):
RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier RDMA/bnxt_re: Add sanity checks on rdev validity RDMA/bnxt_re: Fix issue in the unload path

[...]

Applied, thanks!

[1/4] RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier

      https://git.kernel.org/rdma/rdma/c/a27c6f46dcec8f
[2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
      https://git.kernel.org/rdma/rdma/c/f0df225d12fcb0
[3/4] RDMA/bnxt_re: Fix issue in the unload path
      https://git.kernel.org/rdma/rdma/c/e2f105277411c4
[4/4] RDMA/bnxt_re: Fix the statistics for Gen P7 VF
      https://git.kernel.org/rdma/rdma/c/8238c7bd84209c

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>