RDMA/bnxt_re: Bug fixes for 6.14
Subsystems (2)
Selvin Xavier
maintainercover RDMA/bnxt_re: Bug fixes for 6.14
Includes some of the critical fixes found while testing the 6.14 branch. Please review and apply.
Thanks,
Selvin
Kalesh AP (3):
RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier RDMA/bnxt_re: Add sanity checks on rdev validity RDMA/bnxt_re: Fix issue in the unload path
Selvin Xavier (1):
RDMA/bnxt_re: Fix the statistics for Gen P7 VF
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 - drivers/infiniband/hw/bnxt_re/hw_counters.c | 4 ++-- drivers/infiniband/hw/bnxt_re/main.c | 22 +++++++++++----------- drivers/infiniband/hw/bnxt_re/qplib_res.h | 8 ++++++++ 4 files changed, 21 insertions(+), 14 deletions(-)
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 1/4 RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier
From: Kalesh AP <[email protected]>
In the bnxt_re_async_notifier() callback, the way driver retrieves rdev pointer is wrong. The rdev pointer should be parsed from adev pointer as while registering with the L2 for ULP, driver uses the aux device pointer for the handle.
--- drivers/infiniband/hw/bnxt_re/main.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/main.c +6 -1
@@ -396,11 +396,16 @@ static void bnxt_re_dcb_wq_task(struct work_struct *work)static void bnxt_re_async_notifier (void *handle,struct hwrm_async_event_cmpl *cmpl) { -struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; +struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (handle);struct bnxt_re_dcb_work *dcb_work; +struct bnxt_re_dev *rdev;u32 data1, data2;u16 event_id; + rdev = en_info->rdev; +if (!rdev) +return ; + event_id =le16_to_cpu (cmpl->event_id); data1 =le32_to_cpu (cmpl->event_data1); data2 =le32_to_cpu (cmpl->event_data2);
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 2/4 RDMA/bnxt_re: Add sanity checks on rdev validity
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
There is a possibility that ulp_irq_stop and ulp_irq_start callbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.
--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +5 -0
@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ; rcfw = &rdev->rcfw;if (reset) {
@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {
@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);return 0 ;
Signature · 1 line
2.5.5
Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and ulp_irq_stop. If there is such protection, they can't race.
The main idea of auxiliary bus is to remove the need to implement driver specific ops.
Quoted 13 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0
drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;
This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);return 0 ;Signature · 1 line
2.5.5
Kalesh Anakkur Purayil
maintainerRe: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.
Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the ulp_irq_stop() callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.
We are trying to address the NULL pointer dereference issue here.
The driver specific ops, ulp_irq_stop and ulp_irq_start are required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.
Hope this clarifies your question.
Quoted 49 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-1rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);return 0 ; --2.5.5
Signature · 2 lines
Regards, Kalesh AP
Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 23 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
Quoted 8 lines
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.
Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Quoted 2 lines
We are trying to address the NULL pointer dereference issue here.
You are hiding bugs and not fixing them.
Quoted 61 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 49 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 23 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-1rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);return 0 ; --2.5.5
Signature · 2 lines
Regards, Kalesh AP
Kalesh Anakkur Purayil
maintainerRe: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Hi Leon,
Quoted 29 lines · Leon Romanovsky
Quoted 27 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
Quoted 7 lines
Quoted 4 lines
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
Quoted 2 lines
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.
Quoted 5 lines
Quoted 3 lines
We are trying to address the NULL pointer dereference issue here.
You are hiding bugs and not fixing them.
Yes, but this change is critical for the current design of the driver.
Quoted 64 lines
Quoted 62 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0
-1return 0 ; --2.5.5
--
Regards,
Kalesh AP
Signature · 2 lines
Regards, Kalesh AP
Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 37 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 34 lines · Leon Romanovsky
Quoted 31 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.
Quoted 7 lines
Quoted 2 lines
Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
This is wrong.
Quoted 11 lines
Quoted 2 lines
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.
Quoted 5 lines
Quoted 3 lines
We are trying to address the NULL pointer dereference issue here.
You are hiding bugs and not fixing them.
Yes, but this change is critical for the current design of the driver.
Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/
Thanks
Quoted 69 lines
Quoted 64 lines
Quoted 62 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0
-1return 0 ; --2.5.5
--
Regards,
Kalesh APSignature · 2 lines
Regards, Kalesh AP
Kalesh Anakkur Purayil
maintainerRe: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Hi Leon,
Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
Quoted 8 lines
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
Quoted 6 lines
Quoted 4 lines
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
This is wrong.
We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking auxdrv->suspend callback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.
Quoted 4 lines
Quoted 3 lines
We are trying to address the NULL pointer dereference issue here.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
You are hiding bugs and not fixing them.
Yes, but this change is critical for the current design of the driver.
Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/
Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.
Quoted 74 lines
Thanks
Quoted 69 lines
Quoted 52 lines
Quoted 52 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0
-1return 0 ; --2.5.5
--
Regards,
Kalesh AP--
Regards,
Kalesh AP
Signature · 2 lines
Regards, Kalesh AP
Selvin Xavier
maintainerRe: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 69 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
Quoted 8 lines
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
Quoted 6 lines
Quoted 4 lines
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
This is wrong.
We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking
auxdrv->suspendcallback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.
Quoted 5 lines
Quoted 3 lines
We are trying to address the NULL pointer dereference issue here.
You are hiding bugs and not fixing them.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Yes, but this change is critical for the current design of the driver.
Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.
Hi Leon,
Is it okay to merge this change along with the other fixes in the
series? Its important
for the current driver design.
Thanks,
Quoted 79 lines
Quoted 74 lines
Thanks
Quoted 69 lines
Quoted 52 lines
Quoted 52 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0
-1return 0 ; --2.5.5
--
Regards,
Kalesh AP--
Regards,
Kalesh AP--
Regards,
Kalesh AP
Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 77 lines · Selvin Xavier
Quoted 75 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 42 lines · Leon Romanovsky
Quoted 40 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 36 lines · Leon Romanovsky
Quoted 23 lines · Kalesh Anakkur Purayil
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
There is a possibility that
ulp_irq_stopandulp_irq_startcallbacks will be called when the device is in detached state. This can cause a crash due to NULL pointer dereference as the rdev is already freed.Quoted 6 lines
Description and code doesn't match. If "possibility" exists, there is no protection from concurrent detach and
ulp_irq_stop. If there is such protection, they can't race.The main idea of auxiliary bus is to remove the need to implement driver specific ops.
There is no race condition here.
Let me explain the scenario.
Quoted 8 lines
User is doing a devlink reload reinit. As part of that, the Ethernet driver first invokes the auxiliary bus suspend callback. The RDMA driver will do the unwinding operation and hence rdev will be freed.
After that, during the devlink sequence, Ethernet driver invokes the
ulp_irq_stop()callback and this resulted in the NULL pointer dereference as the RDMA driver is in detached state and the rdev is already freed.Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
Quoted 6 lines
Quoted 4 lines
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
This is wrong.
We understand that. BNXT core driver team has already started working on the auxiliary device removal instead of invoking
auxdrv->suspendcallback in the devlink relaod path. That will avoid these NULL checks in RDMA driver. for time being we need these NULL checks.
That will be posted to net-next tree once internal testing and review is done.Quoted 16 lines
Quoted 7 lines
Quoted 3 lines
BNXT core driver controls reload, it shouldn't call to drivers which doesn't exist.
Since the RDMA driver instance is registered with Ethernet driver, core Ethernet driver invokes the callback.
Quoted 2 lines
Quoted 2 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We are trying to address the NULL pointer dereference issue here.
You are hiding bugs and not fixing them.
Yes, but this change is critical for the current design of the driver.
Please fix it once and for all by doing proper reload sequence. I warned you that setting NULLs to pointers hide bugs.
https://lore.kernel.org/linux-rdma/20250114112555.GG3146852@unreal/
Quoted 6 lines
Quoted 2 lines
Yes, I understand. We will work on the suggestion that you had given based on the new design mentioned in last comment.
Hi Leon,
Is it okay to merge this change along with the other fixes in the
series? Its important
for the current driver design.
Yes, because it is current situation, i will merge it tomorrow.
Thanks
Quoted 85 lines
Thanks,
Quoted 76 lines
Quoted 74 lines
Thanks
Quoted 69 lines
Quoted 52 lines
Quoted 52 lines
The driver specific ops,
ulp_irq_stopandulp_irq_startare required. Broadcom Ethernet and RDMA drivers are designed like that to manage IRQs between them.Hope this clarifies your question.
Quoted 45 lines
Quoted 14 lines
Fixes cc5b9b48d447 ("RDMA/bnxt_re: Recover the device when FW error is detected")Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/main.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/infiniband/hw/bnxt_re/main.c +0
-0drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_stop_irq(void *handle, bool reset) · line 442
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0@@ -438,6 +438,8 @@ static void bnxt_re_stop_irq(void *handle, bool reset)int indx; rdev = en_info->rdev; +if (!rdev) +return ;This can be seen as an example why I'm so negative about assigning NULL to the pointers after object is destroyed. Such assignment makes layer violation much easier job to do.
Thanks
Quoted 19 lines
drivers/infiniband/hw/bnxt_re/main.c +3
-0rcfw = &rdev->rcfw;if (reset) {@@ -466,6 +468,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)int indx, rc; rdev = en_info->rdev; +if (!rdev) +return ; msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {@@ -2438,6 +2442,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state)ibdev_info (&rdev->ibdev,"%s: L2 driver notified to stop en_state 0x%lx" , __func__, en_dev->en_state);bnxt_re_remove_device (rdev,BNXT_RE_PRE_RECOVERY_REMOVE , adev); +bnxt_re_update_en_info_rdev (NULL , en_info, adev);mutex_unlock (&bnxt_re_mutex);Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/bnxt_re/main.c +0
-1return 0 ; --2.5.5
--
Regards,
Kalesh AP--
Regards,
Kalesh APQuoted 3 lines
--
Regards,
Kalesh APAttachments:
* smime.p7s
Re: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
In reply to Kalesh Anakkur PurayilQuoted 7 lines · Kalesh Anakkur Purayil
Quoted 2 lines
Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
What is the difference between your suspend and reloading the driver??
Do you keep the ib device registered during suspend and all uverbs contexts open? How does that work???
Jason
Kalesh Anakkur Purayil
maintainerRe: [PATCH rdma-rc 2/4] RDMA/bnxt_re: Add sanity checks on rdev validity
Quoted 10 lines · Jason Gunthorpe
Quoted 7 lines · Kalesh Anakkur Purayil
Quoted 2 lines
Shouldn't devlink reload completely release all auxiliary drivers? Why are you keeping BNXT RDMA driver during reload?
That is the current design. BNXT core Ethernet driver will not destroy the auxiliary device for RDMA, but just calls the suspend callback. As a result, RDMA driver will remains loaded and remains registered with the Ethernet driver instance.
Quoted 4 lines
What is the difference between your suspend and reloading the driver??
Do you keep the ib device registered during suspend and all uverbs contexts open? How does that work???
We unregister the IB device in both cases.
The only difference is in the suspend path, the communication channel with the Ethernet driver is still active. As a result, the ethernet driver may invoke ULP callbacks even though the IB device is removed.
As I mentioned in my last response to Leon, we are working on to improve this design to remove the auxiliary device completely instead of invoking auxdrv->suspend.
Quoted 2 lines
Jason
Signature · 2 lines
Regards, Kalesh AP
Selvin Xavier
maintainerPATCH 3/4 RDMA/bnxt_re: Fix issue in the unload path
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 - drivers/infiniband/hw/bnxt_re/main.c | 10 ---------- 2 files changed, 11 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +0 -1
@@ -187,7 +187,6 @@ struct bnxt_re_dev {#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29struct net_device *netdev;struct auxiliary_device *adev; -struct notifier_block nb;unsigned int version, major, minor;struct bnxt_qplib_chip_ctx *chip_ctx;struct bnxt_en_dev *en_dev;
drivers/infiniband/hw/bnxt_re/main.c +0 -10
@@ -1359,7 +1359,6 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct auxiliary_device *adev,return NULL ; }/* Default values */ - rdev->nb.notifier_call =NULL ; rdev->netdev = en_dev->net; rdev->en_dev = en_dev; rdev->adev = adev;
@@ -2354,15 +2353,6 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type)static void bnxt_re_remove_device (struct bnxt_re_dev *rdev,u8 op_type,struct auxiliary_device *aux_dev) { -if (rdev->nb.notifier_call) { -unregister_netdevice_notifier (&rdev->nb); - rdev->nb.notifier_call =NULL ; - }else { -/* If notifier is null, we should have already done a -* clean up before coming here. -*/ -return ; - }bnxt_re_setup_cc (rdev, false);ib_unregister_device (&rdev->ibdev);bnxt_re_dev_uninit (rdev, op_type);
Signature · 1 line
2.5.5
Jason Gunthorpe
maintainerRe: [PATCH rdma-rc 3/4] RDMA/bnxt_re: Fix issue in the unload path
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.
Please don't use vauge generic commit titles:
RDMA/bnxt_re: Fix issue in the unload path
Be specific and exact, there are several cases like this in this series and I'm actively wondering if Linus will object to this for rc patches.
RDMA/bnxt_re: Fix missed call to ib_unregister_device()
Since rdev->nb.notifier_call is always NULL the actual work in bnxt_re_remove_device() was skipped
Jason
Selvin Xavier
maintainerRe: [PATCH rdma-rc 3/4] RDMA/bnxt_re: Fix issue in the unload path
Quoted 13 lines · Jason Gunthorpe
Quoted 6 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
The cited comment removed the netdev notifier register call from the driver. But, it did not remove the cleanup code from the unload path. As a result, driver unload is not clean and resulted in undesired behaviour.
Please don't use vauge generic commit titles:
RDMA/bnxt_re: Fix issue in the unload path
Quoted 8 lines
Be specific and exact, there are several cases like this in this series and I'm actively wondering if Linus will object to this for rc patches.
RDMA/bnxt_re: Fix missed call to
ib_unregister_device()Since
rdev->nb.notifier_callis always NULL the actual work inbnxt_re_remove_device()was skipped
Sure. will try to be specific in the commit messages in future. thanks
Quoted 2 lines
Jason
Selvin Xavier
maintainerPATCH 4/4 RDMA/bnxt_re: Fix the statistics for Gen P7 VF
In reply to Selvin XavierGen P7 VF support the extended stats and is prevented by a VF check. Fix the check to issue the FW command for GenP7 VFs also.
--- drivers/infiniband/hw/bnxt_re/hw_counters.c | 4 ++-- drivers/infiniband/hw/bnxt_re/qplib_res.h | 8 ++++++++ 2 files changed, 10 insertions(+), 2 deletions(-)
drivers/infiniband/hw/bnxt_re/hw_counters.c +2 -2
@@ -348,8 +348,8 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, goto done; }bnxt_re_copy_err_stats (rdev, stats, err_s); -if (_is_ext_stats_supported (rdev->dev_attr->dev_cap_flags) && - !rdev->is_virtfn) { +if (bnxt_ext_stats_supported (rdev->chip_ctx, rdev->dev_attr->dev_cap_flags, + rdev->is_virtfn)) { rc =bnxt_re_get_ext_stat (rdev, stats);if (rc) { clear_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS,
drivers/infiniband/hw/bnxt_re/qplib_res.h +8 -0
@@ -547,6 +547,14 @@ static inline bool _is_ext_stats_supported(u16 dev_cap_flags)CREQ_QUERY_FUNC_RESP_SB_EXT_STATS ; } +static inline int bnxt_ext_stats_supported (struct bnxt_qplib_chip_ctx *ctx, +u16 flags,bool virtfn) +{ +/* ext stats supported if cap flag is set AND is a PF OR a Thor2 VF */ +return (_is_ext_stats_supported (flags) && + ((virtfn &&bnxt_qplib_is_chip_gen_p7 (ctx)) || (!virtfn))); +} +static inline bool _is_hw_retx_supported (u16 dev_cap_flags) { return dev_cap_flags &
Signature · 1 line
2.5.5
Leon Romanovsky
AppliedQuoted 6 lines · Selvin Xavier
Includes some of the critical fixes found while testing the 6.14 branch. Please review and apply.
Thanks,
Selvin
Quoted 6 lines
Kalesh AP (3):
RDMA/bnxt_re: Fix an issue inbnxt_re_async_notifierRDMA/bnxt_re: Add sanity checks on rdev validity RDMA/bnxt_re: Fix issue in the unload path[...]
Applied, thanks!
[1/4] RDMA/bnxt_re: Fix an issue in bnxt_re_async_notifier
https://git.kernel.org/rdma/rdma/c/a27c6f46dcec8f [2/4] RDMA/bnxt_re: Add sanity checks on rdev validity https://git.kernel.org/rdma/rdma/c/f0df225d12fcb0 [3/4] RDMA/bnxt_re: Fix issue in the unload path https://git.kernel.org/rdma/rdma/c/e2f105277411c4 [4/4] RDMA/bnxt_re: Fix the statistics for Gen P7 VF https://git.kernel.org/rdma/rdma/c/8238c7bd84209c
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>