RDMA/bnxt_re: Device re-initialization after Firmware error
Subsystems (2)
Selvin Xavier
maintainercover RDMA/bnxt_re: Device re-initialization after Firmware error
Add support for complete re-initialization of the device when driver detects a firmware reset. Code reorg that updates the device handles stored with Auxiliary bus and the bnxt_en driver. bnxt_en driver calls suspend and resume hooks upon error recovery. Driver destroys and recreates the roce device instance upon receiving these calls.
Please review and apply these changes for 6.12.
Thanks,
Selvin Xavier
Chandramohan Akula (2):
RDMA/bnxt_re: Change aux driver data to en_info to hold more
information
RDMA/bnxt_re: Use the aux device for L2 ULP callbacks
Selvin Xavier (2):
RDMA/bnxt_re: Group all operations under add_device and remove_device RDMA/bnxt_re: Recover the device when FW error is detected
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 21 ++++ drivers/infiniband/hw/bnxt_re/main.c | 199 +++++++++++++++++++++--------- drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 + 3 files changed, 160 insertions(+), 61 deletions(-)
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 1/4 RDMA/bnxt_re: Change aux driver data to en_info to hold more information
From: Chandramohan Akula <[email protected]>
rdev will be destroyed and recreated during the FW error recovery scenarios. So to keep the state, if any, use an en_info structure which gets created/freed based on auxiliary device initialization/de-initialization.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 6 +++ drivers/infiniband/hw/bnxt_re/main.c | 73 ++++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 11 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +6 -0
@@ -107,6 +107,11 @@ struct bnxt_re_gsi_context {struct bnxt_re_sqp_entries *sqp_tbl; }; +struct bnxt_re_en_dev_info { +struct bnxt_en_dev *en_dev; +struct bnxt_re_dev *rdev; +}; +#define BNXT_RE_AEQ_IDX 0#define BNXT_RE_NQ_IDX 1#define BNXT_RE_GEN_P5_MAX_VF 64
@@ -155,6 +160,7 @@ struct bnxt_re_dev {#define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29struct net_device *netdev; +struct auxiliary_device *adev;struct notifier_block nb;unsigned int version, major, minor;struct bnxt_qplib_chip_ctx *chip_ctx;
drivers/infiniband/hw/bnxt_re/main.c +62 -11
160 lines not on this page · raw message
Signature · 1 line
2.5.5
kernel test robot
build botRe: [PATCH for-next 1/4] RDMA/bnxt_re: Change aux driver data to en_info to hold more information
kernel test robot noticed the following build warnings:
Hi Selvin,
kernel test robot noticed the following build warnings:
[auto build test WARNING on rdma/for-next]
[also build test WARNING on next-20240910]
[cannot apply to linus/master v6.11-rc7]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Selvin-Xavier/RDMA-bnxt_re-Change-aux-driver-data-to-en_info-to-hold-more-information/20240910-122414 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next patch link: https://lore.kernel.org/r/1725940862-4821-2-git-send-email-selvin.xavier%40broadcom.com patch subject: [PATCH for-next 1/4] RDMA/bnxt_re: Change aux driver data to en_info to hold more information config: s390-allyesconfig (https://download.01.org/0day-ci/archive/20240911/[email protected]/config) compiler: s390-linux-gcc (GCC) 14.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240911/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/infiniband/hw/bnxt_re/main.c: In function 'bnxt_re_remove':
>> drivers/infiniband/hw/bnxt_re/main.c:1939:29: warning: variable 'en_dev' set but not used [-Wunused-but-set-variable]
1939 | struct bnxt_en_dev *en_dev;
| ^~~~~~vim +/en_dev +1939 drivers/infiniband/hw/bnxt_re/main.c
1935
1936 static void bnxt_re_remove(struct auxiliary_device *adev)
1937 {
1938 struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev);
> 1939 struct bnxt_en_dev *en_dev;
1940 struct bnxt_re_dev *rdev;
1941
1942 mutex_lock(&bnxt_re_mutex);
1943 if (!en_info) {
1944 mutex_unlock(&bnxt_re_mutex);
1945 return;
1946 }
1947 en_dev = en_info->en_dev;
1948 rdev = en_info->rdev;
1949 if (!rdev)
1950 goto skip_remove;
1951
1952 if (rdev->nb.notifier_call) {
1953 unregister_netdevice_notifier(&rdev->nb);
1954 rdev->nb.notifier_call = NULL;
1955 } else {
1956 /* If notifier is null, we should have already done a
1957 * clean up before coming here.
1958 */
1959 goto skip_remove;
1960 }
1961 bnxt_re_setup_cc(rdev, false);
1962 ib_unregister_device(&rdev->ibdev);
1963 bnxt_re_dev_uninit(rdev);
1964 ib_dealloc_device(&rdev->ibdev);
1965 skip_remove:
1966 kfree(en_info);
1967 mutex_unlock(&bnxt_re_mutex);
1968 }
1969
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Selvin Xavier
maintainerPATCH 2/4 RDMA/bnxt_re: Use the aux device for L2 ULP callbacks
In reply to Selvin XavierFrom: Chandramohan Akula <[email protected]>
While registering with the L2 for ULP operations, use the aux device pointer as the handle. Aux device has the data bnxt_re_en_dev_info, which is used to store required information for the bnxt_re_suspend and bnxt_re_resume functions.
--- drivers/infiniband/hw/bnxt_re/main.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +20 -6
@@ -305,11 +305,18 @@ static void bnxt_re_shutdown(struct auxiliary_device *adev)static void bnxt_re_stop_irq (void *handle) { -struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; -struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; +struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (handle); +struct bnxt_qplib_rcfw *rcfw; +struct bnxt_re_dev *rdev;struct bnxt_qplib_nq *nq;int indx; +if (!en_info) +return ; + + rdev = en_info->rdev; + rcfw = &rdev->rcfw; +for (indx =BNXT_RE_NQ_IDX ; indx < rdev->num_msix; indx++) { nq = &rdev->nq[indx -1 ];bnxt_qplib_nq_stop_irq (nq, false);
@@ -320,12 +327,19 @@ static void bnxt_re_stop_irq(void *handle)static void bnxt_re_start_irq (void *handle,struct bnxt_msix_entry *ent) { -struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; -struct bnxt_msix_entry *msix_ent = rdev->en_dev->msix_entries; -struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; +struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (handle); +struct bnxt_msix_entry *msix_ent; +struct bnxt_qplib_rcfw *rcfw; +struct bnxt_re_dev *rdev;struct bnxt_qplib_nq *nq;int indx, rc; +if (!en_info) +return ; + + rdev = en_info->rdev; + msix_ent = rdev->en_dev->msix_entries; + rcfw = &rdev->rcfw;if (!ent) {/* Not setting the f/w timeout bit in rcfw. * During the driver unload the first command
@@ -374,7 +388,7 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc =bnxt_register_dev (en_dev, &bnxt_re_ulp_ops, rdev); + rc =bnxt_register_dev (en_dev, &bnxt_re_ulp_ops, rdev->adev);if (!rc) rdev->qplib_res.pdev = rdev->en_dev->pdev;return rc;
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 3/4 RDMA/bnxt_re: Group all operations under add_device and remove_device
In reply to Selvin XavierAdding and removing device need to be handled from multiple contexts when Firmware error recovery is supported. So group all the add and remove operations to add_device and remove_device function.
--- drivers/infiniband/hw/bnxt_re/main.c | 64 +++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 30 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +34 -30
@@ -88,6 +88,7 @@ static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev);static int bnxt_re_hwrm_qcfg (struct bnxt_re_dev *rdev,u32 *db_len,u32 *offset); +static void bnxt_re_setup_cc (struct bnxt_re_dev *rdev,bool enable);static void bnxt_re_set_db_offset (struct bnxt_re_dev *rdev) {struct bnxt_qplib_chip_ctx *cctx;
@@ -1860,6 +1861,16 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) goto re_dev_uninit; } + rdev->nb.notifier_call = bnxt_re_netdev_event; + rc =register_netdevice_notifier (&rdev->nb); +if (rc) { + rdev->nb.notifier_call =NULL ; +pr_err ("%s: Cannot register to netdevice_notifier" , +ROCE_DRV_MODULE_NAME ); +return rc; + } +bnxt_re_setup_cc (rdev, true); +return 0 ; re_dev_uninit:
@@ -1947,21 +1958,13 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier,#define BNXT_ADEV_NAME "bnxt_en" -static void bnxt_re_remove (struct auxiliary_device *adev) +static void bnxt_re_remove_device (struct bnxt_re_dev *rdev, +struct auxiliary_device *aux_dev) { -struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (adev); +struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (aux_dev);struct bnxt_en_dev *en_dev; -struct bnxt_re_dev *rdev; -mutex_lock (&bnxt_re_mutex); -if (!en_info) { -mutex_unlock (&bnxt_re_mutex); -return ; - } en_dev = en_info->en_dev; - rdev = en_info->rdev; -if (!rdev) - goto skip_remove;if (rdev->nb.notifier_call) {unregister_netdevice_notifier (&rdev->nb);
@@ -1970,13 +1973,30 @@ static void bnxt_re_remove(struct auxiliary_device *adev)/* If notifier is null, we should have already done a * clean up before coming here. */ - goto skip_remove; +return ; }bnxt_re_setup_cc (rdev, false);ib_unregister_device (&rdev->ibdev);bnxt_re_dev_uninit (rdev);ib_dealloc_device (&rdev->ibdev); -skip_remove: +} + +static void bnxt_re_remove (struct auxiliary_device *adev) +{ +struct bnxt_re_en_dev_info *en_info =auxiliary_get_drvdata (adev); +struct bnxt_en_dev *en_dev; +struct bnxt_re_dev *rdev; + +mutex_lock (&bnxt_re_mutex); +if (!en_info) { +mutex_unlock (&bnxt_re_mutex); +return ; + } + en_dev = en_info->en_dev; + rdev = en_info->rdev; + +if (rdev) +bnxt_re_remove_device (rdev, adev);kfree (en_info);mutex_unlock (&bnxt_re_mutex); }
@@ -1988,7 +2008,6 @@ static int bnxt_re_probe(struct auxiliary_device *adev, container_of(adev, struct bnxt_aux_priv, aux_dev);struct bnxt_re_en_dev_info *en_info;struct bnxt_en_dev *en_dev; -struct bnxt_re_dev *rdev;int rc; en_dev = aux_priv->edev;
@@ -2004,23 +2023,8 @@ static int bnxt_re_probe(struct auxiliary_device *adev,auxiliary_set_drvdata (adev, en_info); rc =bnxt_re_add_device (adev); -if (rc) { -mutex_unlock (&bnxt_re_mutex); -return rc; - } - - rdev = en_info->rdev; - - rdev->nb.notifier_call = bnxt_re_netdev_event; - rc =register_netdevice_notifier (&rdev->nb); -if (rc) { - rdev->nb.notifier_call =NULL ; -pr_err ("%s: Cannot register to netdevice_notifier" , -ROCE_DRV_MODULE_NAME ); +if (rc) goto err; - } - -bnxt_re_setup_cc (rdev, true);mutex_unlock (&bnxt_re_mutex);return 0 ;
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 4/4 RDMA/bnxt_re: Recover the device when FW error is detected
In reply to Selvin XavierIf the FW crashes, L2 driver gets notified and it notifies the RoCE driver. Currently driver doesn't re-initialize the device. Add support for re-initialize the RoCE device.
RoCE device is removed and re-attached in the ulp_stop and ulp_start respectively. The recovery logic expects the RoCE driver to be registered with L2 driver while its being removed. So the driver avoids unregistering with L2 driver in the recovery path.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 15 +++++++ drivers/infiniband/hw/bnxt_re/main.c | 70 +++++++++++++++++-------------- drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 + 3 files changed, 55 insertions(+), 31 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +15 -0
@@ -91,6 +91,15 @@ struct bnxt_re_ring_attr {u8 mode; }; +/* +* Data structure and defines to handle +* recovery +*/ +#define BNXT_RE_PRE_RECOVERY_REMOVE 0x1 +#define BNXT_RE_COMPLETE_REMOVE 0x2 +#define BNXT_RE_POST_RECOVERY_INIT 0x4 +#define BNXT_RE_COMPLETE_INIT 0x8 +struct bnxt_re_sqp_entries {struct bnxt_qplib_sge sge;u64 wrid;
@@ -224,4 +233,10 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) }extern const struct uapi_definition bnxt_re_uapi_defs[]; + +static inline void bnxt_re_set_pacing_dev_state (struct bnxt_re_dev *rdev) +{ + rdev->qplib_res.pacing_data->dev_err_state = +test_bit (BNXT_RE_FLAG_ERR_DEVICE_DETACHED , &rdev->flags); +}#endif
drivers/infiniband/hw/bnxt_re/main.c +39 -31
210 lines not on this page · raw message
drivers/infiniband/hw/bnxt_re/qplib_res.h +1 -0
@@ -82,6 +82,7 @@ struct bnxt_qplib_db_pacing_data {u32 fifo_room_mask;u32 fifo_room_shift;u32 grc_reg_offset; +u32 dev_err_state; };#define BNXT_QPLIB_DBR_PF_DB_OFFSET 0x10000
Signature · 1 line
2.5.5