Skip to contentSkip to conversation

fwctl/bnxt_fwctl: fwctl for Broadcom Netxtreme devices

PATCH In review Pavan Chebbi 2 participants 7 messages Mar 14 – Mar 27

Subsystems (7)
lore
Share & tools
lore

Pavan Chebbi

author maintainer
#0 lore raw

v6 cover fwctl/bnxt_fwctl: fwctl for Broadcom Netxtreme devices

Introducing bnxt_fwctl which follows along Jason's work [1]. It is an aux bus driver that enables fwctl for Broadcom NetXtreme 574xx, 575xx and 576xx series chipsets by using bnxt driver's capability to talk to devices' firmware.

The first patch moves the ULP definitions to a common place inside include/linux/bnxt/. The second and third patches refactor and extend the existing bnxt aux bus functions to be able to add more than one auxiliary device. The last three patches create an additional bnxt aux device, add bnxt_fwctl, and the documentation.

v6: In patch #4, removed user configurable 'timeout' for FW commands. It helped remove an indirection while accessing request buffer. The driver selects the appropriate timeout value based on the command type. Used kzalloc_obj where applicable. Extended the scope of en_dev_lock to cover entire rpc.

v5: In patch #4, removed support for any commands that require additional DMA buffers to be passed in the command input. Only commands that have the entire input message inlined in the request, are allowed.

v4: In patch #2 fix use after free of auxdev ID in bnxt_aux_dev_release() In patch #4, added new entries of Makefile, Kconfig and MAINTAINERS in a sorted manner.

v3: Addressed Leon's comment on patch#2 to have the lock cover entire lifespan of the auxdevice, so that it is 100pc synchronized with auxdev removing.

v2: Addressed Leon's comments as below:
Patch #2: Converted atomic_t based aux device state handling to lock based functions. Drop passing 'err' to bnxt_ulp_start() Patch #4: Zero-initialized the fw rpc variable making unwind safer

v1: Based on the feedback received on v5 here:
https://lore.kernel.org/netdev/[email protected]/ , posting this series to fwctl tree.
This series has the below changes from v5 that was posted to netdev:
Patch #2: Addressed Leon's comment for defensive prograaming style
for aux dev functions
Patch #4: Removed support for commands that don't comply with fwctl guidelines. Also addressed additional comments from Jonathan
Patch #5: Added info for example python3 program

v5: Addressed the v4's review comments as below:
Patch #2 and #3: Simplified aux bus device creation logic by having the core maintain arrays of pointers to aux devices and their contexts, thereby avoiding function calls from aux dev. [thanks Leon]
Patch #4: Used memdup_user() as suggested by cocci. Addressed additional review comments from Jonathon and Dave. Collected Rb tags from Dave.

v4: In patch #4, added the missing kfree on error for response buffer. Improved documentation in patch #5 based on comments from Dave.

v3: Addressed the review comments as below
Patch #1: Removed redundant common.h [thanks Saeed] Patch #2 and #3 merged into a single patch [thanks Jonathan] Patch #3: Addressed comments from Jonathan
Patch #4 and #5: Addressed comments from Jonathan and Dave

v2: In patch #5, fixed a sparse warning where a __le16 was degraded to an integer. Also addressed kdoc warnings for include/uapi/fwctl/bnxt.h in the same patch.

[1] https://lore.kernel.org/netdev/[email protected]/

The following are changes since commit 1c9982b4961334c1edb0745a04cabd34bc2de675

    Merge tag 'drm-fixes-2026-03-14' of https://gitlab.freedesktop.org/drm/kernel

and are available in the git repository at:
https://github.com/pavanchebbi/linux/tree/fwctl-bnxt-v6

Pavan Chebbi (5):
fwctl/bnxt_en: Move common definitions to include/linux/bnxt/ fwctl/bnxt_en: Refactor aux bus functions to be more generic fwctl/bnxt_en: Create an aux device for fwctl fwctl/bnxt_fwctl: Add bnxt fwctl device fwctl/bnxt_fwctl: Add documentation entries

 .../userspace-api/fwctl/bnxt_fwctl.rst        |  74 ++++
 Documentation/userspace-api/fwctl/fwctl.rst   |   1 +
 Documentation/userspace-api/fwctl/index.rst   |   1 +
 MAINTAINERS                                   |   6 +
 drivers/fwctl/Kconfig                         |  11 +
 drivers/fwctl/Makefile                        |   1 +
 drivers/fwctl/bnxt/Makefile                   |   4 +
 drivers/fwctl/bnxt/main.c                     | 281 ++++++++++++++
 drivers/infiniband/hw/bnxt_re/debugfs.c       |   2 +-
 drivers/infiniband/hw/bnxt_re/main.c          |   2 +-
 drivers/infiniband/hw/bnxt_re/qplib_fp.c      |   2 +-
 drivers/infiniband/hw/bnxt_re/qplib_res.h     |   2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  49 ++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  19 +-
 .../net/ethernet/broadcom/bnxt/bnxt_devlink.c |  10 +-
 .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c |   4 +-
 .../net/ethernet/broadcom/bnxt/bnxt_sriov.c   |   2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 349 +++++++++++-------
 .../bnxt_ulp.h => include/linux/bnxt/ulp.h    |  26 +-
 include/uapi/fwctl/bnxt.h                     |  26 ++
 include/uapi/fwctl/fwctl.h                    |   1 +
 21 files changed, 697 insertions(+), 176 deletions(-)
 create mode 100644 Documentation/userspace-api/fwctl/bnxt_fwctl.rst
 create mode 100644 drivers/fwctl/bnxt/Makefile
 create mode 100644 drivers/fwctl/bnxt/main.c
 rename drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h => include/linux/bnxt/ulp.h (86%)
 create mode 100644 include/uapi/fwctl/bnxt.h
Signature · 1 line
2.39.1
author maintainer
#1 lore raw

PATCH v6 1/5 fwctl/bnxt_en: Move common definitions to include/linux/bnxt/

We have common definitions that are now going to be used by more than one component outside of bnxt (bnxt_re and fwctl)

Move bnxt_ulp.h to include/linux/bnxt/ as ulp.h.

---
 drivers/infiniband/hw/bnxt_re/debugfs.c                         | 2 +-
 drivers/infiniband/hw/bnxt_re/main.c                            | 2 +-
 drivers/infiniband/hw/bnxt_re/qplib_fp.c                        | 2 +-
 drivers/infiniband/hw/bnxt_re/qplib_res.h                       | 2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.c                       | 2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c               | 2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c               | 2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c                 | 2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c                   | 2 +-
 .../broadcom/bnxt/bnxt_ulp.h => include/linux/bnxt/ulp.h        | 0
 10 files changed, 9 insertions(+), 9 deletions(-)
 rename drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h => include/linux/bnxt/ulp.h (100%)
drivers/infiniband/hw/bnxt_re/debugfs.c +1 -1
index a2ad79c3bbd0..5fed2cf66be3 100644
@@ -10,8 +10,8 @@
 #include <linux/pci.h>
 #include <linux/seq_file.h>
 #include <rdma/ib_addr.h>
+#include <linux/bnxt/ulp.h>
 
-#include "bnxt_ulp.h"
 #include "roce_hsi.h"
 #include "qplib_res.h"
 #include "qplib_sp.h"
drivers/infiniband/hw/bnxt_re/main.c +1 -1
index b576f05e3b26..47afccddf55e 100644
@@ -55,8 +55,8 @@
 #include <rdma/ib_umem.h>
 #include <rdma/ib_addr.h>
 #include <linux/hashtable.h>
+#include <linux/bnxt/ulp.h>
 
-#include "bnxt_ulp.h"
 #include "roce_hsi.h"
 #include "qplib_res.h"
 #include "qplib_sp.h"
drivers/infiniband/hw/bnxt_re/qplib_fp.c +1 -1
index 2d7932b3c492..b4c7b8f582ba 100644
@@ -46,6 +46,7 @@
 #include <linux/delay.h>
 #include <linux/prefetch.h>
 #include <linux/if_ether.h>
+#include <linux/bnxt/ulp.h>
 #include <rdma/ib_mad.h>
 
 #include "roce_hsi.h"
@@ -55,7 +56,6 @@
 #include "qplib_sp.h"
 #include "qplib_fp.h"
 #include <rdma/ib_addr.h>
-#include "bnxt_ulp.h"
 #include "bnxt_re.h"
 #include "ib_verbs.h"
 
drivers/infiniband/hw/bnxt_re/qplib_res.h +1 -1
index 9a5dcf97b6f4..0a4a03efeb0b 100644
@@ -39,7 +39,7 @@
 #ifndef __BNXT_QPLIB_RES_H__
 #define __BNXT_QPLIB_RES_H__
 
-#include "bnxt_ulp.h"
+#include <linux/bnxt/ulp.h>
 
 extern const struct bnxt_qplib_gid bnxt_qplib_gid_zero;
 
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -1
index c426a41c3663..618c6b66458b 100644
@@ -59,10 +59,10 @@
 #include <net/netdev_rx_queue.h>
 #include <linux/pci-tph.h>
 #include <linux/bnxt/hsi.h>
+#include <linux/bnxt/ulp.h>
 
 #include "bnxt.h"
 #include "bnxt_hwrm.h"
-#include "bnxt_ulp.h"
 #include "bnxt_sriov.h"
 #include "bnxt_ethtool.h"
 #include "bnxt_dcb.h"
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +1 -1
index 15de802bbac4..230cd95d30a2 100644
@@ -13,12 +13,12 @@
 #include <net/devlink.h>
 #include <net/netdev_lock.h>
 #include <linux/bnxt/hsi.h>
+#include <linux/bnxt/ulp.h>
 #include "bnxt.h"
 #include "bnxt_hwrm.h"
 #include "bnxt_vfr.h"
 #include "bnxt_devlink.h"
 #include "bnxt_ethtool.h"
-#include "bnxt_ulp.h"
 #include "bnxt_ptp.h"
 #include "bnxt_coredump.h"
 #include "bnxt_nvm_defs.h"
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +1 -1
index ba47e8294fff..043e872a8880 100644
@@ -27,9 +27,9 @@
 #include <net/netdev_queues.h>
 #include <net/netlink.h>
 #include <linux/bnxt/hsi.h>
+#include <linux/bnxt/ulp.h>
 #include "bnxt.h"
 #include "bnxt_hwrm.h"
-#include "bnxt_ulp.h"
 #include "bnxt_xdp.h"
 #include "bnxt_ptp.h"
 #include "bnxt_ethtool.h"
drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c +1 -1
index 7f9829287c49..edcc002e4ca3 100644
@@ -17,9 +17,9 @@
 #include <linux/etherdevice.h>
 #include <net/dcbnl.h>
 #include <linux/bnxt/hsi.h>
+#include <linux/bnxt/ulp.h>
 #include "bnxt.h"
 #include "bnxt_hwrm.h"
-#include "bnxt_ulp.h"
 #include "bnxt_sriov.h"
 #include "bnxt_vfr.h"
 #include "bnxt_ethtool.h"
drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +1 -1
index e1e82a72cf1b..11ced44ead29 100644
@@ -22,10 +22,10 @@
 #include <linux/auxiliary_bus.h>
 #include <net/netdev_lock.h>
 #include <linux/bnxt/hsi.h>
+#include <linux/bnxt/ulp.h>
 
 #include "bnxt.h"
 #include "bnxt_hwrm.h"
-#include "bnxt_ulp.h"
 
 static DEFINE_IDA(bnxt_aux_dev_ids);
 
include/linux/bnxt/ulp.h renamed from drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
rename to include/linux/bnxt/ulp.h
Signature · 1 line
2.39.1
author maintainer
#2 lore raw

PATCH v6 2/5 fwctl/bnxt_en: Refactor aux bus functions to be more generic

In reply to Pavan Chebbi

Up until now there was only one auxiliary device that bnxt created and that was for RoCE driver. bnxt fwctl is also going to use an aux bus device that bnxt should create. This requires some nomenclature changes and refactoring of the existing bnxt aux dev functions.

Convert 'aux_priv' and 'edev' members of struct bnxt into arrays where each element contains supported auxbus device's data. Move struct bnxt_aux_priv from bnxt.h to ulp.h because that is where it belongs. Make aux bus init/uninit/add/del functions more generic which will loop through all the aux device types. Make bnxt_ulp_start/stop functions (the only other common functions applicable to any aux device) loop through the aux devices to update their config and states. Make callers of bnxt_ulp_start() call it only when there are no errors.

Also, as an improvement in code, bnxt_register_dev() can skip unnecessary dereferencing of edev from bp, instead use the edev pointer from the function parameter.

Future patches will reuse these functions to add an aux bus device for fwctl.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  47 ++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  19 +-
 .../net/ethernet/broadcom/bnxt/bnxt_devlink.c |   8 +-
 .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c |   2 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 339 +++++++++++-------
 include/linux/bnxt/ulp.h                      |  25 +-
 6 files changed, 273 insertions(+), 167 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +28 -19
index 618c6b66458b..309be8f62095 100644
@@ -6894,7 +6894,8 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, struct bnxt_vnic_info *vnic)
 #endif
 	if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan)
 		req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE);
-	if (vnic->vnic_id == BNXT_VNIC_DEFAULT && bnxt_ulp_registered(bp->edev))
+	if (vnic->vnic_id == BNXT_VNIC_DEFAULT &&
+	    bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA]))
 		req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp));
 
 	return hwrm_req_send(bp, req);
@@ -8029,6 +8030,7 @@ static int bnxt_get_avail_msix(struct bnxt *bp, int num);
 
 static int __bnxt_reserve_rings(struct bnxt *bp)
 {
+	struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
 	struct bnxt_hw_rings hwr = {0};
 	int rx_rings, old_rx_rings, rc;
 	int cp = bp->cp_nr_rings;
@@ -8039,7 +8041,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
 	if (!bnxt_need_reserve_rings(bp))
 		return 0;
 
-	if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+	if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
 		ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
 		if (!ulp_msix)
 			bnxt_set_ulp_stat_ctxs(bp, 0);
@@ -8090,8 +8092,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
 	}
 	rx_rings = min_t(int, rx_rings, hwr.grp);
 	hwr.cp = min_t(int, hwr.cp, bp->cp_nr_rings);
-	if (bnxt_ulp_registered(bp->edev) &&
-	    hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
+	if (bnxt_ulp_registered(edev) && hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
 		hwr.stat -= bnxt_get_ulp_stat_ctxs(bp);
 	hwr.cp = min_t(int, hwr.cp, hwr.stat);
 	rc = bnxt_trim_rings(bp, &rx_rings, &hwr.tx, hwr.cp, sh);
@@ -8135,7 +8136,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
 	    !netif_is_rxfh_configured(bp->dev))
 		bnxt_set_dflt_rss_indir_tbl(bp, NULL);
 
-	if (!bnxt_ulp_registered(bp->edev) && BNXT_NEW_RM(bp)) {
+	if (!bnxt_ulp_registered(edev) && BNXT_NEW_RM(bp)) {
 		int resv_msix, resv_ctx, ulp_ctxs;
 		struct bnxt_hw_resc *hw_resc;
 
@@ -11492,6 +11493,7 @@ static void bnxt_clear_int_mode(struct bnxt *bp)
 
 int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
 {
+	struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
 	bool irq_cleared = false;
 	bool irq_change = false;
 	int tcs = bp->num_tc;
@@ -11501,7 +11503,7 @@ int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
 	if (!bnxt_need_reserve_rings(bp))
 		return 0;
 
-	if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+	if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
 		int ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
 
 		if (ulp_msix > bp->ulp_num_msix_want)
@@ -14591,7 +14593,7 @@ static void bnxt_fw_echo_reply(struct bnxt *bp)
 static void bnxt_ulp_restart(struct bnxt *bp)
 {
 	bnxt_ulp_stop(bp);
-	bnxt_ulp_start(bp, 0);
+	bnxt_ulp_start(bp);
 }
 
 static void bnxt_sp_task(struct work_struct *work)
@@ -14748,7 +14750,7 @@ int bnxt_check_rings(struct bnxt *bp, int tx, int rx, bool sh, int tcs,
 		hwr.cp_p5 = hwr.tx + rx;
 	rc = bnxt_hwrm_check_rings(bp, &hwr);
 	if (!rc && pci_msix_can_alloc_dyn(bp->pdev)) {
-		if (!bnxt_ulp_registered(bp->edev)) {
+		if (!bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
 			hwr.cp += bnxt_get_ulp_msix_num(bp);
 			hwr.cp = min_t(int, hwr.cp, bnxt_get_max_func_irqs(bp));
 		}
@@ -15268,7 +15270,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 			bnxt_dl_health_fw_status_update(bp, true);
 		}
 		netdev_unlock(bp->dev);
-		bnxt_ulp_start(bp, 0);
+		bnxt_ulp_start(bp);
 		bnxt_reenable_sriov(bp);
 		netdev_lock(bp->dev);
 		bnxt_vf_reps_alloc(bp);
@@ -15290,7 +15292,8 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 	bnxt_fw_reset_abort(bp, rc);
 	netdev_unlock(bp->dev);
 ulp_start:
-	bnxt_ulp_start(bp, rc);
+	if (!rc)
+		bnxt_ulp_start(bp);
 }
 
 static int bnxt_init_board(struct pci_dev *pdev, struct net_device *dev)
@@ -16325,12 +16328,13 @@ static void bnxt_remove_one(struct pci_dev *pdev)
 	if (BNXT_PF(bp))
 		__bnxt_sriov_disable(bp);
 
-	bnxt_rdma_aux_device_del(bp);
+	bnxt_aux_devices_del(bp);
 
 	unregister_netdev(dev);
 	bnxt_ptp_clear(bp);
 
-	bnxt_rdma_aux_device_uninit(bp);
+	bnxt_aux_devices_uninit(bp);
+	bnxt_auxdev_id_free(bp, bp->auxdev_id);
 
 	bnxt_free_l2_filters(bp, true);
 	bnxt_free_ntp_fltrs(bp, true);
@@ -16916,7 +16920,9 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	bnxt_set_tpa_flags(bp);
 	bnxt_init_ring_params(bp);
 	bnxt_set_ring_params(bp);
-	bnxt_rdma_aux_device_init(bp);
+	mutex_init(&bp->auxdev_lock);
+	if (!bnxt_auxdev_id_alloc(bp))
+		bnxt_aux_devices_init(bp);
 	rc = bnxt_set_dflt_rings(bp, true);
 	if (rc) {
 		if (BNXT_VF(bp) && rc == -ENODEV) {
@@ -16981,7 +16987,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	bnxt_dl_fw_reporters_create(bp);
 
-	bnxt_rdma_aux_device_add(bp);
+	bnxt_aux_devices_add(bp);
 
 	bnxt_print_device_info(bp);
 
@@ -16989,7 +16995,8 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	return 0;
 init_err_cleanup:
-	bnxt_rdma_aux_device_uninit(bp);
+	bnxt_aux_devices_uninit(bp);
+	bnxt_auxdev_id_free(bp, bp->auxdev_id);
 	bnxt_dl_unregister(bp);
 init_err_dl:
 	bnxt_shutdown_tc(bp);
@@ -17123,9 +17130,10 @@ static int bnxt_resume(struct device *device)
 
 resume_exit:
 	netdev_unlock(bp->dev);
-	bnxt_ulp_start(bp, rc);
-	if (!rc)
+	if (!rc) {
+		bnxt_ulp_start(bp);
 		bnxt_reenable_sriov(bp);
+	}
 	return rc;
 }
 
@@ -17305,9 +17313,10 @@ static void bnxt_io_resume(struct pci_dev *pdev)
 		netif_device_attach(netdev);
 
 	netdev_unlock(netdev);
-	bnxt_ulp_start(bp, err);
-	if (!err)
+	if (!err) {
+		bnxt_ulp_start(bp);
 		bnxt_reenable_sriov(bp);
+	}
 }
 
 static const struct pci_error_handlers bnxt_err_handler = {
drivers/net/ethernet/broadcom/bnxt/bnxt.h +10 -9
index 9a41b9e0423c..42955490942f 100644
@@ -24,12 +24,12 @@
 #include <linux/interrupt.h>
 #include <linux/rhashtable.h>
 #include <linux/crash_dump.h>
-#include <linux/auxiliary_bus.h>
 #include <net/devlink.h>
 #include <net/dst_metadata.h>
 #include <net/xdp.h>
 #include <linux/dim.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/bnxt/ulp.h>
 #ifdef CONFIG_TEE_BNXT_FW
 #include <linux/firmware/broadcom/tee_bnxt_fw.h>
 #endif
@@ -2085,12 +2085,6 @@ struct bnxt_fw_health {
 #define BNXT_FW_IF_RETRY		10
 #define BNXT_FW_SLOT_RESET_RETRY	4
 
-struct bnxt_aux_priv {
-	struct auxiliary_device aux_dev;
-	struct bnxt_en_dev *edev;
-	int id;
-};
-
 enum board_idx {
 	BCM57301,
 	BCM57302,
@@ -2350,8 +2344,8 @@ struct bnxt {
 #define BNXT_CHIP_P5_AND_MINUS(bp)		\
 	(BNXT_CHIP_P3(bp) || BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp))
 
-	struct bnxt_aux_priv	*aux_priv;
-	struct bnxt_en_dev	*edev;
+	struct bnxt_aux_priv	*aux_priv[__BNXT_AUXDEV_MAX];
+	struct bnxt_en_dev	*edev[__BNXT_AUXDEV_MAX];
 
 	struct bnxt_napi	**bnapi;
 
@@ -2763,6 +2757,13 @@ struct bnxt {
 	struct bnxt_ctx_pg_info	*fw_crash_mem;
 	u32			fw_crash_len;
 	struct bnxt_bs_trace_info bs_trace[BNXT_TRACE_MAX];
+	int			auxdev_id;
+	/* synchronize validity checks of available aux devices */
+	struct mutex		auxdev_lock;
+	u8			auxdev_state[__BNXT_AUXDEV_MAX];
+#define	BNXT_ADEV_STATE_NONE	0
+#define	BNXT_ADEV_STATE_INIT	1
+#define	BNXT_ADEV_STATE_ADD	2
 };
 
 #define BNXT_NUM_RX_RING_STATS			8
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +4 -4
index 230cd95d30a2..835f2b413931 100644
@@ -440,13 +440,13 @@ static int bnxt_dl_reload_down(struct devlink *dl, bool netns_change,
 					   "reload is unsupported while VFs are allocated or being configured");
 			netdev_unlock(bp->dev);
 			rtnl_unlock();
-			bnxt_ulp_start(bp, 0);
+			bnxt_ulp_start(bp);
 			return -EOPNOTSUPP;
 		}
 		if (bp->dev->reg_state == NETREG_UNREGISTERED) {
 			netdev_unlock(bp->dev);
 			rtnl_unlock();
-			bnxt_ulp_start(bp, 0);
+			bnxt_ulp_start(bp);
 			return -ENODEV;
 		}
 		if (netif_running(bp->dev))
@@ -578,8 +578,8 @@ static int bnxt_dl_reload_up(struct devlink *dl, enum devlink_reload_action acti
 	}
 	netdev_unlock(bp->dev);
 	rtnl_unlock();
-	if (action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
-		bnxt_ulp_start(bp, rc);
+	if (!rc && action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
+		bnxt_ulp_start(bp);
 	return rc;
 }
 
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +1 -1
index 043e872a8880..046bfae6e610 100644
@@ -5212,7 +5212,7 @@ static void bnxt_self_test(struct net_device *dev, struct ethtool_test *etest,
 
 	memset(buf, 0, sizeof(u64) * bp->num_tests);
 	if (etest->flags & ETH_TEST_FL_OFFLINE &&
-	    bnxt_ulp_registered(bp->edev)) {
+	    bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
 		etest->flags |= ETH_TEST_FL_FAILED;
 		netdev_warn(dev, "Offline tests cannot be run with RoCE driver loaded\n");
 		return;
drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +210 -129

499 lines not on this page · raw message

include/linux/bnxt/ulp.h +20 -5
index 3c5b8a53f715..1a4643c46f86 100644
@@ -10,6 +10,8 @@
 #ifndef BNXT_ULP_H
 #define BNXT_ULP_H
 
+#include <linux/auxiliary_bus.h>
+
 #define BNXT_MIN_ROCE_CP_RINGS	2
 #define BNXT_MIN_ROCE_STAT_CTXS	1
 
@@ -20,6 +22,17 @@
 struct hwrm_async_event_cmpl;
 struct bnxt;
 
+enum bnxt_auxdev_type {
+	BNXT_AUXDEV_RDMA = 0,
+	__BNXT_AUXDEV_MAX
+};
+
+struct bnxt_aux_priv {
+	struct auxiliary_device aux_dev;
+	struct bnxt_en_dev *edev;
+	int id;
+};
+
 struct bnxt_msix_entry {
 	u32	vector;
 	u32	ring_idx;
@@ -110,19 +123,21 @@ void bnxt_set_ulp_stat_ctxs(struct bnxt *bp, int num_ctxs);
 int bnxt_get_ulp_stat_ctxs_in_use(struct bnxt *bp);
 void bnxt_set_dflt_ulp_stat_ctxs(struct bnxt *bp);
 void bnxt_ulp_stop(struct bnxt *bp);
-void bnxt_ulp_start(struct bnxt *bp, int err);
+void bnxt_ulp_start(struct bnxt *bp);
 void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs);
 void bnxt_ulp_irq_stop(struct bnxt *bp);
 void bnxt_ulp_irq_restart(struct bnxt *bp, int err);
 void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl);
-void bnxt_rdma_aux_device_uninit(struct bnxt *bp);
-void bnxt_rdma_aux_device_del(struct bnxt *bp);
-void bnxt_rdma_aux_device_add(struct bnxt *bp);
-void bnxt_rdma_aux_device_init(struct bnxt *bp);
+void bnxt_aux_devices_uninit(struct bnxt *bp);
+void bnxt_aux_devices_del(struct bnxt *bp);
+void bnxt_aux_devices_add(struct bnxt *bp);
+void bnxt_aux_devices_init(struct bnxt *bp);
 int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops,
 		      void *handle);
 void bnxt_unregister_dev(struct bnxt_en_dev *edev);
 int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg);
 void bnxt_register_async_events(struct bnxt_en_dev *edev,
 				unsigned long *events_bmap, u16 max_id);
+int bnxt_auxdev_id_alloc(struct bnxt *bp);
+void bnxt_auxdev_id_free(struct bnxt *bp, int id);
 #endif
Signature · 1 line
2.39.1
author maintainer
#3 lore raw

PATCH v6 3/5 fwctl/bnxt_en: Create an aux device for fwctl

In reply to Pavan Chebbi

Create an additional auxiliary device to support fwctl. The next patch will create bnxt_fwctl and bind to this device.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 12 ++++++++++--
 include/linux/bnxt/ulp.h                      |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +10 -2
index 3705ef3a75c9..052bf69cfa4c 100644
@@ -50,6 +50,8 @@ static bool bnxt_auxdev_is_active(struct bnxt *bp, int idx)
 
 static struct bnxt_aux_device bnxt_aux_devices[__BNXT_AUXDEV_MAX] = {{
 	.name		= "rdma",
+}, {
+	.name		= "fwctl",
 }};
 
 static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent)
@@ -278,6 +280,11 @@ void bnxt_ulp_stop(struct bnxt *bp)
 		aux_priv = bp->aux_priv[i];
 		edev = bp->edev[i];
 		mutex_lock(&edev->en_dev_lock);
+		if (i == BNXT_AUXDEV_FWCTL) {
+			edev->flags |= BNXT_EN_FLAG_ULP_STOPPED;
+			mutex_unlock(&edev->en_dev_lock);
+			continue;
+		}
 		if (!bnxt_ulp_registered(edev) ||
 		    (edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
 			mutex_unlock(&edev->en_dev_lock);
@@ -316,7 +323,7 @@ void bnxt_ulp_start(struct bnxt *bp)
 		aux_priv = bp->aux_priv[i];
 		edev = bp->edev[i];
 		mutex_lock(&edev->en_dev_lock);
-		if (!bnxt_ulp_registered(edev) ||
+		if (i == BNXT_AUXDEV_FWCTL || !bnxt_ulp_registered(edev) ||
 		    !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
 			goto clear_flag_continue;
 		}
@@ -520,7 +527,8 @@ void bnxt_aux_devices_add(struct bnxt *bp)
 			aux_dev = &bp->aux_priv[idx]->aux_dev;
 			rc = auxiliary_device_add(aux_dev);
 			if (rc) {
-				netdev_warn(bp->dev, "Failed to add auxiliary device for ROCE\n");
+				netdev_warn(bp->dev, "Failed to add auxiliary device for auxdev type %d\n",
+					    idx);
 				auxiliary_device_uninit(aux_dev);
 				if (idx == BNXT_AUXDEV_RDMA)
 					bp->flags &= ~BNXT_FLAG_ROCE_CAP;
include/linux/bnxt/ulp.h +1 -0
index 1a4643c46f86..0851ad3394b0 100644
@@ -24,6 +24,7 @@ struct bnxt;
 
 enum bnxt_auxdev_type {
 	BNXT_AUXDEV_RDMA = 0,
+	BNXT_AUXDEV_FWCTL,
 	__BNXT_AUXDEV_MAX
 };
 
Signature · 1 line
2.39.1
author maintainer
#4 lore raw

PATCH v6 4/5 fwctl/bnxt_fwctl: Add bnxt fwctl device

In reply to Pavan Chebbi

Create bnxt_fwctl device. This will bind to bnxt's aux device. On the upper edge, it will register with the fwctl subsystem. It will make use of bnxt's ULP functions to send FW commands.

---
 MAINTAINERS                 |   6 +
 drivers/fwctl/Kconfig       |  11 ++
 drivers/fwctl/Makefile      |   1 +
 drivers/fwctl/bnxt/Makefile |   4 +
 drivers/fwctl/bnxt/main.c   | 281 ++++++++++++++++++++++++++++++++++++
 include/uapi/fwctl/bnxt.h   |  26 ++++
 include/uapi/fwctl/fwctl.h  |   1 +
 7 files changed, 330 insertions(+)
 create mode 100644 drivers/fwctl/bnxt/Makefile
 create mode 100644 drivers/fwctl/bnxt/main.c
 create mode 100644 include/uapi/fwctl/bnxt.h
MAINTAINERS +6 -0
index 61bf550fd37c..ca5e67d281d4 100644
@@ -10535,6 +10535,12 @@ F:	drivers/fwctl/
 F:	include/linux/fwctl.h
 F:	include/uapi/fwctl/
 
+FWCTL BNXT DRIVER
+M:	Pavan Chebbi <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/fwctl/bnxt/
+
 FWCTL MLX5 DRIVER
 M:	Saeed Mahameed <[email protected]>
 R:	Itay Avraham <[email protected]>
drivers/fwctl/Kconfig +11 -0
index b5583b12a011..d1b1925bdaec 100644
@@ -9,6 +9,17 @@ menuconfig FWCTL
 	  fit neatly into an existing subsystem.
 
 if FWCTL
+config FWCTL_BNXT
+	tristate "bnxt control fwctl driver"
+	depends on BNXT
+	help
+	  BNXT provides interface for the user process to access the debug and
+	  configuration registers of the Broadcom NIC hardware family.
+	  This will allow configuration and debug tools to work out of the box on
+	  mainstream kernel.
+
+	  If you don't know what to do here, say N.
+
 config FWCTL_MLX5
 	tristate "mlx5 ConnectX control fwctl driver"
 	depends on MLX5_CORE
drivers/fwctl/Makefile +1 -0
index c093b5f661d6..692e4b8d7beb 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_FWCTL) += fwctl.o
+obj-$(CONFIG_FWCTL_BNXT) += bnxt/
 obj-$(CONFIG_FWCTL_MLX5) += mlx5/
 obj-$(CONFIG_FWCTL_PDS) += pds/
 
drivers/fwctl/bnxt/Makefile new file +4 -0
new file mode 100644
index 000000000000..b47172761f1e
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_FWCTL_BNXT) += bnxt_fwctl.o
+
+bnxt_fwctl-y += main.o
drivers/fwctl/bnxt/main.c new file +281 -0

281 lines not on this page · raw message

include/uapi/fwctl/bnxt.h new file +26 -0
new file mode 100644
index 000000000000..32e0bfb9a836
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2026, Broadcom Inc
+ */
+
+#ifndef _UAPI_FWCTL_BNXT_H_
+#define _UAPI_FWCTL_BNXT_H_
+
+#include <linux/types.h>
+
+enum fwctl_bnxt_commands {
+	FWCTL_BNXT_INLINE_COMMANDS = 0,
+	FWCTL_BNXT_QUERY_COMMANDS,
+	FWCTL_BNXT_SEND_COMMANDS,
+};
+
+/**
+ * struct fwctl_info_bnxt - ioctl(FWCTL_INFO) out_device_data
+ * @uctx_caps: The command capabilities driver accepts.
+ *
+ * Return basic information about the FW interface available.
+ */
+struct fwctl_info_bnxt {
+	__u32 uctx_caps;
+};
+#endif
include/uapi/fwctl/fwctl.h +1 -0
index 716ac0eee42d..2d6d4049c205 100644
@@ -44,6 +44,7 @@ enum fwctl_device_type {
 	FWCTL_DEVICE_TYPE_ERROR = 0,
 	FWCTL_DEVICE_TYPE_MLX5 = 1,
 	FWCTL_DEVICE_TYPE_CXL = 2,
+	FWCTL_DEVICE_TYPE_BNXT = 3,
 	FWCTL_DEVICE_TYPE_PDS = 4,
 };
 
Signature · 1 line
2.39.1
author maintainer
#5 lore raw

PATCH v6 5/5 fwctl/bnxt_fwctl: Add documentation entries

In reply to Pavan Chebbi

Add bnxt_fwctl to the driver and fwctl documentation pages.

---
 .../userspace-api/fwctl/bnxt_fwctl.rst        | 74 +++++++++++++++++++
 Documentation/userspace-api/fwctl/fwctl.rst   |  1 +
 Documentation/userspace-api/fwctl/index.rst   |  1 +
 3 files changed, 76 insertions(+)
 create mode 100644 Documentation/userspace-api/fwctl/bnxt_fwctl.rst
Documentation/userspace-api/fwctl/bnxt_fwctl.rst new file +74 -0
new file mode 100644
index 000000000000..97c9b095cf21
@@ -0,0 +1,74 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================
+fwctl bnxt driver
+=================
+
+:Author: Pavan Chebbi
+
+Overview
+========
+
+BNXT driver makes a fwctl service available through an auxiliary_device.
+The bnxt_fwctl driver binds to this device and registers itself with the
+fwctl subsystem.
+
+The bnxt_fwctl driver is agnostic to the device firmware internals. It
+uses the Upper Layer Protocol (ULP) conduit provided by bnxt to send
+HardWare Resource Manager (HWRM) commands to firmware.
+
+These commands can query or change firmware driven device configurations
+and read/write registers that are useful for debugging.
+
+bnxt_fwctl User API
+===================
+
+Each RPC request contains the HWRM input structure in the fwctl_rpc
+'in' buffer while 'out' will contain the response.
+
+A typical user application can send a FWCTL_INFO command using ioctl()
+to discover bnxt_fwctl's RPC capabilities as shown below:
+
+        ioctl(fd, FWCTL_INFO, &fwctl_info_msg);
+
+where fwctl_info_msg (of type struct fwctl_info) describes bnxt_info_msg
+(of type struct fwctl_info_bnxt). fwctl_info_msg is set up as follows:
+
+        size = sizeof(struct fwctl_info);
+        flags = 0;
+        device_data_len = sizeof(bnxt_info_msg);
+        out_device_data = (__aligned_u64)&bnxt_info_msg;
+
+The uctx_caps of bnxt_info_msg represents the capabilities as described
+in fwctl_bnxt_commands of include/uapi/fwctl/bnxt.h
+
+The FW RPC itself, FWCTL_RPC can be sent using ioctl() as:
+
+        ioctl(fd, FWCTL_RPC, &fwctl_rpc_msg);
+
+where fwctl_rpc_msg (of type struct fwctl_rpc) carries the HWRM command
+in its 'in' buffer. The HWRM input structures are described in
+include/linux/bnxt/hsi.h. An example for HWRM_VER_GET is shown below:
+
+        struct hwrm_ver_get_output resp;
+        struct fwctl_rpc fwctl_rpc_msg;
+        struct hwrm_ver_get_input req;
+
+        req.req_type = HWRM_VER_GET;
+        req.hwrm_intf_maj = HWRM_VERSION_MAJOR;
+        req.hwrm_intf_min = HWRM_VERSION_MINOR;
+        req.hwrm_intf_upd = HWRM_VERSION_UPDATE;
+        req.cmpl_ring = -1;
+        req.target_id = -1;
+
+        fwctl_rpc_msg.size = sizeof(struct fwctl_rpc);
+        fwctl_rpc_msg.scope = FWCTL_RPC_DEBUG_READ_ONLY;
+        fwctl_rpc_msg.in_len = sizeof(req);
+        fwctl_rpc_msg.out_len = sizeof(resp);
+        fwctl_rpc_msg.in = (__aligned_u64)&req;
+        fwctl_rpc_msg.out = (__aligned_u64)&resp;
+
+An example python3 program that can exercise this interface can be found in
+the following git repository:
+
+https://github.com/Broadcom/fwctl-tools
Documentation/userspace-api/fwctl/fwctl.rst +1 -0
index a74eab8d14c6..826817bfd54d 100644
@@ -148,6 +148,7 @@ area resulting in clashes will be resolved in favour of a kernel implementation.
 fwctl User API
 ==============
 
+.. kernel-doc:: include/uapi/fwctl/bnxt.h
 .. kernel-doc:: include/uapi/fwctl/fwctl.h
 .. kernel-doc:: include/uapi/fwctl/mlx5.h
 .. kernel-doc:: include/uapi/fwctl/pds.h
Documentation/userspace-api/fwctl/index.rst +1 -0
index 316ac456ad3b..8062f7629654 100644
@@ -10,5 +10,6 @@ to securely construct and execute RPCs inside device firmware.
    :maxdepth: 1
 
    fwctl
+   bnxt_fwctl
    fwctl-cxl
    pds_fwctl
Signature · 1 line
2.39.1
#6 lore raw
In reply to Pavan Chebbi
Quoted 11 lines · Pavan Chebbi

Introducing bnxt_fwctl which follows along Jason's work [1]. It is an aux bus driver that enables fwctl for Broadcom NetXtreme 574xx, 575xx and 576xx series chipsets by using bnxt driver's capability to talk to devices' firmware.

Pavan Chebbi (5):
fwctl/bnxt_en: Move common definitions to include/linux/bnxt/ fwctl/bnxt_en: Refactor aux bus functions to be more generic fwctl/bnxt_en: Create an aux device for fwctl fwctl/bnxt_fwctl: Add bnxt fwctl device fwctl/bnxt_fwctl: Add documentation entries

Appled, thanks

Jason