pds_core driver
Subsystems (4)
v9 cover pds_core driver
Summary:
--------
This patchset implements a new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus and through a couple of EXPORTed functions for use initially in VFio and vDPA feature specific drivers.
To keep this patchset to a manageable size, the pds_vdpa and pds_vfio drivers have been split out into their own patchsets to be reviewed separately.
Detail:
-------
AMD/Pensando is making available a new set of devices for supporting vDPA, VFio, and potentially other features in the Distributed Services Card (DSC). These features are implemented through a PF that serves as a Core device for controlling and configuring its VF devices. These VF devices have separate drivers that use the auxiliary_bus to work through the Core device as the control path.
Currently, the DSC supports standard ethernet operations using the ionic driver. This is not replaced by the Core-based devices - these new devices are in addition to the existing Ethernet device. Typical DSC configurations will include both PDS devices and Ionic Eth devices. However, there is a potential future path for ethernet services to come through this device as well.
The Core device is a new PCI PF/VF device managed by a new driver 'pds_core'. The PF device has access to an admin queue for configuring the services used by the VFs, and sets up auxiliary_bus devices for each vDPA VF for communicating with the drivers for the vDPA devices. The VFs may be for VFio or vDPA, and other services in the future; these VF types are selected as part of the DSC internal FW configurations, which is out of the scope of this patchset.
When the vDPA support set is enabled in the core PF through its devlink param, auxiliary_bus devices are created for each VF that supports the feature. The vDPA driver then connects to and uses this auxiliary_device to do control path configuration through the PF device. This can then be used with the vdpa kernel module to provide devices for virtio_vdpa kernel module for host interfaces, or vhost_vdpa kernel module for interfaces exported into your favorite VM.
A cheap ASCII diagram of a vDPA instance looks something like this:
,----------.
| vdpa |
'----------'
| ||
ctl data
| ||
.----------. ||
| pds_vdpa | ||
'----------' ||
| ||
pds_core.vDPA.1 ||
| ||
.---------------. ||
| pds_core | ||
'---------------' ||
|| || ||
09:00.0 09:00.1
== PCI ============================================
|| ||
.----------. .----------.
,------| PF |---| VF |-------,
| '----------' '----------' |
| DSC |
| |
------------------------------------------Changes:
v9:
- change pdsc field name id to uid to clarify the unique id used for aux device
- remove unnecessary pf->state and other checks in aux device creation
- hardcode fw slotnames for devlink info, don't use strings from FW
- handle errors from PDS_CORE_CMD_INIT devcmd call
- tighten up health thread use of config_lock
- remove pdsc_queue_health_check() layer over queuing health check
- start pds_core.rst file in first patch, add to it incrementally
- give more user interaction info in commit messages
- removed a few more extraneous includes
v8:
- fixed deadlock problem, use devl_health_reporter_destroy() when devlink is locked
- don't clear client_id until after auxiliary_device_uninit()
v7:
- use explicit devlink locking and devl_* APIs
- move some of devlink setup logic into probe and remove
- use debugfs_create_u{type}() for state and queue head and tail
- add include for linux/vmalloc.h
v6:
- removed version.h include noticed by kernel test robot's version check
- fixed up the more egregious checkpatch line length complaints
- make sure pdsc_auxbus_dev_register() checks padev pointer errcode
v5:
- added devlink health reporter for FW issues
- removed asic_type, asic_rev, serial_num, fw_version from debugfs as they are available through other means
- trimed OS info in pdsc_identify(), we don't need to send that much info to the FW
- removed reg/unreg from auxbus client API, they are now in the core when VF is started
- removed need for pdsc definition in client by simplifying the padev to only carry struct pci_dev pointers rather than full struct pdsc to the pf and vf
- removed the unused pdsc argument in pdsc_notify()
- moved include/linux/pds/pds_core.h to driver/../pds_core/core.h
- restored a few pds_core_if.h interface values and structs that are shared with FW source
- moved final config_lock unlock to before tear down of timer and workqueue to be sure there are no deadlocks while waiting for any stragglers
- changed use of PAGE_SIZE to local PDS_PAGE_SIZE to keep with FW layout needs without regard to kernel PAGE_SIZE configuration
- removed the redundant *adminqcq argument from pdsc_adminq_post()
v4:
- reworked to attach to both Core PF and vDPA VF PCI devices
- now creates auxiliary_device as part of each VF PCI probe, removes them on PCI remove
- auxiliary devices now use simple unique id rather than PCI address for identifier
- replaced home-grown event publishing with kernel-based notifier service
- dropped live_migration parameter, not needed when not creating aux device for it
- replaced devm_* functions with traditional interfaces
- added MAINTAINERS entry
- removed lingering traces of set/get_vf attribute adminq commands
- trimmed some include lists
- cleaned a kernel test robot complaint about a stray unused variable
Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
v3:
- changed names from "pensando" to "amd" and updated copyright strings
- dropped the DEVLINK_PARAM_GENERIC_ID_FW_BANK for future development
- changed the auxiliary device creation to be triggered by the PCI bus event BOUND_DRIVER, and torn down at UNBIND_DRIVER in order to properly handle users using the sysfs bind/unbind functions
- dropped some noisy log messages
- rebased to current net-next
RFC to v2:
- added separate devlink param patches for DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION and DEVLINK_PARAM_GENERIC_ID_FW_BANK, and dropped the driver specific implementations
- updated descriptions for the new devlink parameters
- dropped netdev support
- dropped vDPA patches, will followup later
- separated fw update and fw bank select into their own patches
RFC:
Shannon Nelson (14):
pds_core: initial framework for pds_core PF driver
pds_core: add devcmd device interfaces
pds_core: health timer and workqueue
pds_core: add devlink health facilities
pds_core: set up device and adminq
pds_core: Add adminq processing and commands
pds_core: add FW update feature to devlink
pds_core: set up the VIF definitions and defaults
pds_core: add initial VF device handling
pds_core: add auxiliary_bus devices
pds_core: devlink params for enabling VIF support
pds_core: add the aux client API
pds_core: publish events to the clients
pds_core: Kconfig and pds_core.rst
.../device_drivers/ethernet/amd/pds_core.rst | 144 ++++ .../device_drivers/ethernet/index.rst | 1 + MAINTAINERS | 9 + drivers/net/ethernet/amd/Kconfig | 12 + drivers/net/ethernet/amd/Makefile | 1 + drivers/net/ethernet/amd/pds_core/Makefile | 14 + drivers/net/ethernet/amd/pds_core/adminq.c | 291 ++++++++ drivers/net/ethernet/amd/pds_core/auxbus.c | 241 +++++++ drivers/net/ethernet/amd/pds_core/core.c | 616 +++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 325 +++++++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 202 ++++++ drivers/net/ethernet/amd/pds_core/dev.c | 352 ++++++++++ drivers/net/ethernet/amd/pds_core/devlink.c | 180 +++++ drivers/net/ethernet/amd/pds_core/fw.c | 194 ++++++ drivers/net/ethernet/amd/pds_core/main.c | 489 +++++++++++++ include/linux/pds/pds_adminq.h | 645 ++++++++++++++++++ include/linux/pds/pds_auxbus.h | 44 ++ include/linux/pds/pds_common.h | 97 +++ include/linux/pds/pds_core_if.h | 540 +++++++++++++++ include/linux/pds/pds_intr.h | 163 +++++ 20 files changed, 4560 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c create mode 100644 drivers/net/ethernet/amd/pds_core/core.c create mode 100644 drivers/net/ethernet/amd/pds_core/core.h create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c create mode 100644 drivers/net/ethernet/amd/pds_core/main.c create mode 100644 include/linux/pds/pds_adminq.h create mode 100644 include/linux/pds/pds_auxbus.h create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core_if.h create mode 100644 include/linux/pds/pds_intr.h
Signature · 1 line
2.17.1
PATCH v9 1/14 pds_core: initial framework for pds_core PF driver
This is the initial PCI driver framework for the new pds_core device driver and its family of devices. This does the very basics of registering for the new PF PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.
--- .../device_drivers/ethernet/amd/pds_core.rst | 40 ++ .../device_drivers/ethernet/index.rst | 1 + drivers/net/ethernet/amd/pds_core/Makefile | 8 + drivers/net/ethernet/amd/pds_core/core.h | 63 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++ drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++ include/linux/pds/pds_common.h | 14 + include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++ 8 files changed, 985 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile create mode 100644 drivers/net/ethernet/amd/pds_core/core.h create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/amd/pds_core/main.c create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core_if.h
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst new file +40 -0
@@ -0,0 +1,40 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +======================================================== +Linux Driver for the AMD/Pensando(R) DSC adapter family +======================================================== + +Copyright(c) 2023 Advanced Micro Devices, Inc + +Identifying the Adapter +======================= + +To find if one or more AMD/Pensando PCI Core devices are installed on the +host, check for the PCI devices:: + + # lspci -d 1dd8:100c + b5:00.0 Processing accelerators: Pensando Systems Device 100c + b6:00.0 Processing accelerators: Pensando Systems Device 100c + +If such devices are listed as above, then the pds_core.ko driver should find +and configure them for use. There should be log entries in the kernel +messages such as these:: + + $ dmesg | grep pds_core + pds_core 0000:b5:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b5:00.0: FW: 1.60.0-73 + pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b6:00.0: FW: 1.60.0-73 + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by AMD/Pensando personnel:: + + [email protected] + +For more specific support needs, please use the AMD/Pensando driver support +email:: + + [email protected]
Documentation/networking/device_drivers/ethernet/index.rst +1 -0
@@ -14,6 +14,7 @@ Contents: 3com/vortex amazon/ena altera/altera_tse + amd/pds_core aquantia/atlantic chelsio/cxgb cirrus/cs89x0
drivers/net/ethernet/amd/pds_core/Makefile new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Advanced Micro Devices, Inc. + +obj-$(CONFIG_PDS_CORE) := pds_core.o + +pds_core-y := main.o + +pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.h new file +63 -0
63 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/debugfs.c new file +34 -0
@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifdef CONFIG_DEBUG_FS + +#include <linux/pci.h> + +#include "core.h" + +static struct dentry *pdsc_dir; + +void pdsc_debugfs_create (void ) +{ + pdsc_dir =debugfs_create_dir (PDS_CORE_DRV_NAME ,NULL ); +} + +void pdsc_debugfs_destroy (void ) +{ +debugfs_remove_recursive (pdsc_dir); +} + +void pdsc_debugfs_add_dev (struct pdsc *pdsc) +{ + pdsc->dentry =debugfs_create_dir (pci_name (pdsc->pdev), pdsc_dir); + +debugfs_create_ulong ("state" ,0400 , pdsc->dentry, &pdsc->state); +} + +void pdsc_debugfs_del_dev (struct pdsc *pdsc) +{ +debugfs_remove_recursive (pdsc->dentry); + pdsc->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */
drivers/net/ethernet/amd/pds_core/main.c new file +285 -0
285 lines not on this page · raw message
include/linux/pds/pds_common.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core" + +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) +#define PDS_PAGE_SIZE 4096 + +#endif /* _PDS_COMMON_H_ */
include/linux/pds/pds_core_if.h new file +540 -0
540 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver
Quoted 23 lines · Shannon Nelson
This is the initial PCI driver framework for the new
pds_coredevice driver and its family of devices. This does the very basics of registering for the new PF PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 40 ++ .../device_drivers/ethernet/index.rst | 1 + drivers/net/ethernet/amd/pds_core/Makefile | 8 + drivers/net/ethernet/amd/pds_core/core.h | 63 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++ drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++ include/linux/pds/pds_common.h | 14 + include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++ 8 files changed, 985 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile create mode 100644 drivers/net/ethernet/amd/pds_core/core.h create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/amd/pds_core/main.c create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core_if.h
<...>
drivers/net/ethernet/amd/pds_core/core.h line 61
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/core.h new file +11
-0+#ifdef CONFIG_DEBUG_FS +void pdsc_debugfs_create (void ); +void pdsc_debugfs_destroy (void ); +void pdsc_debugfs_add_dev (struct pdsc *pdsc); +void pdsc_debugfs_del_dev (struct pdsc *pdsc); +#else +static inline void pdsc_debugfs_create (void ) { } +static inline void pdsc_debugfs_destroy (void ) { } +static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { } +#endif
I don't think that you need CONFIG_DEBUG_FS guard as debugfs code is built to handle this case, so you can call to internal debugfs_*() calls without completed initialization of debugfs.
Quoted 74 lines
(quoted diff) +2
-0+ +#endif /* _PDSC_H_ */drivers/net/ethernet/amd/pds_core/debugfs.c new file +34
-0@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifdef CONFIG_DEBUG_FS + +#include <linux/pci.h> + +#include "core.h" + +static struct dentry *pdsc_dir; + +void pdsc_debugfs_create (void ) +{ + pdsc_dir =debugfs_create_dir (PDS_CORE_DRV_NAME ,NULL ); +} + +void pdsc_debugfs_destroy (void ) +{ +debugfs_remove_recursive (pdsc_dir); +} + +void pdsc_debugfs_add_dev (struct pdsc *pdsc) +{ + pdsc->dentry =debugfs_create_dir (pci_name (pdsc->pdev), pdsc_dir); + +debugfs_create_ulong ("state" ,0400 , pdsc->dentry, &pdsc->state); +} + +void pdsc_debugfs_del_dev (struct pdsc *pdsc) +{ +debugfs_remove_recursive (pdsc->dentry); + pdsc->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/main.c new file +26
-0@@ -0,0 +1,285 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt + +#include <linux/pci.h> + +#include <linux/pds/pds_common.h> + +#include "core.h" + +MODULE_DESCRIPTION (PDSC_DRV_DESCRIPTION ); +MODULE_AUTHOR ("Advanced Micro Devices, Inc" ); +MODULE_LICENSE ("GPL" ); + +/* Supported devices */ +static const struct pci_device_id pdsc_id_table[] = { + {PCI_VDEVICE (PENSANDO ,PCI_DEVICE_ID_PENSANDO_CORE_PF ) }, + {0 , }/* end of table */ +}; +MODULE_DEVICE_TABLE (pci, pdsc_id_table); + +static void pdsc_unmap_bars (struct pdsc *pdsc) +{ +struct pdsc_dev_bar *bars = pdsc->bars; +unsigned int i;
drivers/net/ethernet/amd/pds_core/main.c line 36
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/main.c new file +10
-0+ +for (i =0 ; i <PDS_CORE_BARS_MAX ; i++) { +if (bars[i].vaddr) { +pci_iounmap (pdsc->pdev, bars[i].vaddr); + bars[i].vaddr =NULL ; + } + + bars[i].len =0 ; + bars[i].bus_addr =0 ; + bars[i].res_index =0 ;
Why are you clearing it? You are going to release bars[] anyway. It will be great to remove this zeroing pattern from whole driver as it hides use-after-free bugs.
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/main.c new file +6
-0+ } +} + +static int pdsc_map_bars (struct pdsc *pdsc) +{ +struct pdsc_dev_bar *bar = pdsc->bars;
drivers/net/ethernet/amd/pds_core/main.c line 52
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/main.c new file +10
-0+struct pci_dev *pdev = pdsc->pdev; +struct device *dev = pdsc->dev; +struct pdsc_dev_bar *bars; +unsigned int i, j; +int num_bars =0 ; +int err; +u32 sig; + + bars = pdsc->bars; + num_bars =0 ;
You set it to zero 4 lines above.
Quoted 1 line
drivers/net/ethernet/amd/pds_core/main.c new file +1
-0+
<...>
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/main.c new file +2
-0+module_init (pdsc_init_module); +module_exit (pdsc_cleanup_module);include/linux/pds/pds_common.h new file +0
-0
include/linux/pds/pds_common.h line 7
Quoted 7 lines
include/linux/pds/pds_common.h new file +7
-0@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core"
It is KBUILD_MODNAME.
include/linux/pds/pds_common.h line 12
Quoted 5 lines
include/linux/pds/pds_common.h new file +5
-0+ +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) +#define PDS_PAGE_SIZE 4096
Thanks
Re: [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver
Quoted 1 line · Leon Romanovsky
Thanks for the time you put into these review notes, I appreciate it.
Quoted 27 lines · Shannon Nelson
Quoted 23 lines
This is the initial PCI driver framework for the new
pds_coredevice driver and its family of devices. This does the very basics of registering for the new PF PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 40 ++ .../device_drivers/ethernet/index.rst | 1 + drivers/net/ethernet/amd/pds_core/Makefile | 8 + drivers/net/ethernet/amd/pds_core/core.h | 63 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++ drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++ include/linux/pds/pds_common.h | 14 + include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++ 8 files changed, 985 insertions(+)create mode 100644
Documentation/networking/device_drivers/ethernet/amd/pds_core.rstcreate mode 100644drivers/net/ethernet/amd/pds_core/Makefile
create mode 100644drivers/net/ethernet/amd/pds_core/core.h
create mode 100644drivers/net/ethernet/amd/pds_core/debugfs.c
create mode 100644drivers/net/ethernet/amd/pds_core/main.c
create mode 100644include/linux/pds/pds_common.h
create mode 100644include/linux/pds/pds_core_if.h<...>
Quoted 15 lines
drivers/net/ethernet/amd/pds_core/core.h line 61
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/core.h new file +11
-0+#ifdef CONFIG_DEBUG_FS +void pdsc_debugfs_create (void ); +void pdsc_debugfs_destroy (void ); +void pdsc_debugfs_add_dev (struct pdsc *pdsc); +void pdsc_debugfs_del_dev (struct pdsc *pdsc); +#else +static inline void pdsc_debugfs_create (void ) { } +static inline void pdsc_debugfs_destroy (void ) { } +static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { } +#endif I don't think that you need
CONFIG_DEBUG_FSguard as debugfs code is built to handle this case, so you can call to internal debugfs_*() calls without completed initialization of debugfs.
Old habits... sure, we can pull that out.
Quoted 81 lines
Quoted 81 lines
(quoted diff) +2
-0+ +#endif /* _PDSC_H_ */drivers/net/ethernet/amd/pds_core/debugfs.c new file +34
-0@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifdef CONFIG_DEBUG_FS + +#include <linux/pci.h> + +#include "core.h" + +static struct dentry *pdsc_dir; + +void pdsc_debugfs_create (void ) +{ + pdsc_dir =debugfs_create_dir (PDS_CORE_DRV_NAME ,NULL ); +} + +void pdsc_debugfs_destroy (void ) +{ +debugfs_remove_recursive (pdsc_dir); +} + +void pdsc_debugfs_add_dev (struct pdsc *pdsc) +{ + pdsc->dentry =debugfs_create_dir (pci_name (pdsc->pdev), pdsc_dir); + +debugfs_create_ulong ("state" ,0400 , pdsc->dentry, &pdsc->state); +} + +void pdsc_debugfs_del_dev (struct pdsc *pdsc) +{ +debugfs_remove_recursive (pdsc->dentry); + pdsc->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/main.c new file +32
-0@@ -0,0 +1,285 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt + +#include <linux/pci.h> + +#include <linux/pds/pds_common.h> + +#include "core.h" + +MODULE_DESCRIPTION (PDSC_DRV_DESCRIPTION ); +MODULE_AUTHOR ("Advanced Micro Devices, Inc" ); +MODULE_LICENSE ("GPL" ); + +/* Supported devices */ +static const struct pci_device_id pdsc_id_table[] = { + {PCI_VDEVICE (PENSANDO ,PCI_DEVICE_ID_PENSANDO_CORE_PF ) }, + {0 , }/* end of table */ +}; +MODULE_DEVICE_TABLE (pci, pdsc_id_table); + +static void pdsc_unmap_bars (struct pdsc *pdsc) +{ +struct pdsc_dev_bar *bars = pdsc->bars; +unsigned int i; + +for (i =0 ; i <PDS_CORE_BARS_MAX ; i++) { +if (bars[i].vaddr) { +pci_iounmap (pdsc->pdev, bars[i].vaddr); + bars[i].vaddr =NULL ; + }
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/main.c line 36
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/main.c new file +4
-0+ + bars[i].len =0 ; + bars[i].bus_addr =0 ; + bars[i].res_index =0 ;Why are you clearing it? You are going to release bars[] anyway. It will be great to remove this zeroing pattern from whole driver as it hides use-after-free bugs.
These are from old habits of cleaning up when done with something. Some of these kinds of zeroing are useful for checks later to see if something is still valid, but you are right, not all of it is really necessary.
Quoted 11 lines
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/main.c new file +10
-0+ } +} + +static int pdsc_map_bars (struct pdsc *pdsc) +{ +struct pdsc_dev_bar *bar = pdsc->bars; +struct pci_dev *pdev = pdsc->pdev; +struct device *dev = pdsc->dev; +struct pdsc_dev_bar *bars; +unsigned int i, j;
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/main.c line 52
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/main.c new file +6
-0+int num_bars =0 ; +int err; +u32 sig; + + bars = pdsc->bars; + num_bars =0 ;You set it to zero 4 lines above.
Will fix
Quoted 14 lines
Quoted 2 lines
drivers/net/ethernet/amd/pds_core/main.c new file +1
-0+<...>
Quoted 9 lines
drivers/net/ethernet/amd/pds_core/main.c new file +2
-0+module_init (pdsc_init_module); +module_exit (pdsc_cleanup_module);include/linux/pds/pds_common.h new file +1
-0@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
Quoted 8 lines
include/linux/pds/pds_common.h line 7
Quoted 6 lines
include/linux/pds/pds_common.h new file +6
-0+/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core"It is
KBUILD_MODNAME.
yep, will fix
Quoted 8 lines
include/linux/pds/pds_common.h line 12
Quoted 6 lines
include/linux/pds/pds_common.h new file +5
-0+ +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) +#define PDS_PAGE_SIZE 4096Thanks
PATCH v9 2/14 pds_core: add devcmd device interfaces
In reply to Shannon NelsonThe devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
--- drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/core.h | 52 +++ drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 33 +- include/linux/pds/pds_common.h | 61 ++++ include/linux/pds/pds_intr.h | 163 +++++++++ 8 files changed, 763 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/core.c create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c create mode 100644 include/linux/pds/pds_intr.h
drivers/net/ethernet/amd/pds_core/Makefile +3 -1
@@ -3,6 +3,8 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o -pds_core-y := main.o +pds_core-y := main.o \ + dev.o \ + core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include "core.h" + +int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ; + +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +} + +void pdsc_teardown (struct pdsc *pdsc,bool removing) +{ +pdsc_devcmd_reset (pdsc); + +if (removing) { +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + } + +if (pdsc->kern_dbpage) { +iounmap (pdsc->kern_dbpage); + pdsc->kern_dbpage =NULL ; + } + +set_bit (PDSC_S_FW_DEAD , &pdsc->state); +}
drivers/net/ethernet/amd/pds_core/core.h +52 -0
84 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/debugfs.c +68 -0
72 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/dev.c new file +349 -0
349 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +31 -2
@@ -148,6 +148,18 @@ static int pdsc_init_pf(struct pdsc *pdsc)if (err) goto err_out_release_regions; +mutex_init (&pdsc->devcmd_lock); +mutex_init (&pdsc->config_lock); + +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_FW_DEAD , &pdsc->state); + + err =pdsc_setup (pdsc,PDSC_SETUP_INIT ); +if (err) + goto err_out_unmap_bars; + +mutex_unlock (&pdsc->config_lock); + dl =priv_to_devlink (pdsc);devl_lock (dl);devl_register (dl);
@@ -155,6 +167,12 @@ static int pdsc_init_pf(struct pdsc *pdsc)return 0 ; +err_out_unmap_bars: +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); +pci_free_irq_vectors (pdsc->pdev); +pdsc_unmap_bars (pdsc); err_out_release_regions:pci_release_regions (pdsc->pdev);
@@ -251,8 +269,19 @@ static void pdsc_remove(struct pci_dev *pdev)devl_unregister (dl);devl_unlock (dl); -pdsc_unmap_bars (pdsc); -pci_release_regions (pdev); +if (!pdev->is_virtfn) { +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); + +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); + +pci_free_irq_vectors (pdev); +pdsc_unmap_bars (pdsc); +pci_release_regions (pdev); + }pci_clear_master (pdev);pci_disable_device (pdev);
include/linux/pds/pds_common.h +61 -0
65 lines not on this page · raw message
include/linux/pds/pds_intr.h new file +163 -0
163 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
Quoted 19 lines · Shannon Nelson
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/core.h | 52 +++ drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 33 +- include/linux/pds/pds_common.h | 61 ++++ include/linux/pds/pds_intr.h | 163 +++++++++ 8 files changed, 763 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/core.c create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c create mode 100644 include/linux/pds/pds_intr.h
<...>
drivers/net/ethernet/amd/pds_core/core.c line 8
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/core.c new file +3
-0+int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ;
You don't need to set value as you overwrite it immediately.
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/core.c new file +11
-0+ +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +}
<...>
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +13
-0+static int irqs_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pdsc_intr_info *intr_info; +int i; + +seq_printf (seq,"index vector name (nintrs %d)\n" , pdsc->nintrs); + +if (!pdsc->intr_info) +return 0 ; + +for (i =0 ; i < pdsc->nintrs; i++) { + intr_info = &pdsc->intr_info[i];
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 95
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +10
-0+if (!intr_info->vector) +continue ; + +seq_printf (seq,"% 3d % 3d %s\n" , + i, intr_info->vector, intr_info->name); + } + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (irqs);
I'm curious why existing IRQ core support is not enough?
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +6
-0+ +void pdsc_debugfs_add_irqs (struct pdsc *pdsc) +{ +debugfs_create_file ("irqs" ,0400 , pdsc->dentry, pdsc, &irqs_fops); +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/dev.c new file +0
-0
drivers/net/ethernet/amd/pds_core/dev.c line 10
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +10
-0@@ -0,0 +1,349 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/errno.h> +#include <linux/pci.h> +#include <linux/utsname.h> + +#include "core.h" + +int pdsc_err_to_errno (enum pds_core_status_code code)
All users of this function, call to pdsc_devcmd_status() first. Probably they need to be combined.
Quoted 44 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +44
-0+{ +switch (code) { +case PDS_RC_SUCCESS : +return 0 ; +case PDS_RC_EVERSION : +case PDS_RC_EQTYPE : +case PDS_RC_EQID : +case PDS_RC_EINVAL : +case PDS_RC_ENOSUPP : +return -EINVAL ; +case PDS_RC_EPERM : +return -EPERM ; +case PDS_RC_ENOENT : +return -ENOENT ; +case PDS_RC_EAGAIN : +return -EAGAIN ; +case PDS_RC_ENOMEM : +return -ENOMEM ; +case PDS_RC_EFAULT : +return -EFAULT ; +case PDS_RC_EBUSY : +return -EBUSY ; +case PDS_RC_EEXIST : +return -EEXIST ; +case PDS_RC_EVFID : +return -ENODEV ; +case PDS_RC_ECLIENT : +return -ECHILD ; +case PDS_RC_ENOSPC : +return -ENOSPC ; +case PDS_RC_ERANGE : +return -ERANGE ; +case PDS_RC_BAD_ADDR : +return -EFAULT ; +case PDS_RC_EOPCODE : +case PDS_RC_EINTR : +case PDS_RC_DEV_CMD : +case PDS_RC_ERROR : +case PDS_RC_ERDMA : +case PDS_RC_EIO : +default : +return -EIO ; + } +}
<...>
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +4
-0+static u8 pdsc_devcmd_status (struct pdsc *pdsc) +{ +return ioread8 (&pdsc->cmd_regs->comp.status); +}
<...>
drivers/net/ethernet/amd/pds_core/dev.c line 197
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +3
-0+int pdsc_devcmd_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 };
There is no need to put 0 while using {} initialization.
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+union pds_core_dev_cmd cmd = { + .opcode =PDS_CORE_CMD_INIT , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +}
<...>
drivers/net/ethernet/amd/pds_core/dev.c line 328
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +4
-0+/* Get intr_info struct array for tracking */ + pdsc->intr_info =kcalloc (nintrs,sizeof (*pdsc->intr_info),GFP_KERNEL ); +if (!pdsc->intr_info) { + err = -ENOSPC ;
The general convention is to return -ENOMEM for memorly allocation failures.
Quoted 2 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +2
-0+ goto err_out; + }
<...>
Quoted 20 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +20
-0+/* +* enum pds_core_status_code - Device command return codes +*/ +enum pds_core_status_code { +PDS_RC_SUCCESS =0 ,/* Success */ +PDS_RC_EVERSION =1 ,/* Incorrect version for request */ +PDS_RC_EOPCODE =2 ,/* Invalid cmd opcode */ +PDS_RC_EIO =3 ,/* I/O error */ +PDS_RC_EPERM =4 ,/* Permission denied */ +PDS_RC_EQID =5 ,/* Bad qid */ +PDS_RC_EQTYPE =6 ,/* Bad qtype */ +PDS_RC_ENOENT =7 ,/* No such element */ +PDS_RC_EINTR =8 ,/* operation interrupted */ +PDS_RC_EAGAIN =9 ,/* Try again */ +PDS_RC_ENOMEM =10 ,/* Out of memory */ +PDS_RC_EFAULT =11 ,/* Bad address */ +PDS_RC_EBUSY =12 ,/* Device or resource busy */ +PDS_RC_EEXIST =13 ,/* object already exists */ +PDS_RC_EINVAL =14 ,/* Invalid argument */ +PDS_RC_ENOSPC =15 ,/* No space left or alloc failure */
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +10
-0+PDS_RC_ERANGE =16 ,/* Parameter out of range */ +PDS_RC_BAD_ADDR =17 ,/* Descriptor contains a bad ptr */ +PDS_RC_DEV_CMD =18 ,/* Device cmd attempted on AdminQ */ +PDS_RC_ENOSUPP =19 ,/* Operation not supported */ +PDS_RC_ERROR =29 ,/* Generic error */ +PDS_RC_ERDMA =30 ,/* Generic RDMA error */ +PDS_RC_EVFID =31 ,/* VF ID does not exist */ +PDS_RC_BAD_FW =32 ,/* FW file is invalid or corrupted */ + PDS_RC_ECLIENT = 33, /* No such client id */ +};
We asked from Intel to remove custom error codes and we would like to ask it here too. Please use standard in-kernel errors.
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +3
-0+ +enum pds_core_driver_type { +PDS_DRIVER_LINUX =1 ,
This is only relevant here, everything else is not applicable.
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +7
-0+PDS_DRIVER_WIN =2 , +PDS_DRIVER_DPDK =3 , +PDS_DRIVER_FREEBSD =4 , +PDS_DRIVER_IPXE =5 , + PDS_DRIVER_ESXI = 6, +}; +
Thanks
Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
Quoted 22 lines · Leon Romanovsky
Quoted 16 lines · Shannon Nelson
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/core.h | 52 +++ drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 33 +- include/linux/pds/pds_common.h | 61 ++++ include/linux/pds/pds_intr.h | 163 +++++++++ 8 files changed, 763 insertions(+), 3 deletions(-)Quoted 3 lines
create mode 100644
drivers/net/ethernet/amd/pds_core/core.ccreate mode 100644drivers/net/ethernet/amd/pds_core/dev.ccreate mode 100644include/linux/pds/pds_intr.h
Quoted 7 lines
<...>
drivers/net/ethernet/amd/pds_core/core.c line 8
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/core.c new file +3
-0+int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ;You don't need to set value as you overwrite it immediately.
will fix
Quoted 32 lines
Quoted 12 lines
drivers/net/ethernet/amd/pds_core/core.c new file +11
-0+ +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +}<...>
Quoted 17 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +17
-0+static int irqs_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pdsc_intr_info *intr_info; +int i; + +seq_printf (seq,"index vector name (nintrs %d)\n" , pdsc->nintrs); + +if (!pdsc->intr_info) +return 0 ; + +for (i =0 ; i < pdsc->nintrs; i++) { + intr_info = &pdsc->intr_info[i]; +if (!intr_info->vector) +continue ; + + seq_printf(seq, "% 3d % 3d %s\n",
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 95
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +6
-0+ i, intr_info->vector, intr_info->name); + } + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (irqs);I'm curious why existing IRQ core support is not enough?
Yes, that one is not necessary, will remove.
Quoted 19 lines
Quoted 19 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +6
-0+ +void pdsc_debugfs_add_irqs (struct pdsc *pdsc) +{ +debugfs_create_file ("irqs" ,0400 , pdsc->dentry, pdsc, &irqs_fops); +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0@@ -0,0 +1,349 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/errno.h> +#include <linux/pci.h>
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c line 10
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0+#include <linux/utsname.h> + +#include "core.h" + +int pdsc_err_to_errno (enum pds_core_status_code code)All users of this function, call to
pdsc_devcmd_status()first. Probably they need to be combined.
This is also called from pdsc_adminq_post() which doesn't use pdsc_devcmd_status().
Quoted 55 lines
Quoted 45 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +44
-0+{ +switch (code) { +case PDS_RC_SUCCESS : +return 0 ; +case PDS_RC_EVERSION : +case PDS_RC_EQTYPE : +case PDS_RC_EQID : +case PDS_RC_EINVAL : +case PDS_RC_ENOSUPP : +return -EINVAL ; +case PDS_RC_EPERM : +return -EPERM ; +case PDS_RC_ENOENT : +return -ENOENT ; +case PDS_RC_EAGAIN : +return -EAGAIN ; +case PDS_RC_ENOMEM : +return -ENOMEM ; +case PDS_RC_EFAULT : +return -EFAULT ; +case PDS_RC_EBUSY : +return -EBUSY ; +case PDS_RC_EEXIST : +return -EEXIST ; +case PDS_RC_EVFID : +return -ENODEV ; +case PDS_RC_ECLIENT : +return -ECHILD ; +case PDS_RC_ENOSPC : +return -ENOSPC ; +case PDS_RC_ERANGE : +return -ERANGE ; +case PDS_RC_BAD_ADDR : +return -EFAULT ; +case PDS_RC_EOPCODE : +case PDS_RC_EINTR : +case PDS_RC_DEV_CMD : +case PDS_RC_ERROR : +case PDS_RC_ERDMA : +case PDS_RC_EIO : +default : +return -EIO ; + } +}<...>
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +4
-0+static u8 pdsc_devcmd_status (struct pdsc *pdsc) +{ +return ioread8 (&pdsc->cmd_regs->comp.status); +}<...>
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/dev.c line 197
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +3
-0+int pdsc_devcmd_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 };There is no need to put 0 while using {} initialization.
will fix.
Quoted 10 lines
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+union pds_core_dev_cmd cmd = { + .opcode =PDS_CORE_CMD_INIT , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +}<...>
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c line 328
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +4
-0+/* Get intr_info struct array for tracking */ + pdsc->intr_info =kcalloc (nintrs,sizeof (*pdsc->intr_info),GFP_KERNEL ); +if (!pdsc->intr_info) { + err = -ENOSPC ;The general convention is to return
-ENOMEMfor memorly allocation failures.
will fix
Quoted 31 lines
Quoted 3 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +2
-0+ goto err_out; + }<...>
Quoted 25 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +25
-0+/* +* enum pds_core_status_code - Device command return codes +*/ +enum pds_core_status_code { +PDS_RC_SUCCESS =0 ,/* Success */ +PDS_RC_EVERSION =1 ,/* Incorrect version for request */ +PDS_RC_EOPCODE =2 ,/* Invalid cmd opcode */ +PDS_RC_EIO =3 ,/* I/O error */ +PDS_RC_EPERM =4 ,/* Permission denied */ +PDS_RC_EQID =5 ,/* Bad qid */ +PDS_RC_EQTYPE =6 ,/* Bad qtype */ +PDS_RC_ENOENT =7 ,/* No such element */ +PDS_RC_EINTR =8 ,/* operation interrupted */ +PDS_RC_EAGAIN =9 ,/* Try again */ +PDS_RC_ENOMEM =10 ,/* Out of memory */ +PDS_RC_EFAULT =11 ,/* Bad address */ +PDS_RC_EBUSY =12 ,/* Device or resource busy */ +PDS_RC_EEXIST =13 ,/* object already exists */ +PDS_RC_EINVAL =14 ,/* Invalid argument */ +PDS_RC_ENOSPC =15 ,/* No space left or alloc failure */ +PDS_RC_ERANGE =16 ,/* Parameter out of range */ +PDS_RC_BAD_ADDR =17 ,/* Descriptor contains a bad ptr */ +PDS_RC_DEV_CMD =18 ,/* Device cmd attempted on AdminQ */ +PDS_RC_ENOSUPP =19 ,/* Operation not supported */ +PDS_RC_ERROR =29 ,/* Generic error */
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0+PDS_RC_ERDMA =30 ,/* Generic RDMA error */ +PDS_RC_EVFID =31 ,/* VF ID does not exist */ +PDS_RC_BAD_FW =32 ,/* FW file is invalid or corrupted */ + PDS_RC_ECLIENT = 33, /* No such client id */ +};We asked from Intel to remove custom error codes and we would like to ask it here too. Please use standard in-kernel errors.
These are part of the device interface defined by the device firmware and include some that aren't in the errno set. This is why we use pdsc_err_to_errno() in pdsc_devcmd_wait() and pdsc_adminq_post(), so that we can change these status codes that we get from the device into standard kernel error codes. We try to report both in error messages, but only return the kernel errno.
However, I see in one place in pdsc_devcmd_wait() we're using the status codes where we could use the errno, so I'll fix that up.
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +3
-0+ +enum pds_core_driver_type { +PDS_DRIVER_LINUX =1 ,This is only relevant here, everything else is not applicable.
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+PDS_DRIVER_WIN =2 , +PDS_DRIVER_DPDK =3 , +PDS_DRIVER_FREEBSD =4 , +PDS_DRIVER_IPXE =5 , + PDS_DRIVER_ESXI = 6, +};
Yes, they are rather pointless for the Linux kernel, but it is part of documenting the device interface.
Quoted 3 lines
Quoted 1 line
+
Thanks
Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
Quoted 24 lines · Shannon Nelson
Quoted 23 lines · Leon Romanovsky
Quoted 19 lines · Shannon Nelson
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/core.h | 52 +++ drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 33 +- include/linux/pds/pds_common.h | 61 ++++ include/linux/pds/pds_intr.h | 163 +++++++++ 8 files changed, 763 insertions(+), 3 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/core.ccreate mode 100644drivers/net/ethernet/amd/pds_core/dev.ccreate mode 100644include/linux/pds/pds_intr.h<...>
<...>
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
#endif /* CONFIG_DEBUG_FS */drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0@@ -0,0 +1,349 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/errno.h> +#include <linux/pci.h> Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c line 10
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0+#include <linux/utsname.h> + +#include "core.h" + +int pdsc_err_to_errno (enum pds_core_status_code code)All users of this function, call to
pdsc_devcmd_status()first. Probably they need to be combined.
Quoted 2 lines
This is also called from
pdsc_adminq_post()which doesn't usepdsc_devcmd_status().
I probably missed that.
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +44
-0+{ +switch (code) { +case PDS_RC_SUCCESS : +return 0 ; +case PDS_RC_EVERSION : +case PDS_RC_EQTYPE : +case PDS_RC_EQID : +case PDS_RC_EINVAL : +case PDS_RC_ENOSUPP : +return -EINVAL ; +case PDS_RC_EPERM : +return -EPERM ; +case PDS_RC_ENOENT : +return -ENOENT ; +case PDS_RC_EAGAIN : +return -EAGAIN ; +case PDS_RC_ENOMEM : +return -ENOMEM ; +case PDS_RC_EFAULT : +return -EFAULT ; +case PDS_RC_EBUSY : +return -EBUSY ; +case PDS_RC_EEXIST : +return -EEXIST ; +case PDS_RC_EVFID : +return -ENODEV ; +case PDS_RC_ECLIENT : +return -ECHILD ; +case PDS_RC_ENOSPC : +return -ENOSPC ; +case PDS_RC_ERANGE : +return -ERANGE ; +case PDS_RC_BAD_ADDR : +return -EFAULT ; +case PDS_RC_EOPCODE : +case PDS_RC_EINTR : +case PDS_RC_DEV_CMD : +case PDS_RC_ERROR : +case PDS_RC_ERDMA : +case PDS_RC_EIO : +default : +return -EIO ; + } +}
<...>
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +25
-0+/* +* enum pds_core_status_code - Device command return codes +*/ +enum pds_core_status_code { +PDS_RC_SUCCESS =0 ,/* Success */ +PDS_RC_EVERSION =1 ,/* Incorrect version for request */ +PDS_RC_EOPCODE =2 ,/* Invalid cmd opcode */ +PDS_RC_EIO =3 ,/* I/O error */ +PDS_RC_EPERM =4 ,/* Permission denied */ +PDS_RC_EQID =5 ,/* Bad qid */ +PDS_RC_EQTYPE =6 ,/* Bad qtype */ +PDS_RC_ENOENT =7 ,/* No such element */ +PDS_RC_EINTR =8 ,/* operation interrupted */ +PDS_RC_EAGAIN =9 ,/* Try again */ +PDS_RC_ENOMEM =10 ,/* Out of memory */ +PDS_RC_EFAULT =11 ,/* Bad address */ +PDS_RC_EBUSY =12 ,/* Device or resource busy */ +PDS_RC_EEXIST =13 ,/* object already exists */ +PDS_RC_EINVAL =14 ,/* Invalid argument */ +PDS_RC_ENOSPC =15 ,/* No space left or alloc failure */ +PDS_RC_ERANGE =16 ,/* Parameter out of range */ +PDS_RC_BAD_ADDR =17 ,/* Descriptor contains a bad ptr */ +PDS_RC_DEV_CMD =18 ,/* Device cmd attempted on AdminQ */ +PDS_RC_ENOSUPP =19 ,/* Operation not supported */ +PDS_RC_ERROR =29 ,/* Generic error */ Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +5
-0+PDS_RC_ERDMA =30 ,/* Generic RDMA error */ +PDS_RC_EVFID =31 ,/* VF ID does not exist */ +PDS_RC_BAD_FW =32 ,/* FW file is invalid or corrupted */ + PDS_RC_ECLIENT = 33, /* No such client id */ +};We asked from Intel to remove custom error codes and we would like to ask it here too. Please use standard in-kernel errors.
Quoted 6 lines
These are part of the device interface defined by the device firmware and include some that aren't in the errno set. This is why we use
pdsc_err_to_errno()inpdsc_devcmd_wait()andpdsc_adminq_post(), so that we can change these status codes that we get from the device into standard kernel error codes. We try to report both in error messages, but only return the kernel errno.
You don't really need to create separate enum for that and place it in include/linux/pds/pds_common.h like you did.
You FW returns u8 status, which you can feed to pdsc_err_to_errno(). In the latter function, you will declare this translation enum. Such core reorg will esnure that you still have meaningful FW statuses while keeping them in limited code namespace.
Quoted 19 lines
However, I see in one place in
pdsc_devcmd_wait()we're using the status codes where we could use the errno, so I'll fix that up.Quoted 13 lines
drivers/net/ethernet/amd/pds_core/dev.c
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +3
-0+ +enum pds_core_driver_type { +PDS_DRIVER_LINUX =1 ,This is only relevant here, everything else is not applicable.
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+PDS_DRIVER_WIN =2 , +PDS_DRIVER_DPDK =3 , +PDS_DRIVER_FREEBSD =4 , +PDS_DRIVER_IPXE =5 , + PDS_DRIVER_ESXI = 6, +};
Quoted 2 lines
Yes, they are rather pointless for the Linux kernel, but it is part of documenting the device interface.
It is not used in upstream kernel.
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
+
Thanks
Jakub Kicinski
maintainerRe: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
Quoted 9 lines · On Thu, 13 Apr 2023 11:33:25 +0300 Leon Romanovsky wrote:
Quoted 9 lines
Quoted 8 lines
This is only relevant here, everything else is not applicable.
Quoted 6 lines
include/linux/pds/pds_common.h +6
-0+PDS_DRIVER_WIN =2 , +PDS_DRIVER_DPDK =3 , +PDS_DRIVER_FREEBSD =4 , +PDS_DRIVER_IPXE =5 , + PDS_DRIVER_ESXI = 6, +};
Quoted 4 lines
Quoted 2 lines
Yes, they are rather pointless for the Linux kernel, but it is part of documenting the device interface.
It is not used in upstream kernel.
If Shannon prefers to keep the full enum, I think that's fine. Declaring the full interface is often the only way a user can try to figure out how the API works, as most vendors don't publish their APIs.
Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
In reply to Leon RomanovskyQuoted 11 lines · Leon Romanovsky
Quoted 9 lines · Shannon Nelson
Quoted 8 lines · Leon Romanovsky
Quoted 6 lines · Shannon Nelson
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>
[...]
Quoted 42 lines
Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
include/linux/pds/pds_common.h +25
-0+/* +* enum pds_core_status_code - Device command return codes +*/ +enum pds_core_status_code { +PDS_RC_SUCCESS =0 ,/* Success */ +PDS_RC_EVERSION =1 ,/* Incorrect version for request */ +PDS_RC_EOPCODE =2 ,/* Invalid cmd opcode */ +PDS_RC_EIO =3 ,/* I/O error */ +PDS_RC_EPERM =4 ,/* Permission denied */ +PDS_RC_EQID =5 ,/* Bad qid */ +PDS_RC_EQTYPE =6 ,/* Bad qtype */ +PDS_RC_ENOENT =7 ,/* No such element */ +PDS_RC_EINTR =8 ,/* operation interrupted */ +PDS_RC_EAGAIN =9 ,/* Try again */ +PDS_RC_ENOMEM =10 ,/* Out of memory */ +PDS_RC_EFAULT =11 ,/* Bad address */ +PDS_RC_EBUSY =12 ,/* Device or resource busy */ +PDS_RC_EEXIST =13 ,/* object already exists */ +PDS_RC_EINVAL =14 ,/* Invalid argument */ +PDS_RC_ENOSPC =15 ,/* No space left or alloc failure */ +PDS_RC_ERANGE =16 ,/* Parameter out of range */ +PDS_RC_BAD_ADDR =17 ,/* Descriptor contains a bad ptr */ +PDS_RC_DEV_CMD =18 ,/* Device cmd attempted on AdminQ */ +PDS_RC_ENOSUPP =19 ,/* Operation not supported */ +PDS_RC_ERROR =29 ,/* Generic error */ Quoted 8 lines
include/linux/pds/pds_common.h line 43
Quoted 5 lines
include/linux/pds/pds_common.h +5
-0+PDS_RC_ERDMA =30 ,/* Generic RDMA error */ +PDS_RC_EVFID =31 ,/* VF ID does not exist */ +PDS_RC_BAD_FW =32 ,/* FW file is invalid or corrupted */ + PDS_RC_ECLIENT = 33, /* No such client id */ +};We asked from Intel to remove custom error codes and we would like to ask it here too. Please use standard in-kernel errors.
Quoted 6 lines
These are part of the device interface defined by the device firmware and include some that aren't in the errno set. This is why we use
pdsc_err_to_errno()inpdsc_devcmd_wait()andpdsc_adminq_post(), so that we can change these status codes that we get from the device into standard kernel error codes. We try to report both in error messages, but only return the kernel errno.
Quoted 7 lines
You don't really need to create separate enum for that and place it in
include/linux/pds/pds_common.hlike you did.You FW returns u8 status, which you can feed to
pdsc_err_to_errno(). In the latter function, you will declare this translation enum. Such core reorg will esnure that you still have meaningful FW statuses while keeping them in limited code namespace.
Yes, I can see moving it down into the decode function itself can work and keep it limited in scope. I'll take care of that.
Thanks,
sln
PATCH v9 3/14 pds_core: health timer and workqueue
In reply to Shannon NelsonAdd in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
The firmware is polled every 5 seconds to be sure that it is still alive and that the FW generation didn't change.
The alive check looks to see that the PCI bus is still readable and the fw_status still has the RUNNING bit on. If not alive, the driver stops activity and tears things down. When the FW recovers and the alive check again succeeds, the driver sets back up for activity.
The generation check looks at the fw_generation to see if it has changed, which can happen if the FW crashed and recovered or was updated in between health checks. If changed, the driver counts that as though the alive test failed and forces the fw_down/fw_up cycle.
--- drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 8 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 3 ++ drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++ 4 files changed, 109 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +61 -0
@@ -34,3 +34,64 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } + +static void pdsc_fw_down (struct pdsc *pdsc) +{ +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: already happening\n" , __func__); +return ; + } + +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); +} + +static void pdsc_fw_up (struct pdsc *pdsc) +{ +int err; + +if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: fw not dead\n" , __func__); +return ; + } + + err =pdsc_setup (pdsc,PDSC_SETUP_RECOVERY ); +if (err) + goto err_out; + +return ; + +err_out: +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); +} + +void pdsc_health_thread (struct work_struct *work) +{ + struct pdsc *pdsc = container_of(work, struct pdsc, health_work); +unsigned long mask; +bool healthy; + +mutex_lock (&pdsc->config_lock); + +/* Don't do a check when in a transition state */ + mask =BIT_ULL (PDSC_S_INITING_DRIVER ) | +BIT_ULL (PDSC_S_STOPPING_DRIVER ); +if (pdsc->state & mask) + goto out_unlock; + + healthy =pdsc_is_fw_good (pdsc); +dev_dbg (pdsc->dev,"%s: health %d fw_status %#02x fw_heartbeat %d\n" , + __func__, healthy, pdsc->fw_status, pdsc->last_hb); + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +if (healthy) +pdsc_fw_up (pdsc); + }else { +if (!healthy) +pdsc_fw_down (pdsc); + } + + pdsc->fw_generation = pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ; + +out_unlock: +mutex_unlock (&pdsc->config_lock); +}
drivers/net/ethernet/amd/pds_core/core.h +8 -0
@@ -12,6 +12,8 @@#include <linux/pds/pds_intr.h> #define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver" + +#define PDSC_WATCHDOG_SECS 5#define PDSC_TEARDOWN_RECOVERY false#define PDSC_TEARDOWN_REMOVING true#define PDSC_SETUP_RECOVERY false
@@ -63,12 +65,17 @@ struct pdsc {u8 fw_generation;unsigned long last_fw_time;u32 last_hb; +struct timer_list wdtimer; +unsigned int wdtimer_period; +struct work_struct health_work;struct pdsc_devinfo dev_info;struct pds_core_dev_identity dev_ident;unsigned int nintrs;struct pdsc_intr_info *intr_info;/* array of nintrs elements */ +struct workqueue_struct *wq; +unsigned int devcmd_timeout;struct mutex devcmd_lock;/* lock for dev_cmd operations */ struct mutex config_lock;/* lock for configuration operations */
@@ -111,5 +118,6 @@ int pdsc_dev_init(struct pdsc *pdsc);int pdsc_setup (struct pdsc *pdsc,bool init);void pdsc_teardown (struct pdsc *pdsc,bool removing); +void pdsc_health_thread (struct work_struct *work);#endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/dev.c +3 -0
@@ -177,6 +177,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, err =pdsc_devcmd_wait (pdsc, max_seconds);memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); +if (err == -ENXIO || err == -ETIMEDOUT ) +queue_work (pdsc->wq, &pdsc->health_work); +return err; }
drivers/net/ethernet/amd/pds_core/main.c +37 -0
@@ -20,6 +20,17 @@ static const struct pci_device_id pdsc_id_table[] = { };MODULE_DEVICE_TABLE (pci, pdsc_id_table); +static void pdsc_wdtimer_cb (struct timer_list *t) +{ +struct pdsc *pdsc =from_timer (pdsc, t, wdtimer); + +dev_dbg (pdsc->dev,"%s: jiffies %ld\n" , __func__, jiffies); +mod_timer (&pdsc->wdtimer, +round_jiffies (jiffies + pdsc->wdtimer_period)); + +queue_work (pdsc->wq, &pdsc->health_work); +} +static void pdsc_unmap_bars (struct pdsc *pdsc) {struct pdsc_dev_bar *bars = pdsc->bars;
@@ -130,8 +141,11 @@ static int pdsc_init_vf(struct pdsc *vf)return -1 ; } +#define PDSC_WQ_NAME_LEN 24 +static int pdsc_init_pf (struct pdsc *pdsc) { +char wq_name[PDSC_WQ_NAME_LEN ];struct devlink *dl;int err;
@@ -148,6 +162,13 @@ static int pdsc_init_pf(struct pdsc *pdsc)if (err) goto err_out_release_regions; +/* General workqueue and timer, but don't start timer yet */ +snprintf (wq_name,sizeof (wq_name),"%s.%d" ,PDS_CORE_DRV_NAME , pdsc->uid); + pdsc->wq =create_singlethread_workqueue (wq_name); +INIT_WORK (&pdsc->health_work, pdsc_health_thread); +timer_setup (&pdsc->wdtimer, pdsc_wdtimer_cb,0 ); + pdsc->wdtimer_period =PDSC_WATCHDOG_SECS *HZ ; +mutex_init (&pdsc->devcmd_lock);mutex_init (&pdsc->config_lock);
@@ -165,10 +186,19 @@ static int pdsc_init_pf(struct pdsc *pdsc)devl_register (dl);devl_unlock (dl); +/* Lastly, start the health check timer */ +mod_timer (&pdsc->wdtimer,round_jiffies (jiffies + pdsc->wdtimer_period)); +return 0 ; err_out_unmap_bars:mutex_unlock (&pdsc->config_lock); +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq); + pdsc->wq =NULL ; + }mutex_destroy (&pdsc->config_lock);mutex_destroy (&pdsc->devcmd_lock);pci_free_irq_vectors (pdsc->pdev);
@@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)devl_unlock (dl);if (!pdev->is_virtfn) { +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq); + pdsc->wq =NULL ; + } +mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state);
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 03/14] pds_core: health timer and workqueue
Quoted 25 lines · Shannon Nelson
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
The firmware is polled every 5 seconds to be sure that it is still alive and that the FW generation didn't change.
The alive check looks to see that the PCI bus is still readable and the
fw_statusstill has the RUNNING bit on. If not alive, the driver stops activity and tears things down. When the FW recovers and the alive check again succeeds, the driver sets back up for activity.The generation check looks at the
fw_generationto see if it has changed, which can happen if the FW crashed and recovered or was updated in between health checks. If changed, the driver counts that as though the alive test failed and forces the fw_down/fw_up cycle.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 8 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 3 ++ drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++ 4 files changed, 109 insertions(+)
<...>
drivers/net/ethernet/amd/pds_core/main.c static int pdsc_init_pf(struct pdsc *pdsc) · line 199
Quoted 7 lines
err_out_unmap_bars:drivers/net/ethernet/amd/pds_core/main.c +4
-0mutex_unlock (&pdsc->config_lock); +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq);
There is no need to call to flush_workqueue() as destroy_workqueue() will do it.
drivers/net/ethernet/amd/pds_core/main.c static void pdsc_remove(struct pci_dev *pdev) · line 306
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/main.c +6
-0+ pdsc->wq =NULL ; + }mutex_destroy (&pdsc->config_lock);mutex_destroy (&pdsc->devcmd_lock);pci_free_irq_vectors (pdsc->pdev);@@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)devl_unlock (dl);if (!pdev->is_virtfn) { +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq);
Same
Review static void pdsc_remove(struct pci_dev *pdev) · line 307
Quoted 1 line
(quoted diff) +1
-0+ pdsc->wq = NULL;
Not really needed, pdsc is released.
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/main.c +2
-0+ } +mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state);Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 03/14] pds_core: health timer and workqueue
Quoted 38 lines · Leon Romanovsky
Quoted 25 lines · Shannon Nelson
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
The firmware is polled every 5 seconds to be sure that it is still alive and that the FW generation didn't change.
The alive check looks to see that the PCI bus is still readable and the
fw_statusstill has the RUNNING bit on. If not alive, the driver stops activity and tears things down. When the FW recovers and the alive check again succeeds, the driver sets back up for activity.The generation check looks at the
fw_generationto see if it has changed, which can happen if the FW crashed and recovered or was updated in between health checks. If changed, the driver counts that as though the alive test failed and forces the fw_down/fw_up cycle.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 8 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 3 ++ drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++ 4 files changed, 109 insertions(+)<...>
drivers/net/ethernet/amd/pds_core/main.c static int pdsc_init_pf(struct pdsc *pdsc) · line 199
Quoted 7 lines
err_out_unmap_bars: mutex_unlock(&pdsc->config_lock);drivers/net/ethernet/amd/pds_core/main.c +4
-0+del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq);
Quoted 2 lines
There is no need to call to
flush_workqueue()asdestroy_workqueue()will do it.
will fix
Quoted 17 lines
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/main.c +2
-0+ pdsc->wq =NULL ; + }mutex_destroy (&pdsc->config_lock);mutex_destroy (&pdsc->devcmd_lock);pci_free_irq_vectors (pdsc->pdev);drivers/net/ethernet/amd/pds_core/main.c static void pdsc_remove(struct pci_dev *pdev) · line 306
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/main.c +4
-0@@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)devl_unlock (dl);if (!pdev->is_virtfn) { +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq);Same
Quoted 3 lines
Review static void pdsc_remove(struct pci_dev *pdev) · line 307
Quoted 1 line
(quoted diff) +1
-0+ pdsc->wq = NULL;Not really needed, pdsc is released.
will fix
Quoted 9 lines
Quoted 9 lines
drivers/net/ethernet/amd/pds_core/main.c +2
-1+ } +mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); --2.17.1
PATCH v9 4/14 pds_core: add devlink health facilities
In reply to Shannon NelsonAdd devlink health reporting on top of our fw watchdog.
Example:
# devlink health show pci/0000:2b:00.0 reporter fw
pci/0000:2b:00.0:
reporter fw
state healthy error 0 recover 0
# devlink health diagnose pci/0000:2b:00.0 reporter fw
Status: healthy State: 1 Generation: 0 Recoveries: 0--- .../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++ drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.c | 6 +++ drivers/net/ethernet/amd/pds_core/core.h | 6 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++ 6 files changed, 84 insertions(+) create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +12 -0
@@ -26,6 +26,18 @@ messages such as these:: pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) pds_core 0000:b6:00.0: FW: 1.60.0-73 +Health Reporters +================ + +The driver supports a devlink health reporter for FW status:: + + # devlink health show pci/0000:2b:00.0 reporter fw + pci/0000:2b:00.0: + reporter fw + state healthy error 0 recover 0 + # devlink health diagnose pci/0000:2b:00.0 reporter fw + Status: healthy State: 1 Generation: 0 Recoveries: 0 + Support =======
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ + devlink.o \ dev.o \ core.o
drivers/net/ethernet/amd/pds_core/core.c +6 -0
@@ -42,6 +42,8 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +devlink_health_report (pdsc->fw_reporter,"FW down reported" , pdsc); +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); }
@@ -58,6 +60,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; + pdsc->fw_recoveries++; +devlink_health_reporter_state_update (pdsc->fw_reporter, +DEVLINK_HEALTH_REPORTER_STATE_HEALTHY ); +return ; err_out:
drivers/net/ethernet/amd/pds_core/core.h +6 -0
@@ -68,6 +68,8 @@ struct pdsc {struct timer_list wdtimer;unsigned int wdtimer_period;struct work_struct health_work; +struct devlink_health_reporter *fw_reporter; +u32 fw_recoveries;struct pdsc_devinfo dev_info;struct pds_core_dev_identity dev_ident;
@@ -88,6 +90,10 @@ struct pdsc {u64 __iomem *kern_dbpage; }; +int pdsc_fw_reporter_diagnose (struct devlink_health_reporter *reporter, +struct devlink_fmsg *fmsg, +struct netlink_ext_ack *extack); +#ifdef CONFIG_DEBUG_FS void pdsc_debugfs_create (void );void pdsc_debugfs_destroy (void );
drivers/net/ethernet/amd/pds_core/devlink.c new file +37 -0
@@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include "core.h" + +int pdsc_fw_reporter_diagnose (struct devlink_health_reporter *reporter, +struct devlink_fmsg *fmsg, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_health_reporter_priv (reporter); +int err =0 ; + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state)) + err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"dead" ); +else if (!pdsc_is_fw_good (pdsc)) + err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"unhealthy" ); +else + err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"healthy" ); +if (err) +return err; + + err =devlink_fmsg_u32_pair_put (fmsg,"State" , + pdsc->fw_status & + ~PDS_CORE_FW_STS_F_GENERATION ); +if (err) +return err; + err =devlink_fmsg_u32_pair_put (fmsg,"Generation" , + pdsc->fw_generation >>4 ); +if (err) +return err; + err =devlink_fmsg_u32_pair_put (fmsg,"Recoveries" , + pdsc->fw_recoveries); +if (err) +return err; + +return 0 ; +}
drivers/net/ethernet/amd/pds_core/main.c +22 -0
@@ -141,10 +141,16 @@ static int pdsc_init_vf(struct pdsc *vf)return -1 ; } +static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = { + .name ="fw" , + .diagnose = pdsc_fw_reporter_diagnose, +}; +#define PDSC_WQ_NAME_LEN 24static int pdsc_init_pf (struct pdsc *pdsc) { +struct devlink_health_reporter *hr;char wq_name[PDSC_WQ_NAME_LEN ];struct devlink *dl;int err;
@@ -183,6 +189,16 @@ static int pdsc_init_pf(struct pdsc *pdsc) dl =priv_to_devlink (pdsc);devl_lock (dl); + + hr =devl_health_reporter_create (dl, &pdsc_fw_reporter_ops,0 , pdsc); +if (IS_ERR (hr)) { +dev_warn (pdsc->dev,"Failed to create fw reporter: %pe\n" , hr); + err =PTR_ERR (hr); +devl_unlock (dl); + goto err_out_teardown; + } + pdsc->fw_reporter = hr; +devl_register (dl);devl_unlock (dl);
@@ -191,6 +207,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)return 0 ; +err_out_teardown: +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); err_out_unmap_bars:mutex_unlock (&pdsc->config_lock);del_timer_sync (&pdsc->wdtimer);
@@ -297,6 +315,10 @@ static void pdsc_remove(struct pci_dev *pdev) dl =priv_to_devlink (pdsc);devl_lock (dl);devl_unregister (dl); +if (pdsc->fw_reporter) { +devl_health_reporter_destroy (pdsc->fw_reporter); + pdsc->fw_reporter =NULL ; + }devl_unlock (dl);if (!pdev->is_virtfn) {
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 04/14] pds_core: add devlink health facilities
Quoted 20 lines · Shannon Nelson
Add devlink health reporting on top of our fw watchdog.
Example:
# devlink health show pci/0000:2b:00.0 reporter fw pci/0000:2b:00.0: reporter fw state healthy error 0 recover 0 # devlink health diagnose pci/0000:2b:00.0 reporter fw Status: healthy State: 1 Generation: 0 Recoveries: 0Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++ drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.c | 6 +++ drivers/net/ethernet/amd/pds_core/core.h | 6 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++ 6 files changed, 84 insertions(+) create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
<...>
drivers/net/ethernet/amd/pds_core/devlink.c line 13
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +8
-0+int pdsc_fw_reporter_diagnose (struct devlink_health_reporter *reporter, +struct devlink_fmsg *fmsg, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_health_reporter_priv (reporter); +int err =0 ; + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state))
How is this check protected from race with your health workqueue added in previous patch?
drivers/net/ethernet/amd/pds_core/devlink.c line 15
Quoted 2 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +2
-0+ err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"dead" ); + else if (!pdsc_is_fw_good(pdsc))
Same question.
drivers/net/ethernet/amd/pds_core/devlink.c line 20
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +5
-0+ err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"unhealthy" ); + else + err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"healthy" ); +if (err) +return err;
Thanks
Re: [PATCH v9 net-next 04/14] pds_core: add devlink health facilities
Quoted 37 lines · Leon Romanovsky
Quoted 20 lines · Shannon Nelson
Add devlink health reporting on top of our fw watchdog.
Example:
# devlink health show pci/0000:2b:00.0 reporter fw pci/0000:2b:00.0: reporter fw state healthy error 0 recover 0 # devlink health diagnose pci/0000:2b:00.0 reporter fw Status: healthy State: 1 Generation: 0 Recoveries: 0Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++ drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.c | 6 +++ drivers/net/ethernet/amd/pds_core/core.h | 6 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++ 6 files changed, 84 insertions(+)create mode 100644
drivers/net/ethernet/amd/pds_core/devlink.c<...>
drivers/net/ethernet/amd/pds_core/devlink.c line 13
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +8
-0+int pdsc_fw_reporter_diagnose (struct devlink_health_reporter *reporter, +struct devlink_fmsg *fmsg, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_health_reporter_priv (reporter); +int err =0 ; + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state))How is this check protected from race with your health workqueue added in previous patch?
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/devlink.c line 15
Quoted 2 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +2
-0+ err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"dead" ); + else if (!pdsc_is_fw_good(pdsc))Same question.
Yes, it would be good to wrap these in the config_lock.
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/devlink.c line 20
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/devlink.c new file +5
-0+ err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"unhealthy" ); + else + err =devlink_fmsg_string_pair_put (fmsg,"Status" ,"healthy" ); +if (err) +return err;Thanks
PATCH v9 5/14 pds_core: set up device and adminq
In reply to Shannon NelsonSet up the basic adminq and notifyq queue structures. These are used mostly by the client drivers for feature configuration. These are essentially the same adminq and notifyq as in the ionic driver.
Part of this includes querying for device identity and FW information, so we can make that available to devlink dev info.
$ devlink dev info pci/0000:b5:00.0
pci/0000:b5:00.0:
driver pds_core
serial_number FLM18420073
versions:
fixed:
asic.id 0x0
asic.rev 0x0
running:
fw 1.51.0-73
stored:
fw.goldfw 1.15.9-C-22
fw.mainfwa 1.60.0-73
fw.mainfwb 1.60.0-57--- .../device_drivers/ethernet/amd/pds_core.rst | 47 ++ drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++- drivers/net/ethernet/amd/pds_core/core.h | 154 +++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++ drivers/net/ethernet/amd/pds_core/main.c | 17 +- include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++ 7 files changed, 1438 insertions(+), 4 deletions(-) create mode 100644 include/linux/pds/pds_adminq.h
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +47 -0
@@ -26,6 +26,53 @@ messages such as these:: pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) pds_core 0000:b6:00.0: FW: 1.60.0-73 +Driver and firmware version information can be gathered with devlink:: + + $ devlink dev info pci/0000:b5:00.0 + pci/0000:b5:00.0: + driver pds_core + serial_number FLM18420073 + versions: + fixed: + asic.id 0x0 + asic.rev 0x0 + running: + fw 1.51.0-73 + stored: + fw.goldfw 1.15.9-C-22 + fw.mainfwa 1.60.0-73 + fw.mainfwb 1.60.0-57 + +Info versions +============= + +The ``pds_core`` driver reports the following versions + +.. list-table:: devlink info versions implemented + :widths: 5 5 90 + + * - Name + - Type + - Description + * - ``fw`` + - running + - Version of firmware running on the device + * - ``fw.goldfw`` + - stored + - Version of firmware stored in the goldfw slot + * - ``fw.mainfwa`` + - stored + - Version of firmware stored in the mainfwa slot + * - ``fw.mainfwb`` + - stored + - Version of firmware stored in the mainfwb slot + * - ``asic.id`` + - fixed + - The ASIC type for this device + * - ``asic.rev`` + - fixed + - The revision of the ASIC for this device + Health Reporters ================
drivers/net/ethernet/amd/pds_core/core.c +447 -3
492 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/core.h +154 -0
214 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/debugfs.c +76 -0
80 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/devlink.c +61 -0
67 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +16 -1
@@ -136,6 +136,13 @@ static int pdsc_map_bars(struct pdsc *pdsc)return err; } +void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num) +{ +return pci_iomap_range (pdsc->pdev, + pdsc->bars[PDS_CORE_PCI_BAR_DBELL ].res_index, + (u64 )page_num <<PAGE_SHIFT ,PAGE_SIZE ); +} +static int pdsc_init_vf (struct pdsc *vf) {return -1 ;
@@ -177,6 +184,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)mutex_init (&pdsc->devcmd_lock);mutex_init (&pdsc->config_lock); +spin_lock_init (&pdsc->adminq_lock);mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_FW_DEAD , &pdsc->state);
@@ -184,6 +192,9 @@ static int pdsc_init_pf(struct pdsc *pdsc) err =pdsc_setup (pdsc,PDSC_SETUP_INIT );if (err) goto err_out_unmap_bars; + err =pdsc_start (pdsc); +if (err) + goto err_out_teardown;mutex_unlock (&pdsc->config_lock);
@@ -195,7 +206,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)dev_warn (pdsc->dev,"Failed to create fw reporter: %pe\n" , hr); err =PTR_ERR (hr);devl_unlock (dl); - goto err_out_teardown; + goto err_out_stop; } pdsc->fw_reporter = hr;
@@ -207,6 +218,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)return 0 ; +err_out_stop: +pdsc_stop (pdsc); err_out_teardown:pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); err_out_unmap_bars:
@@ -228,6 +241,7 @@ static int pdsc_init_pf(struct pdsc *pdsc) }static const struct devlink_ops pdsc_dl_ops = { + .info_get = pdsc_dl_info_get, };static const struct devlink_ops pdsc_dl_vf_ops = {
@@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); +pdsc_stop (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );mutex_unlock (&pdsc->config_lock);mutex_destroy (&pdsc->config_lock);
include/linux/pds/pds_adminq.h new file +637 -0
637 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 05/14] pds_core: set up device and adminq
Quoted 34 lines · Shannon Nelson
Set up the basic adminq and notifyq queue structures. These are used mostly by the client drivers for feature configuration. These are essentially the same adminq and notifyq as in the ionic driver.
Part of this includes querying for device identity and FW information, so we can make that available to devlink dev info.
$ devlink dev info pci/0000:b5:00.0 pci/0000:b5:00.0: driver pds_core serial_number FLM18420073 versions: fixed: asic.id 0x0 asic.rev 0x0 running: fw 1.51.0-73 stored: fw.goldfw 1.15.9-C-22 fw.mainfwa 1.60.0-73 fw.mainfwb 1.60.0-57Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 47 ++ drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++- drivers/net/ethernet/amd/pds_core/core.h | 154 +++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++ drivers/net/ethernet/amd/pds_core/main.c | 17 +- include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++ 7 files changed, 1438 insertions(+), 4 deletions(-) create mode 100644 include/linux/pds/pds_adminq.h
<...>
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/core.c +8
-0+void pdsc_stop (struct pdsc *pdsc) +{ +if (pdsc->wq) +flush_workqueue (pdsc->wq); + +pdsc_mask_interrupts (pdsc); +} +
<...>
drivers/net/ethernet/amd/pds_core/main.c static void pdsc_remove(struct pci_dev *pdev) · line 349
Quoted 6 lines
static const struct
devlink_opspdsc_dl_vf_ops= {drivers/net/ethernet/amd/pds_core/main.c +1
-0@@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); +pdsc_stop (pdsc);
You are calling to flush this workqueue in a couple of line above.
Quoted 12 lines
drivers/net/ethernet/amd/pds_core/main.c +0
-0pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );mutex_unlock (&pdsc->config_lock);mutex_destroy (&pdsc->config_lock);include/linux/pds/pds_adminq.h new file +3
-0@@ -0,0 +1,637 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ +
<...>
include/linux/pds/pds_adminq.h line 633
Quoted 5 lines
include/linux/pds/pds_adminq.h new file +5
-0+/* The color bit is a 'done' bit for the completion descriptors +* where the meaning alternates between '1' and '0' for alternating +* passes through the completion descriptor ring. +*/ +static inline u8 pdsc_color_match (u8 color,u8 done_color)
static inline bool?
Quoted 7 lines
include/linux/pds/pds_adminq.h new file +4
-0+{ +return (!!(color &PDS_COMP_COLOR_MASK )) == done_color; +} +#endif /* _PDS_CORE_ADMINQ_H_ */ Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 05/14] pds_core: set up device and adminq
Quoted 50 lines · Leon Romanovsky
Quoted 34 lines · Shannon Nelson
Set up the basic adminq and notifyq queue structures. These are used mostly by the client drivers for feature configuration. These are essentially the same adminq and notifyq as in the ionic driver.
Part of this includes querying for device identity and FW information, so we can make that available to devlink dev info.
$ devlink dev info pci/0000:b5:00.0 pci/0000:b5:00.0: driver pds_core serial_number FLM18420073 versions: fixed: asic.id 0x0 asic.rev 0x0 running: fw 1.51.0-73 stored: fw.goldfw 1.15.9-C-22 fw.mainfwa 1.60.0-73 fw.mainfwb 1.60.0-57Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 47 ++ drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++- drivers/net/ethernet/amd/pds_core/core.h | 154 +++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++ drivers/net/ethernet/amd/pds_core/main.c | 17 +- include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++ 7 files changed, 1438 insertions(+), 4 deletions(-)create mode 100644
include/linux/pds/pds_adminq.h<...>
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/core.c +8
-0+void pdsc_stop (struct pdsc *pdsc) +{ +if (pdsc->wq) +flush_workqueue (pdsc->wq); + +pdsc_mask_interrupts (pdsc); +} +<...>
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/main.c static void pdsc_remove(struct pci_dev *pdev) · line 349
Quoted 6 lines
static const struct devlink_ops pdsc_dl_vf_ops = {drivers/net/ethernet/amd/pds_core/main.c +1
-0@@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); +pdsc_stop (pdsc);You are calling to flush this workqueue in a couple of line above.
I guess we really want it flushed...
I'll look at that to see if we really need the extra flushing.
Quoted 16 lines
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/main.c +0
-0pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );mutex_unlock (&pdsc->config_lock);mutex_destroy (&pdsc->config_lock);include/linux/pds/pds_adminq.h new file +3
-0@@ -0,0 +1,637 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ +<...>
Quoted 7 lines
include/linux/pds/pds_adminq.h line 633
Quoted 5 lines
include/linux/pds/pds_adminq.h new file +5
-0+/* The color bit is a 'done' bit for the completion descriptors +* where the meaning alternates between '1' and '0' for alternating +* passes through the completion descriptor ring. +*/ +static inline u8 pdsc_color_match (u8 color,u8 done_color)static inline bool?
Sure.
Quoted 8 lines
Quoted 8 lines
include/linux/pds/pds_adminq.h new file +4
-1+{ +return (!!(color &PDS_COMP_COLOR_MASK )) == done_color; +} +#endif /* _PDS_CORE_ADMINQ_H_ */ --2.17.1
PATCH v9 6/14 pds_core: Add adminq processing and commands
In reply to Shannon NelsonAdd the service routines for submitting and processing the adminq messages and for handling notifyq events.
--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/adminq.c | 289 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.c | 11 - include/linux/pds/pds_adminq.h | 8 + 4 files changed, 298 insertions(+), 11 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ dev.o \ + adminq.o \ core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/adminq.c new file +289 -0
289 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/core.c +0 -11
@@ -6,17 +6,6 @@#include "core.h" -void pdsc_work_thread (struct work_struct *work) -{ -/* stub */ -} - -irqreturn_t pdsc_adminq_isr (int irq,void *data) -{ -/* stub */ -return IRQ_HANDLED ; -} -void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;
include/linux/pds/pds_adminq.h +8 -0
@@ -4,6 +4,8 @@#ifndef _PDS_CORE_ADMINQ_H_#define _PDS_CORE_ADMINQ_H_ +#define PDSC_ADMINQ_MAX_POLL_INTERVAL 256 +enum pds_core_adminq_flags {PDS_AQ_FLAG_FASTPOLL =BIT (1 ),/* completion poll at 1ms */ };
@@ -634,4 +636,10 @@ static inline u8 pdsc_color_match(u8 color, u8 done_color) {return (!!(color &PDS_COMP_COLOR_MASK )) == done_color; } + +int pdsc_adminq_post (void *pdsc, +union pds_core_adminq_cmd *cmd, +union pds_core_adminq_comp *comp, +bool fast_poll); +#endif /* _PDS_CORE_ADMINQ_H_ */
Signature · 1 line
2.17.1
PATCH v9 7/14 pds_core: add FW update feature to devlink
In reply to Shannon NelsonAdd in the support for doing firmware updates. Of the two main banks available, a and b, this updates the one not in use and then selects it for the next boot.
Example:
devlink dev flash pci/0000:b2:00.0 \ file pensando/dsc_fw_1.63.0-22.tar
--- .../device_drivers/ethernet/amd/pds_core.rst | 10 + drivers/net/ethernet/amd/pds_core/Makefile | 3 +- drivers/net/ethernet/amd/pds_core/core.h | 5 + drivers/net/ethernet/amd/pds_core/devlink.c | 9 + drivers/net/ethernet/amd/pds_core/fw.c | 194 ++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 1 + 6 files changed, 221 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +10 -0
@@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions - fixed - The revision of the ASIC for this device +Firmware Management +=================== + +The ``flash`` command can update a the DSC firmware. The downloaded firmware +will be saved into either of firmware bank 1 or bank 2, whichever is not +currrently in use, and that bank will used for the next boot:: + + # devlink dev flash pci/0000:b5:00.0 \ + file pensando/dsc_fw_1.63.0-22.tar + Health Reporters ================
drivers/net/ethernet/amd/pds_core/Makefile +2 -1
@@ -7,6 +7,7 @@ pds_core-y := main.o \ devlink.o \ dev.o \ adminq.o \ - core.o + core.o \ + fw.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.h +5 -0
@@ -226,6 +226,9 @@ int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter, struct netlink_ext_ack *extack);int pdsc_dl_info_get (struct devlink *dl,struct devlink_info_req *req,struct netlink_ext_ack *extack); +int pdsc_dl_flash_update (struct devlink *dl, +struct devlink_flash_update_params *params, +struct netlink_ext_ack *extack);void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num);
@@ -280,4 +283,6 @@ void pdsc_process_adminq(struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data); +int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw, +struct netlink_ext_ack *extack);#endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/devlink.c +9 -0
@@ -3,6 +3,15 @@#include "core.h" +int pdsc_dl_flash_update (struct devlink *dl, +struct devlink_flash_update_params *params, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_priv (dl); + +return pdsc_firmware_update (pdsc, params->fw, extack); +} +static char *fw_slotnames[] = {"fw.goldfw" ,"fw.mainfwa" ,
drivers/net/ethernet/amd/pds_core/fw.c new file +194 -0
194 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +1 -0
@@ -242,6 +242,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)static const struct devlink_ops pdsc_dl_ops = { .info_get = pdsc_dl_info_get, + .flash_update = pdsc_dl_flash_update, };static const struct devlink_ops pdsc_dl_vf_ops = {
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink
Quoted 9 lines · Shannon Nelson
Add in the support for doing firmware updates. Of the two main banks available, a and b, this updates the one not in use and then selects it for the next boot.
Example:
devlink dev flash pci/0000:b2:00.0 \ file pensando/dsc_fw_1.63.0-22.tarSigned-off-by Shannon Nelson <[email protected]>
Hi Shannon,
some minor feedback from my side.
...
Quoted 5 lines
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +0
-0
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst The ``pds_core`` driver reports the following versions · line 81
Quoted 9 lines
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +6
-0@@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions - fixed - The revision of the ASIC for this device +Firmware Management +=================== + +The ``flash`` command can update a the DSC firmware. The downloaded firmware +will be saved into either of firmware bank 1 or bank 2, whichever is not +currrently in use, and that bank will used for the next boot::
nit: s/currrently/currently/
...
Quoted 1 line
drivers/net/ethernet/amd/pds_core/fw.c +0
-0
...
Quoted 47 lines
drivers/net/ethernet/amd/pds_core/fw.c +47
-0+int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw, +struct netlink_ext_ack *extack) +{ +u32 buf_sz, copy_sz, offset; +struct devlink *dl; +int next_interval; +u64 data_addr; +int err =0 ; +u8 fw_slot; + +dev_info (pdsc->dev,"Installing firmware\n" ); + + dl =priv_to_devlink (pdsc); +devlink_flash_update_status_notify (dl,"Preparing to flash" , +NULL ,0 ,0 ); + + buf_sz =sizeof (pdsc->cmd_regs->data); + +dev_dbg (pdsc->dev, +"downloading firmware - size %d part_sz %d nparts %lu\n" , + (int )fw->size, buf_sz,DIV_ROUND_UP (fw->size, buf_sz)); + + offset =0 ; + next_interval =0 ; + data_addr = offsetof(struct pds_core_dev_cmd_regs , data); +while (offset < fw->size) { +if (offset >= next_interval) { +devlink_flash_update_status_notify (dl,"Downloading" , +NULL , offset, + fw->size); + next_interval = offset + + (fw->size /PDSC_FW_INTERVAL_FRACTION ); + } + + copy_sz = min_t(unsigned int, buf_sz, fw->size - offset); +mutex_lock (&pdsc->devcmd_lock); +memcpy_toio (&pdsc->cmd_regs->data, fw->data + offset, copy_sz); + err =pdsc_devcmd_fw_download_locked (pdsc, data_addr, + offset, copy_sz); +mutex_unlock (&pdsc->devcmd_lock); +if (err) { +dev_err (pdsc->dev, +"download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n" , + offset, data_addr, copy_sz,ERR_PTR (err)); +NL_SET_ERR_MSG_MOD (extack,"Segment download failed" ); + goto err_out; + }
drivers/net/ethernet/amd/pds_core/fw.c line 154
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/fw.c +10
-0+ offset += copy_sz; + } +devlink_flash_update_status_notify (dl,"Downloading" ,NULL , + fw->size, fw->size); + +devlink_flash_update_timeout_notify (dl,"Installing" ,NULL , +PDSC_FW_INSTALL_TIMEOUT ); + + fw_slot =pdsc_devcmd_fw_install (pdsc); +if (fw_slot <0 ) {
The type of fs_slot is u8.
But the return type of pdsc_devcmd_fw_install is int, (I think) it can return negative error values, and that case is checked on the line above.
Perhaps the type of fw_slot should be int?
Flagged by Coccinelle as:
drivers/net/ethernet/amd/pds_core/fw.c:154:5-12: WARNING: Unsigned expression compared with zero: fw_slot < 0
And Smatch as:
drivers/net/ethernet/amd/pds_core/fw.c:154 pdsc_firmware_update() warn: impossible condition '(fw_slot < 0) => (0-255 < 0)'
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/fw.c +5
-0+ err = fw_slot; +dev_err (pdsc->dev,"install failed: %pe\n" ,ERR_PTR (err)); +NL_SET_ERR_MSG_MOD (extack,"Failed to start firmware install" ); + goto err_out; + }
...
Re: [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink
Quoted 10 lines · Simon Horman
Quoted 8 lines · Shannon Nelson
Add in the support for doing firmware updates. Of the two main banks available, a and b, this updates the one not in use and then selects it for the next boot.
Example:
devlink dev flash pci/0000:b2:00.0 \ file pensando/dsc_fw_1.63.0-22.tar
Quoted 5 lines
Quoted 1 line
Signed-off-by Shannon Nelson <[email protected]>Hi Shannon,
some minor feedback from my side.
Thanks, I'll take care of these.
sln
Quoted 114 lines
...
Quoted 5 lines
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +0
-0Documentation/networking/device_drivers/ethernet/amd/pds_core.rst The ``pds_core`` driver reports the following versions · line 81
Quoted 9 lines
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +6
-0@@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions - fixed - The revision of the ASIC for this device +Firmware Management +=================== + +The ``flash`` command can update a the DSC firmware. The downloaded firmware +will be saved into either of firmware bank 1 or bank 2, whichever is not +currrently in use, and that bank will used for the next boot::nit: s/currrently/currently/
...
Quoted 1 line
drivers/net/ethernet/amd/pds_core/fw.c +0
-0...
Quoted 47 lines
drivers/net/ethernet/amd/pds_core/fw.c +47
-0+int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw, +struct netlink_ext_ack *extack) +{ +u32 buf_sz, copy_sz, offset; +struct devlink *dl; +int next_interval; +u64 data_addr; +int err =0 ; +u8 fw_slot; + +dev_info (pdsc->dev,"Installing firmware\n" ); + + dl =priv_to_devlink (pdsc); +devlink_flash_update_status_notify (dl,"Preparing to flash" , +NULL ,0 ,0 ); + + buf_sz =sizeof (pdsc->cmd_regs->data); + +dev_dbg (pdsc->dev, +"downloading firmware - size %d part_sz %d nparts %lu\n" , + (int )fw->size, buf_sz,DIV_ROUND_UP (fw->size, buf_sz)); + + offset =0 ; + next_interval =0 ; + data_addr = offsetof(struct pds_core_dev_cmd_regs , data); +while (offset < fw->size) { +if (offset >= next_interval) { +devlink_flash_update_status_notify (dl,"Downloading" , +NULL , offset, + fw->size); + next_interval = offset + + (fw->size /PDSC_FW_INTERVAL_FRACTION ); + } + + copy_sz = min_t(unsigned int, buf_sz, fw->size - offset); +mutex_lock (&pdsc->devcmd_lock); +memcpy_toio (&pdsc->cmd_regs->data, fw->data + offset, copy_sz); + err =pdsc_devcmd_fw_download_locked (pdsc, data_addr, + offset, copy_sz); +mutex_unlock (&pdsc->devcmd_lock); +if (err) { +dev_err (pdsc->dev, +"download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n" , + offset, data_addr, copy_sz,ERR_PTR (err)); +NL_SET_ERR_MSG_MOD (extack,"Segment download failed" ); + goto err_out; + }drivers/net/ethernet/amd/pds_core/fw.c line 154
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/fw.c +10
-0+ offset += copy_sz; + } +devlink_flash_update_status_notify (dl,"Downloading" ,NULL , + fw->size, fw->size); + +devlink_flash_update_timeout_notify (dl,"Installing" ,NULL , +PDSC_FW_INSTALL_TIMEOUT ); + + fw_slot =pdsc_devcmd_fw_install (pdsc); +if (fw_slot <0 ) {The type of
fs_slotis u8.
But the return type ofpdsc_devcmd_fw_installis int, (I think) it can return negative error values, and that case is checked on the line above.Perhaps the type of
fw_slotshould be int?Flagged by Coccinelle as:
drivers/net/ethernet/amd/pds_core/fw.c:154:5-12: WARNING: Unsigned expression compared with zero:
fw_slot< 0And Smatch as:
drivers/net/ethernet/amd/pds_core/fw.c:154pdsc_firmware_update()warn: impossible condition '(fw_slot< 0) => (0-255 < 0)'drivers/net/ethernet/amd/pds_core/fw.c line 159
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/fw.c +5
-0+ err = fw_slot; +dev_err (pdsc->dev,"install failed: %pe\n" ,ERR_PTR (err)); +NL_SET_ERR_MSG_MOD (extack,"Failed to start firmware install" ); + goto err_out; + }...
--
You received this message because you are subscribed to the Google Groups "Pensando Drivers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/pensando.io/d/msgid/drivers/ZDQuxBlfH5foSEFA%40corigine.com.
PATCH v9 8/14 pds_core: set up the VIF definitions and defaults
In reply to Shannon NelsonThe Virtual Interfaces (VIFs) supported by the DSC's configuration (vDPA, Eth, RDMA, etc) are reported in the dev_ident struct and made visible in debugfs. At this point only vDPA is supported in this driver - the other interfaces are defined for future use but not yet realized.
--- drivers/net/ethernet/amd/pds_core/core.c | 48 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 12 ++++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 24 +++++++++++ include/linux/pds/pds_common.h | 19 ++++++++ 4 files changed, 103 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +48 -0
@@ -357,6 +357,43 @@ static int pdsc_core_init(struct pdsc *pdsc)return err; } +static struct pdsc_viftype pdsc_viftype_defaults[] = { + [PDS_DEV_TYPE_VDPA ] = { .name =PDS_DEV_TYPE_VDPA_STR , + .vif_id =PDS_DEV_TYPE_VDPA , + .dl_id =DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET }, + [PDS_DEV_TYPE_MAX ] = {0 } +}; + +static int pdsc_viftypes_init (struct pdsc *pdsc) +{ +enum pds_core_vif_types vt; + + pdsc->viftype_status =kzalloc (sizeof (pdsc_viftype_defaults), +GFP_KERNEL ); +if (!pdsc->viftype_status) +return -ENOMEM ; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +bool vt_support; + +if (!pdsc_viftype_defaults[vt].name) +continue ; + +/* Grab the defaults */ + pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt]; + +/* See what the Core device has for support */ + vt_support = !!le16_to_cpu (pdsc->dev_ident.vif_types[vt]); +dev_dbg (pdsc->dev,"VIF %s is %ssupported\n" , + pdsc->viftype_status[vt].name, + vt_support ?"" :"not " ); + + pdsc->viftype_status[vt].supported = vt_support; + } + +return 0 ; +} +int pdsc_setup (struct pdsc *pdsc,bool init) {int numdescs;
@@ -399,6 +436,14 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (err) goto err_out_teardown; +/* Set up the VIFs */ + err =pdsc_viftypes_init (pdsc); +if (err) + goto err_out_teardown; + +if (init) +pdsc_debugfs_add_viftype (pdsc); +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ;
@@ -415,6 +460,9 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)pdsc_qcq_free (pdsc, &pdsc->notifyqcq);pdsc_qcq_free (pdsc, &pdsc->adminqcq); +kfree (pdsc->viftype_status); + pdsc->viftype_status =NULL ; +if (pdsc->intr_info) {for (i =0 ; i < pdsc->nintrs; i++)pdsc_intr_free (pdsc, i);
drivers/net/ethernet/amd/pds_core/core.h +12 -0
@@ -123,6 +123,15 @@ struct pdsc_qcq {struct dentry *dentry; }; +struct pdsc_viftype { +char *name; +bool supported; +bool enabled; +int dl_id; +int vif_id; +struct pds_auxiliary_dev *padev; +}; +/* No state flags set means we are in a steady running state */ enum pdsc_state_flags {PDSC_S_FW_DEAD ,/* stopped, wait on startup or recovery */
@@ -174,6 +183,7 @@ struct pdsc {struct pdsc_qcq adminqcq;struct pdsc_qcq notifyqcq;u64 last_eid; +struct pdsc_viftype *viftype_status; };/** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -238,6 +248,7 @@ void pdsc_debugfs_destroy(void);void pdsc_debugfs_add_dev (struct pdsc *pdsc);void pdsc_debugfs_del_dev (struct pdsc *pdsc);void pdsc_debugfs_add_ident (struct pdsc *pdsc); +void pdsc_debugfs_add_viftype (struct pdsc *pdsc);void pdsc_debugfs_add_irqs (struct pdsc *pdsc);void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq);void pdsc_debugfs_del_qcq (struct pdsc_qcq *qcq);
@@ -247,6 +258,7 @@ static inline void pdsc_debugfs_destroy(void) { }static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_ident (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_add_viftype (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_irqs (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq) { }
drivers/net/ethernet/amd/pds_core/debugfs.c +24 -0
@@ -70,6 +70,30 @@ void pdsc_debugfs_add_ident(struct pdsc *pdsc) pdsc, &identity_fops); } +static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name) +continue ; + +seq_printf (seq,"%s\t%d supported %d enabled\n" , + pdsc->viftype_status[vt].name, + pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype); + +void pdsc_debugfs_add_viftype (struct pdsc *pdsc) +{ +debugfs_create_file ("viftypes" ,0400 , pdsc->dentry, + pdsc, &viftype_fops); +} +static int irqs_show (struct seq_file *seq,void *v) {struct pdsc *pdsc = seq->private;
include/linux/pds/pds_common.h +19 -0
@@ -51,6 +51,25 @@ enum pds_core_driver_type { PDS_DRIVER_ESXI = 6, }; +enum pds_core_vif_types { +PDS_DEV_TYPE_CORE =0 , +PDS_DEV_TYPE_VDPA =1 , +PDS_DEV_TYPE_VFIO =2 , +PDS_DEV_TYPE_ETH =3 , +PDS_DEV_TYPE_RDMA =4 , +PDS_DEV_TYPE_LM =5 , + +/* new ones added before this line */ +PDS_DEV_TYPE_MAX =16 /* don't change - used in struct size */ +}; + +#define PDS_DEV_TYPE_CORE_STR "Core" +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_DEV_TYPE_VFIO_STR "VFio" +#define PDS_DEV_TYPE_ETH_STR "Eth" +#define PDS_DEV_TYPE_RDMA_STR "RDMA" +#define PDS_DEV_TYPE_LM_STR "LM" +#define PDS_CORE_IFNAMSIZ 16/**
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
Quoted 5 lines · Shannon Nelson
The Virtual Interfaces (VIFs) supported by the DSC's configuration (vDPA, Eth, RDMA, etc) are reported in the
dev_identstruct and made visible in debugfs. At this point only vDPA is supported in this driver - the other interfaces are defined for future use but not yet realized.
Let's add only supported modes for now.
<...>
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +7
-0+static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name)
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_add_ident(struct pdsc *pdsc) · line 89
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +10
-0+continue ; + +seq_printf (seq,"%s\t%d supported %d enabled\n" , + pdsc->viftype_status[vt].name, + pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype);
I think that it is handled by devlink.
Thanks
Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Shannon Nelson
The Virtual Interfaces (VIFs) supported by the DSC's configuration (vDPA, Eth, RDMA, etc) are reported in the
dev_identstruct and made visible in debugfs. At this point only vDPA is supported in this driver - the other interfaces are defined for future use but not yet realized.Let's add only supported modes for now.
As stated, only the vDPA feature is supported in the driver.
Quoted 14 lines
<...>
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +11
-0+static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name) +continue ; + + seq_printf(seq, "%s\t%d supported %d enabled\n", + pdsc->viftype_status[vt].name,
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_add_ident(struct pdsc *pdsc) · line 89
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +6
-0+ pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype);I think that it is handled by devlink.
Yes, this is handled for those for which the driver sets up parameters (only vDPA), but that doesn't help when we want to see what the device is actually advertising. This shows us what the device is thinking, which we find useful in debugging, but it doesn't belong in devlink information.
Quoted 2 lines
Thanks
Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
Quoted 36 lines · Shannon Nelson
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Shannon Nelson
The Virtual Interfaces (VIFs) supported by the DSC's configuration (vDPA, Eth, RDMA, etc) are reported in the
dev_identstruct and made visible in debugfs. At this point only vDPA is supported in this driver - the other interfaces are defined for future use but not yet realized.Let's add only supported modes for now.
As stated, only the vDPA feature is supported in the driver.
Quoted 14 lines
<...>
Quoted 11 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +11
-0+static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name) +continue ; + + seq_printf(seq, "%s\t%d supported %d enabled\n", + pdsc->viftype_status[vt].name,Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_add_ident(struct pdsc *pdsc) · line 89
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +6
-0+ pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype);I think that it is handled by devlink.
Quoted 4 lines
Yes, this is handled for those for which the driver sets up parameters (only vDPA), but that doesn't help when we want to see what the device is actually advertising. This shows us what the device is thinking, which we find useful in debugging, but it doesn't belong in devlink information.
I'm not sure, users somehow need to know if they can or can't enable certain feature and it is impossible without advertising it.
All drivers which support auxbus need this feature.
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks
PATCH v9 9/14 pds_core: add initial VF device handling
In reply to Shannon NelsonThis is the initial VF PCI driver framework for the new pds_vdpa VF device, which will work in conjunction with an auxiliary_bus client of the pds_core driver. This does the very basics of registering for the new VF device, setting up debugfs entries, and registering with devlink.
--- drivers/net/ethernet/amd/pds_core/core.h | 9 +++++ drivers/net/ethernet/amd/pds_core/main.c | 48 +++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)
drivers/net/ethernet/amd/pds_core/core.h +9 -0
@@ -30,6 +30,12 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc_vf { +struct pds_auxiliary_dev *padev; +u16 index; +__le16 vif_types[PDS_DEV_TYPE_MAX ]; +}; +struct pdsc_devinfo {u8 asic_type;u8 asic_rev;
@@ -147,6 +153,9 @@ struct pdsc {struct dentry *dentry;struct device *dev;struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX ]; +struct pdsc_vf *vfs; +int num_vfs; +int vf_id;int hw_index;int uid;
drivers/net/ethernet/amd/pds_core/main.c +47 -1
74 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v9 10/14 pds_core: add auxiliary_bus devices
In reply to Shannon NelsonAn auxiliary_bus device is created for each vDPA type VF at VF probe and destroyed at VF remove. The aux device name comes from the driver name + VIF type + the unique id assigned at PCI probe. The VFs are always removed on PF remove, so there should be no issues with VFs trying to access missing PF structures.
The auxiliary_device names will look like "pds_core.vDPA.nn" where 'nn' is the VF's uid.
--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 6 ++ drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++- include/linux/pds/pds_auxbus.h | 16 +++ include/linux/pds/pds_common.h | 1 + 6 files changed, 170 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c create mode 100644 include/linux/pds/pds_auxbus.h
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ + auxbus.o \ dev.o \ adminq.o \ core.o \
drivers/net/ethernet/amd/pds_core/auxbus.c new file +112 -0
112 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/core.h +6 -0
@@ -30,8 +30,11 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc ; +struct pdsc_vf {struct pds_auxiliary_dev *padev; +struct pdsc *vf;u16 index;__le16 vif_types[PDS_DEV_TYPE_MAX ]; };
@@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf); +int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf); +void pdsc_process_adminq (struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data);
drivers/net/ethernet/amd/pds_core/main.c +34 -2
@@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)static int pdsc_init_vf (struct pdsc *vf) {struct devlink *dl; +struct pdsc *pf; +int err; + + pf =pdsc_get_pf_struct (vf->pdev); +if (IS_ERR_OR_NULL (pf)) +return PTR_ERR (pf) ?:-1 ; vf->vf_id =pci_iov_vf_id (vf->pdev);
@@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)devl_register (dl);devl_unlock (dl); -return 0 ; + pf->vfs[vf->vf_id].vf = vf; + err =pdsc_auxbus_dev_add_vf (vf, pf); +if (err) { +devl_lock (dl); +devl_unregister (dl); +devl_unlock (dl); + } + +return err; }static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
@@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev) }devl_unlock (dl); -if (!pdev->is_virtfn) { +if (pdev->is_virtfn) { +struct pdsc *pf; + + pf =pdsc_get_pf_struct (pdsc->pdev); +if (!IS_ERR (pf)) { +pdsc_auxbus_dev_del_vf (pdsc, pf); + pf->vfs[pdsc->vf_id].vf =NULL ; + } + }else { +/* Remove the VFs and their aux_bus connections before other +* cleanup so that the clients can use the AdminQ to cleanly +* shut themselves down. +*/ pdsc_sriov_configure (pdev,0 );del_timer_sync (&pdsc->wdtimer);
@@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = { .sriov_configure = pdsc_sriov_configure, }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev) +{ +return pci_iov_get_pf_drvdata (vf_pdev, &pdsc_driver); +} +EXPORT_SYMBOL_GPL (pdsc_get_pf_struct); +static int __init pdsc_init_module (void ) {pdsc_debugfs_create ();
include/linux/pds/pds_auxbus.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev { +struct auxiliary_device aux_dev; +struct pci_dev *vf_pdev; +struct pci_dev *pf_pdev; +u16 client_id; +void *priv; +}; +#endif /* _PDSC_AUXBUS_H_ */
include/linux/pds/pds_common.h +1 -0
@@ -91,4 +91,5 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
Quoted 59 lines · Shannon Nelson
An
auxiliary_busdevice is created for each vDPA type VF at VF probe and destroyed at VF remove. The aux device name comes from the driver name + VIF type + the unique id assigned at PCI probe. The VFs are always removed on PF remove, so there should be no issues with VFs trying to access missing PF structures.The
auxiliary_devicenames will look like "pds_core.vDPA.nn" where 'nn' is the VF's uid.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 6 ++ drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++- include/linux/pds/pds_auxbus.h | 16 +++ include/linux/pds/pds_common.h | 1 + 6 files changed, 170 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c create mode 100644 include/linux/pds/pds_auxbus.hdrivers/net/ethernet/amd/pds_core/Makefile +1
-0@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ + auxbus.o \ dev.o \ adminq.o \ core.o \drivers/net/ethernet/amd/pds_core/auxbus.c new file +20
-0@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pci.h> + +#include "core.h" +#include <linux/pds/pds_auxbus.h> + +static void pdsc_auxbus_dev_release (struct device *dev) +{ +struct pds_auxiliary_dev *padev = + container_of(dev, struct pds_auxiliary_dev, aux_dev.dev); + +kfree (padev); +} + +static struct pds_auxiliary_dev *pdsc_auxbus_dev_register (struct pdsc *vf, +struct pdsc *pf, +char *name) +{
drivers/net/ethernet/amd/pds_core/auxbus.c line 30
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +10
-0+struct auxiliary_device *aux_dev; +struct pds_auxiliary_dev *padev; +int err; + + padev =kzalloc (sizeof (*padev),GFP_KERNEL ); +if (!padev) +return ERR_PTR (-ENOMEM ); + + padev->vf_pdev = vf->pdev; + padev->pf_pdev = pf->pdev;
Why do you need to store pointer to PF device in your VF devices? pci_physfn() will return it from vf_pdev for you.
Quoted 66 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +66
-0+ + aux_dev = &padev->aux_dev; + aux_dev->name = name; + aux_dev->id = vf->uid; + aux_dev->dev.parent = vf->dev; + aux_dev->dev.release = pdsc_auxbus_dev_release; + + err =auxiliary_device_init (aux_dev); +if (err <0 ) { +dev_warn (vf->dev,"auxiliary_device_init of %s failed: %pe\n" , + name,ERR_PTR (err)); + goto err_out; + } + + err =auxiliary_device_add (aux_dev); +if (err) { +dev_warn (vf->dev,"auxiliary_device_add of %s failed: %pe\n" , + name,ERR_PTR (err)); + goto err_out_uninit; + } + +return padev; + +err_out_uninit: +auxiliary_device_uninit (aux_dev); +err_out: +kfree (padev); +return ERR_PTR (err); +} + +int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf) +{ +struct pds_auxiliary_dev *padev; +int err =0 ; + +mutex_lock (&pf->config_lock); + + padev = pf->vfs[vf->vf_id].padev; +if (padev) { +auxiliary_device_delete (&padev->aux_dev); +auxiliary_device_uninit (&padev->aux_dev); + } + pf->vfs[vf->vf_id].padev =NULL ; + +mutex_unlock (&pf->config_lock); +return err; +} + +int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf) +{ +struct pds_auxiliary_dev *padev; +enum pds_core_vif_types vt; +int err =0 ; + +mutex_lock (&pf->config_lock); + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +u16 vt_support; + +/* Verify that the type is supported and enabled */ + vt_support = !!le16_to_cpu (pf->dev_ident.vif_types[vt]); +if (!(vt_support && + pf->viftype_status[vt].supported && + pf->viftype_status[vt].enabled)) +continue ; +
drivers/net/ethernet/amd/pds_core/auxbus.c line 106
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +10
-0+ padev =pdsc_auxbus_dev_register (vf, pf, + pf->viftype_status[vt].name); +if (IS_ERR (padev)) { + err =PTR_ERR (padev); + goto out_unlock; + } + pf->vfs[vf->vf_id].padev = padev; + +/* We only support a single type per VF, so jump out here */ +break ;
You need to decide, or you implement loop correctly (without break and proper unfolding) or you don't implement loop yet at all.
And can we please find another name for functions and parameters which don't include VF in it as it is not correct anymore.
In ideal world, it will be great to have same probe flow for PF and VF while everything is controlled through FW and auxbus. For PF, you won't advertise any aux devices, but the flow will continue to be the same.
Thanks
Quoted 135 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +6
-0+ } + +out_unlock: +mutex_unlock (&pf->config_lock); +return err; +}drivers/net/ethernet/amd/pds_core/core.h +6
-0@@ -30,8 +30,11 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc ; +struct pdsc_vf {struct pds_auxiliary_dev *padev; +struct pdsc *vf;u16 index;__le16 vif_types[PDS_DEV_TYPE_MAX ]; };@@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf); +int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf); +void pdsc_process_adminq (struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data);drivers/net/ethernet/amd/pds_core/main.c +34
-2@@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)static int pdsc_init_vf (struct pdsc *vf) {struct devlink *dl; +struct pdsc *pf; +int err; + + pf =pdsc_get_pf_struct (vf->pdev); +if (IS_ERR_OR_NULL (pf)) +return PTR_ERR (pf) ?:-1 ; vf->vf_id =pci_iov_vf_id (vf->pdev);@@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)devl_register (dl);devl_unlock (dl); -return 0 ; + pf->vfs[vf->vf_id].vf = vf; + err =pdsc_auxbus_dev_add_vf (vf, pf); +if (err) { +devl_lock (dl); +devl_unregister (dl); +devl_unlock (dl); + } + +return err; }static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {@@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev) }devl_unlock (dl); -if (!pdev->is_virtfn) { +if (pdev->is_virtfn) { +struct pdsc *pf; + + pf =pdsc_get_pf_struct (pdsc->pdev); +if (!IS_ERR (pf)) { +pdsc_auxbus_dev_del_vf (pdsc, pf); + pf->vfs[pdsc->vf_id].vf =NULL ; + } + }else { +/* Remove the VFs and their aux_bus connections before other +* cleanup so that the clients can use the AdminQ to cleanly +* shut themselves down. +*/ pdsc_sriov_configure (pdev,0 );del_timer_sync (&pdsc->wdtimer);@@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = { .sriov_configure = pdsc_sriov_configure, }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev) +{ +return pci_iov_get_pf_drvdata (vf_pdev, &pdsc_driver); +} +EXPORT_SYMBOL_GPL (pdsc_get_pf_struct); +static int __init pdsc_init_module (void ) {pdsc_debugfs_create ();include/linux/pds/pds_auxbus.h new file +16
-0@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev { +struct auxiliary_device aux_dev; +struct pci_dev *vf_pdev; +struct pci_dev *pf_pdev; +u16 client_id; +void *priv; +}; +#endif /* _PDSC_AUXBUS_H_ */ include/linux/pds/pds_common.h +1
-0@@ -91,4 +91,5 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */ Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
Quoted 66 lines · Leon Romanovsky
Quoted 64 lines · Shannon Nelson
An
auxiliary_busdevice is created for each vDPA type VF at VF probe and destroyed at VF remove. The aux device name comes from the driver name + VIF type + the unique id assigned at PCI probe. The VFs are always removed on PF remove, so there should be no issues with VFs trying to access missing PF structures.The
auxiliary_devicenames will look like "pds_core.vDPA.nn" where 'nn' is the VF's uid.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 6 ++ drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++- include/linux/pds/pds_auxbus.h | 16 +++ include/linux/pds/pds_common.h | 1 + 6 files changed, 170 insertions(+), 2 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/auxbus.ccreate mode 100644include/linux/pds/pds_auxbus.hdrivers/net/ethernet/amd/pds_core/Makefile +1
-0@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ + auxbus.o \ dev.o \ adminq.o \ core.o \drivers/net/ethernet/amd/pds_core/auxbus.c new file +25
-0@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pci.h> + +#include "core.h" +#include <linux/pds/pds_auxbus.h> + +static void pdsc_auxbus_dev_release (struct device *dev) +{ +struct pds_auxiliary_dev *padev = + container_of(dev, struct pds_auxiliary_dev, aux_dev.dev); + +kfree (padev); +} + +static struct pds_auxiliary_dev *pdsc_auxbus_dev_register (struct pdsc *vf, +struct pdsc *pf, +char *name) +{ +struct auxiliary_device *aux_dev; +struct pds_auxiliary_dev *padev; +int err; + + padev =kzalloc (sizeof (*padev),GFP_KERNEL );
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/auxbus.c line 30
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +5
-0+if (!padev) +return ERR_PTR (-ENOMEM ); + + padev->vf_pdev = vf->pdev; + padev->pf_pdev = pf->pdev;Why do you need to store pointer to PF device in your VF devices?
pci_physfn()will return it fromvf_pdevfor you.
will fix
Quoted 72 lines
Quoted 72 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +71
-0+ + aux_dev = &padev->aux_dev; + aux_dev->name = name; + aux_dev->id = vf->uid; + aux_dev->dev.parent = vf->dev; + aux_dev->dev.release = pdsc_auxbus_dev_release; + + err =auxiliary_device_init (aux_dev); +if (err <0 ) { +dev_warn (vf->dev,"auxiliary_device_init of %s failed: %pe\n" , + name,ERR_PTR (err)); + goto err_out; + } + + err =auxiliary_device_add (aux_dev); +if (err) { +dev_warn (vf->dev,"auxiliary_device_add of %s failed: %pe\n" , + name,ERR_PTR (err)); + goto err_out_uninit; + } + +return padev; + +err_out_uninit: +auxiliary_device_uninit (aux_dev); +err_out: +kfree (padev); +return ERR_PTR (err); +} + +int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf) +{ +struct pds_auxiliary_dev *padev; +int err =0 ; + +mutex_lock (&pf->config_lock); + + padev = pf->vfs[vf->vf_id].padev; +if (padev) { +auxiliary_device_delete (&padev->aux_dev); +auxiliary_device_uninit (&padev->aux_dev); + } + pf->vfs[vf->vf_id].padev =NULL ; + +mutex_unlock (&pf->config_lock); +return err; +} + +int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf) +{ +struct pds_auxiliary_dev *padev; +enum pds_core_vif_types vt; +int err =0 ; + +mutex_lock (&pf->config_lock); + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +u16 vt_support; + +/* Verify that the type is supported and enabled */ + vt_support = !!le16_to_cpu (pf->dev_ident.vif_types[vt]); +if (!(vt_support && + pf->viftype_status[vt].supported && + pf->viftype_status[vt].enabled)) +continue ; + + padev =pdsc_auxbus_dev_register (vf, pf, + pf->viftype_status[vt].name); +if (IS_ERR (padev)) { + err =PTR_ERR (padev); + goto out_unlock;
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/auxbus.c line 106
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +5
-0+ } + pf->vfs[vf->vf_id].padev = padev; + +/* We only support a single type per VF, so jump out here */ +break ;You need to decide, or you implement loop correctly (without break and proper unfolding) or you don't implement loop yet at all.
Sure - we can simplify specifically for the single case we have for now and add the loop later when new features are added.
Quoted 7 lines
And can we please find another name for functions and parameters which don't include VF in it as it is not correct anymore.
In ideal world, it will be great to have same probe flow for PF and VF while everything is controlled through FW and auxbus. For PF, you won't advertise any aux devices, but the flow will continue to be the same.
Since we currently only have VFs and not more finely grained sub-functions, these seem to still make sense and help define the context of the operations. I can find places where we can reduce the use of 'VF'. Would you prefer PF and SF to PF and VF where the difference is important?
Quoted 138 lines
Thanks
Quoted 135 lines
drivers/net/ethernet/amd/pds_core/auxbus.c new file +6
-0+ } + +out_unlock: +mutex_unlock (&pf->config_lock); +return err; +}drivers/net/ethernet/amd/pds_core/core.h +6
-0@@ -30,8 +30,11 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc ; +struct pdsc_vf {struct pds_auxiliary_dev *padev; +struct pdsc *vf;u16 index;__le16 vif_types[PDS_DEV_TYPE_MAX ]; };@@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf); +int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf); +void pdsc_process_adminq (struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data);drivers/net/ethernet/amd/pds_core/main.c +34
-2@@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)static int pdsc_init_vf (struct pdsc *vf) {struct devlink *dl; +struct pdsc *pf; +int err; + + pf =pdsc_get_pf_struct (vf->pdev); +if (IS_ERR_OR_NULL (pf)) +return PTR_ERR (pf) ?:-1 ; vf->vf_id =pci_iov_vf_id (vf->pdev);@@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)devl_register (dl);devl_unlock (dl); -return 0 ; + pf->vfs[vf->vf_id].vf = vf; + err =pdsc_auxbus_dev_add_vf (vf, pf); +if (err) { +devl_lock (dl); +devl_unregister (dl); +devl_unlock (dl); + } + +return err; }static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {@@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev) }devl_unlock (dl); -if (!pdev->is_virtfn) { +if (pdev->is_virtfn) { +struct pdsc *pf; + + pf =pdsc_get_pf_struct (pdsc->pdev); +if (!IS_ERR (pf)) { +pdsc_auxbus_dev_del_vf (pdsc, pf); + pf->vfs[pdsc->vf_id].vf =NULL ; + } + }else { +/* Remove the VFs and their aux_bus connections before other +* cleanup so that the clients can use the AdminQ to cleanly +* shut themselves down. +*/ pdsc_sriov_configure (pdev,0 );del_timer_sync (&pdsc->wdtimer);@@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = { .sriov_configure = pdsc_sriov_configure, }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev) +{ +return pci_iov_get_pf_drvdata (vf_pdev, &pdsc_driver); +} +EXPORT_SYMBOL_GPL (pdsc_get_pf_struct); +static int __init pdsc_init_module (void ) {pdsc_debugfs_create ();include/linux/pds/pds_auxbus.h new file +16
-0@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev { +struct auxiliary_device aux_dev; +struct pci_dev *vf_pdev; +struct pci_dev *pf_pdev; +u16 client_id; +void *priv; +}; +#endif /* _PDSC_AUXBUS_H_ */ include/linux/pds/pds_common.h +1
-1@@ -91,4 +91,5 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */ --2.17.1
Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
Quoted 23 lines · Shannon Nelson
Quoted 22 lines · Leon Romanovsky
Quoted 20 lines · Shannon Nelson
An
auxiliary_busdevice is created for each vDPA type VF at VF probe and destroyed at VF remove. The aux device name comes from the driver name + VIF type + the unique id assigned at PCI probe. The VFs are always removed on PF remove, so there should be no issues with VFs trying to access missing PF structures.The
auxiliary_devicenames will look like "pds_core.vDPA.nn" where 'nn' is the VF's uid.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 6 ++ drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++- include/linux/pds/pds_auxbus.h | 16 +++ include/linux/pds/pds_common.h | 1 + 6 files changed, 170 insertions(+), 2 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/auxbus.ccreate mode 100644include/linux/pds/pds_auxbus.h
<...>
Quoted 4 lines
Quoted 4 lines
And can we please find another name for functions and parameters which don't include VF in it as it is not correct anymore.
Quoted 8 lines
Quoted 3 lines
In ideal world, it will be great to have same probe flow for PF and VF while everything is controlled through FW and auxbus. For PF, you won't advertise any aux devices, but the flow will continue to be the same.
Since we currently only have VFs and not more finely grained sub-functions, these seem to still make sense and help define the context of the operations. I can find places where we can reduce the use of 'VF'. Would you prefer PF and SF to PF and VF where the difference is important?
I'm talking about VF names in auxbus code. It simply doesn't belong there as you are creating/deleting auxiliary devices for specific physical devices. They by chance your VFs.
"PF/VF probe flow" is my wishful thinking, it is fine to leave it as is.
Thanks
PATCH v9 11/14 pds_core: devlink params for enabling VIF support
In reply to Shannon NelsonAdd the devlink parameter switches so the user can enable the features supported by the VFs. The only feature supported at the moment is vDPA.
Example:
devlink dev param set pci/0000:2b:00.0 \
name enable_vnet cmode runtime value true
--- .../device_drivers/ethernet/amd/pds_core.rst | 19 +++++ drivers/net/ethernet/amd/pds_core/core.h | 7 ++ drivers/net/ethernet/amd/pds_core/devlink.c | 73 +++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 34 +++++++-- 4 files changed, 127 insertions(+), 6 deletions(-)
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +19 -0
@@ -73,6 +73,25 @@ The ``pds_core`` driver reports the following versions - fixed - The revision of the ASIC for this device +Parameters +========== + +The ``pds_core`` driver implements the following generic +parameters for controlling the functionality to be made available +as auxiliary_bus devices. + +.. list-table:: Generic parameters implemented + :widths: 5 5 8 82 + + * - Name + - Mode + - Type + - Description + * - ``enable_vnet`` + - runtime + - Boolean + - Enables vDPA functionality through an auxiliary_bus device + Firmware Management ===================
drivers/net/ethernet/amd/pds_core/core.h +7 -0
@@ -251,6 +251,13 @@ int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,int pdsc_dl_flash_update (struct devlink *dl,struct devlink_flash_update_params *params,struct netlink_ext_ack *extack); +int pdsc_dl_enable_get (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx); +int pdsc_dl_enable_set (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx); +int pdsc_dl_enable_validate (struct devlink *dl,u32 id, +union devlink_param_value val, +struct netlink_ext_ack *extack);void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num);
drivers/net/ethernet/amd/pds_core/devlink.c +73 -0
79 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +28 -6
@@ -210,6 +210,14 @@ static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = { .diagnose = pdsc_fw_reporter_diagnose, }; +static const struct devlink_param pdsc_dl_params[] = { +DEVLINK_PARAM_GENERIC (ENABLE_VNET , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate), +}; +#define PDSC_WQ_NAME_LEN 24static int pdsc_init_pf (struct pdsc *pdsc)
@@ -257,13 +265,19 @@ static int pdsc_init_pf(struct pdsc *pdsc) dl =priv_to_devlink (pdsc);devl_lock (dl); + err =devl_params_register (dl, pdsc_dl_params, +ARRAY_SIZE (pdsc_dl_params)); +if (err) { +dev_warn (pdsc->dev,"Failed to register devlink params: %pe\n" , +ERR_PTR (err)); + goto err_out_unlock_dl; + } hr =devl_health_reporter_create (dl, &pdsc_fw_reporter_ops,0 , pdsc);if (IS_ERR (hr)) {dev_warn (pdsc->dev,"Failed to create fw reporter: %pe\n" , hr); err =PTR_ERR (hr); -devl_unlock (dl); - goto err_out_stop; + goto err_out_unreg_params; } pdsc->fw_reporter = hr;
@@ -275,7 +289,11 @@ static int pdsc_init_pf(struct pdsc *pdsc)return 0 ; -err_out_stop: +err_out_unreg_params: +devl_params_unregister (dl, pdsc_dl_params, +ARRAY_SIZE (pdsc_dl_params)); +err_out_unlock_dl: +devl_unlock (dl);pdsc_stop (pdsc); err_out_teardown:pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );
@@ -387,9 +405,13 @@ static void pdsc_remove(struct pci_dev *pdev) dl =priv_to_devlink (pdsc);devl_lock (dl);devl_unregister (dl); -if (pdsc->fw_reporter) { -devl_health_reporter_destroy (pdsc->fw_reporter); - pdsc->fw_reporter =NULL ; +if (!pdev->is_virtfn) { +if (pdsc->fw_reporter) { +devl_health_reporter_destroy (pdsc->fw_reporter); + pdsc->fw_reporter =NULL ; + } +devl_params_unregister (dl, pdsc_dl_params, +ARRAY_SIZE (pdsc_dl_params)); }devl_unlock (dl);
Signature · 1 line
2.17.1
PATCH v9 12/14 pds_core: add the aux client API
In reply to Shannon NelsonAdd the client API operations for running adminq commands. The core registers the client with the FW, then the client has a context for requesting adminq services. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet.
--- drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 28 +++++ 2 files changed, 160 insertions(+), 3 deletions(-)
drivers/net/ethernet/amd/pds_core/auxbus.c +132 -3
@@ -6,6 +6,115 @@#include "core.h" #include <linux/pds/pds_auxbus.h> +static int pds_client_register (struct pdsc *vf,struct pdsc *pf,char *name) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +int err; +u16 ci; + +if (pf->state) +return -ENXIO ; + + cmd.client_reg.opcode =PDS_AQ_CMD_CLIENT_REG ; +snprintf (cmd.client_reg.devname,sizeof (cmd.client_reg.devname), +"%s.%s.%d" ,PDS_CORE_DRV_NAME , name, vf->uid); + + err =pdsc_adminq_post (pf, &cmd, &comp, false); +if (err) { +dev_info (pf->dev,"register dev_name %s with DSC failed, status %d: %pe\n" , + name, comp.status,ERR_PTR (err)); +return err; + } + + ci =le16_to_cpu (comp.client_reg.client_id); +if (!ci) { +dev_err (pf->dev,"%s: device returned null client_id\n" , + __func__); +return -EIO ; + } + +dev_dbg (pf->dev,"%s: device returned client_id %d for %s\n" , + __func__, ci, cmd.client_reg.devname); + +return ci; +} + +static int pds_client_unregister (struct pdsc *pf,u16 client_id) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +int err; + +if (pf->state) +return -ENXIO ; + + cmd.client_unreg.opcode =PDS_AQ_CMD_CLIENT_UNREG ; + cmd.client_unreg.client_id =cpu_to_le16 (client_id); + + err =pdsc_adminq_post (pf, &cmd, &comp, false); +if (err) +dev_info (pf->dev,"unregister client_id %d failed, status %d: %pe\n" , + client_id, comp.status,ERR_PTR (err)); + +return err; +} + +/** +* pds_client_adminq_cmd - Process an adminq request for the client +* @padev: ptr to the client device +* @req: ptr to buffer with request +* @req_len: length of actual struct used for request +* @resp: ptr to buffer where answer is to be copied +* @flags: optional flags from pds_core_adminq_flags +* +* Return: 0 on success, or +* negative for error +* +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +static int pds_client_adminq_cmd (struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags) +{ +union pds_core_adminq_cmd cmd = {0 }; +struct pdsc *pf; +size_t cp_len; +int err; + + pf =pci_get_drvdata (padev->pf_pdev); + +dev_dbg (pf->dev,"%s: %s opcode %d\n" , + __func__,dev_name (&padev->aux_dev.dev), req->opcode); + +if (pf->state) +return -ENXIO ; + +/* Wrap the client's request */ + cmd.client_request.opcode =PDS_AQ_CMD_CLIENT_CMD ; + cmd.client_request.client_id =cpu_to_le16 (padev->client_id); + cp_len =min_t (size_t, req_len,sizeof (cmd.client_request.client_cmd)); +memcpy (cmd.client_request.client_cmd, req, cp_len); + + err =pdsc_adminq_post (pf, &cmd, resp, + !!(flags &PDS_AQ_FLAG_FASTPOLL )); +if (err && err != -EAGAIN ) +dev_info (pf->dev,"client admin cmd failed: %pe\n" , +ERR_PTR (err)); + +return err; +} + +static struct pds_core_ops pds_core_ops = { + .adminq_cmd = pds_client_adminq_cmd, +}; +static void pdsc_auxbus_dev_release (struct device *dev) { struct pds_auxiliary_dev *padev =
@@ -16,7 +125,9 @@ static void pdsc_auxbus_dev_release(struct device *dev)static struct pds_auxiliary_dev *pdsc_auxbus_dev_register (struct pdsc *vf,struct pdsc *pf, -char *name) +u16 client_id, +char *name, +struct pds_core_ops *ops) {struct auxiliary_device *aux_dev;struct pds_auxiliary_dev *padev;
@@ -28,6 +139,8 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf, padev->vf_pdev = vf->pdev; padev->pf_pdev = pf->pdev; + padev->ops = ops; + padev->client_id = client_id; aux_dev = &padev->aux_dev; aux_dev->name = name;
@@ -67,8 +180,10 @@ int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf) padev = pf->vfs[vf->vf_id].padev;if (padev) { +pds_client_unregister (pf, padev->client_id);auxiliary_device_delete (&padev->aux_dev);auxiliary_device_uninit (&padev->aux_dev); + padev->client_id =0 ; } pf->vfs[vf->vf_id].padev =NULL ;
@@ -80,6 +195,7 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf) {struct pds_auxiliary_dev *padev;enum pds_core_vif_types vt; +int client_id;int err =0 ;mutex_lock (&pf->config_lock);
@@ -94,9 +210,22 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf) pf->viftype_status[vt].enabled))continue ; - padev =pdsc_auxbus_dev_register (vf, pf, - pf->viftype_status[vt].name); +/* need to register with FW and get the client_id before +* creating the aux device so that the aux client can run +* adminq commands as part its probe +*/ + client_id =pds_client_register (vf, pf, + pf->viftype_status[vt].name); +if (client_id <0 ) { + err = client_id; + goto out_unlock; + } + + padev =pdsc_auxbus_dev_register (vf, pf, client_id, + pf->viftype_status[vt].name, + &pds_core_ops);if (IS_ERR (padev)) { +pds_client_unregister (pf, client_id); err =PTR_ERR (padev); goto out_unlock; }
include/linux/pds/pds_auxbus.h +28 -0
@@ -10,7 +10,35 @@ struct pds_auxiliary_dev {struct auxiliary_device aux_dev;struct pci_dev *vf_pdev;struct pci_dev *pf_pdev; +struct pds_core_ops *ops;u16 client_id; void *priv; }; + +/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};#endif /* _PDSC_AUXBUS_H_ */
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
Quoted 12 lines · Shannon Nelson
Add the client API operations for running adminq commands. The core registers the client with the FW, then the client has a context for requesting adminq services. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 28 +++++ 2 files changed, 160 insertions(+), 3 deletions(-)
<...>
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +4
-0+static struct pds_core_ops pds_core_ops = { + .adminq_cmd = pds_client_adminq_cmd, +}; +
<...>
include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 13
Quoted 9 lines
include/linux/pds/pds_auxbus.h +1
-0@@ -10,7 +10,35 @@ struct pds_auxiliary_dev {struct auxiliary_device aux_dev;struct pci_dev *vf_pdev;struct pci_dev *pf_pdev; +struct pds_core_ops *ops;
I honestly don't understand why pds_core functionality is espoused through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks and that functionality shouldn't operate on auxdevice.
Thanks
Quoted 34 lines
include/linux/pds/pds_auxbus.h +27
-0u16 client_id; void *priv; }; + +/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};#endif /* _PDSC_AUXBUS_H_ */ Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
Quoted 24 lines · Leon Romanovsky
Quoted 12 lines · Shannon Nelson
Add the client API operations for running adminq commands. The core registers the client with the FW, then the client has a context for requesting adminq services. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 28 +++++ 2 files changed, 160 insertions(+), 3 deletions(-)<...>
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +4
-0+static struct pds_core_ops pds_core_ops = { + .adminq_cmd = pds_client_adminq_cmd, +}; +<...>
Quoted 13 lines
include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 13
Quoted 9 lines
include/linux/pds/pds_auxbus.h +1
-0@@ -10,7 +10,35 @@ struct pds_auxiliary_dev {struct auxiliary_device aux_dev;struct pci_dev *vf_pdev;struct pci_dev *pf_pdev; +struct pds_core_ops *ops;I honestly don't understand why
pds_corefunctionality is espoused through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks and that functionality shouldn't operate on auxdevice.
The original design had several more operations and wrapped all the interaction into a single defined interface rather that polluting the kernel with additional direct EXPORTed functions from the PF. Since much has disappeared as we simplified the interface and don't yet have use for some of them, this ops struct with its single entry is the last vestige of that idea.
Perhaps it is time to put it away and add one more EXPORTed function. We can revisit this idea if/when the interface grows again.
Quoted 37 lines
Thanks
Quoted 34 lines
include/linux/pds/pds_auxbus.h +27
-1u16 client_id; void *priv; }; + +/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};#endif /* _PDSC_AUXBUS_H_ */ --2.17.1
Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
Quoted 15 lines · Shannon Nelson
Quoted 14 lines · Leon Romanovsky
Quoted 12 lines · Shannon Nelson
Add the client API operations for running adminq commands. The core registers the client with the FW, then the client has a context for requesting adminq services. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 28 +++++ 2 files changed, 160 insertions(+), 3 deletions(-)
<...>
Quoted 22 lines
Quoted 14 lines
include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 13
Quoted 10 lines
include/linux/pds/pds_auxbus.h +1
-0@@ -10,7 +10,35 @@ struct pds_auxiliary_dev {struct auxiliary_device aux_dev;struct pci_dev *vf_pdev;struct pci_dev *pf_pdev; +struct pds_core_ops *ops;I honestly don't understand why
pds_corefunctionality is espoused through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks and that functionality shouldn't operate on auxdevice.The original design had several more operations and wrapped all the interaction into a single defined interface rather that polluting the kernel with additional direct EXPORTed functions from the PF. Since much has disappeared as we simplified the interface and don't yet have use for some of them, this ops struct with its single entry is the last vestige of that idea.
Quoted 2 lines
Perhaps it is time to put it away and add one more EXPORTed function. We can revisit this idea if/when the interface grows again.
Thanks
PATCH v9 13/14 pds_core: publish events to the clients
In reply to Shannon NelsonWhen the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to the clients.
The entry points pdsc_register_notify() and pdsc_unregister_notify() are EXPORTed for other drivers that want to listen for these events.
--- drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++ drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 3 ++ include/linux/pds/pds_common.h | 2 ++ 4 files changed, 39 insertions(+)
drivers/net/ethernet/amd/pds_core/adminq.c +2 -0
@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_RESET , comp);break ;case PDS_EVENT_XCVR :
drivers/net/ethernet/amd/pds_core/core.c +32 -0
@@ -6,6 +6,25 @@#include "core.h" +static BLOCKING_NOTIFIER_HEAD (pds_notify_chain); + +int pdsc_register_notify (struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_register_notify); + +void pdsc_unregister_notify (struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_unregister_notify); + +void pdsc_notify (unsigned long event,void *data) +{ +blocking_notifier_call_chain (&pds_notify_chain, event, data); +} +void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;
@@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)static void pdsc_fw_down (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =0 , + }; +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) {dev_err (pdsc->dev,"%s: already happening\n" , __func__);return ; } +/* Notify clients of fw_down */ devlink_health_report (pdsc->fw_reporter,"FW down reported" , pdsc); +pdsc_notify (PDS_EVENT_RESET , &reset_event);pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );
@@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)static void pdsc_fw_up (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =1 , + };int err;if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) {
@@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; +/* Notify clients of fw_up */ pdsc->fw_recoveries++;devlink_health_reporter_state_update (pdsc->fw_reporter,DEVLINK_HEALTH_REPORTER_STATE_HEALTHY ); +pdsc_notify (PDS_EVENT_RESET , &reset_event);return ;
drivers/net/ethernet/amd/pds_core/core.h +3 -0
@@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb); +void pdsc_notify (unsigned long event,void *data);int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf);int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf);
include/linux/pds/pds_common.h +2 -0
@@ -91,5 +91,7 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb);void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 21 lines · Shannon Nelson
When the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to the clients.
The entry points
pdsc_register_notify()andpdsc_unregister_notify()are EXPORTed for other drivers that want to listen for these events.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++ drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 3 ++ include/linux/pds/pds_common.h | 2 ++ 4 files changed, 39 insertions(+)drivers/net/ethernet/amd/pds_core/adminq.c +0
-0
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 30
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);
Aren't you "resending" standard netdev event?
It will be better to send only custom, specific to pds_core events, while leaving general ones to netdev.
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 36
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_RESET , comp);
We can argue if clients should get this event. Once reset is detected, the pds_core should close devices by deleting aux drivers.
Thanks
Quoted 105 lines
drivers/net/ethernet/amd/pds_core/adminq.c +0
-0break ;case PDS_EVENT_XCVR :drivers/net/ethernet/amd/pds_core/core.c +32
-0@@ -6,6 +6,25 @@#include "core.h" +static BLOCKING_NOTIFIER_HEAD (pds_notify_chain); + +int pdsc_register_notify (struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_register_notify); + +void pdsc_unregister_notify (struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_unregister_notify); + +void pdsc_notify (unsigned long event,void *data) +{ +blocking_notifier_call_chain (&pds_notify_chain, event, data); +} +void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;@@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)static void pdsc_fw_down (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =0 , + }; +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) {dev_err (pdsc->dev,"%s: already happening\n" , __func__);return ; } +/* Notify clients of fw_down */ devlink_health_report (pdsc->fw_reporter,"FW down reported" , pdsc); +pdsc_notify (PDS_EVENT_RESET , &reset_event);pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );@@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)static void pdsc_fw_up (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =1 , + };int err;if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) {@@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; +/* Notify clients of fw_up */ pdsc->fw_recoveries++;devlink_health_reporter_state_update (pdsc->fw_reporter,DEVLINK_HEALTH_REPORTER_STATE_HEALTHY ); +pdsc_notify (PDS_EVENT_RESET , &reset_event);return ;drivers/net/ethernet/amd/pds_core/core.h +3
-0@@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb); +void pdsc_notify (unsigned long event,void *data);int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf);int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf);include/linux/pds/pds_common.h +2
-0@@ -91,5 +91,7 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb);void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */ Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 18 lines · Leon Romanovsky
Quoted 16 lines · Shannon Nelson
When the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to the clients.
The entry points
pdsc_register_notify()andpdsc_unregister_notify()are EXPORTed for other drivers that want to listen for these events.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++ drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 3 ++ include/linux/pds/pds_common.h | 2 ++ 4 files changed, 39 insertions(+)
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 30
Quoted 9 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);Aren't you "resending" standard netdev event?
It will be better to send only custom, specific topds_coreevents, while leaving general ones to netdev.
We have no netdev in pds_core, so we have to publish this to clients that might have a netdev or some other need to know.
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 36
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_RESET , comp);We can argue if clients should get this event. Once reset is detected, the
pds_coreshould close devices by deleting aux drivers.
We can get a reset signal from the device when it has done a crash recovery or when it is preparing to do an update, and this allows clients to quiesce their operations when reset.state==0 and restart when they see reset.state==1
Quoted 108 lines
Thanks
Quoted 105 lines
drivers/net/ethernet/amd/pds_core/adminq.c +0
-0break ;case PDS_EVENT_XCVR :drivers/net/ethernet/amd/pds_core/core.c +32
-0@@ -6,6 +6,25 @@#include "core.h" +static BLOCKING_NOTIFIER_HEAD (pds_notify_chain); + +int pdsc_register_notify (struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_register_notify); + +void pdsc_unregister_notify (struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_unregister_notify); + +void pdsc_notify (unsigned long event,void *data) +{ +blocking_notifier_call_chain (&pds_notify_chain, event, data); +} +void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;@@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)static void pdsc_fw_down (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =0 , + }; +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) {dev_err (pdsc->dev,"%s: already happening\n" , __func__);return ; } +/* Notify clients of fw_down */ devlink_health_report (pdsc->fw_reporter,"FW down reported" , pdsc); +pdsc_notify (PDS_EVENT_RESET , &reset_event);pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );@@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)static void pdsc_fw_up (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =1 , + };int err;if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) {@@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; +/* Notify clients of fw_up */ pdsc->fw_recoveries++;devlink_health_reporter_state_update (pdsc->fw_reporter,DEVLINK_HEALTH_REPORTER_STATE_HEALTHY ); +pdsc_notify (PDS_EVENT_RESET , &reset_event);return ;drivers/net/ethernet/amd/pds_core/core.h +3
-0@@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb); +void pdsc_notify (unsigned long event,void *data);int pdsc_auxbus_dev_add_vf (struct pdsc *vf,struct pdsc *pf);int pdsc_auxbus_dev_del_vf (struct pdsc *vf,struct pdsc *pf);include/linux/pds/pds_common.h +2
-1@@ -91,5 +91,7 @@ enum pds_core_logical_qtype {PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ }; +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb);void *pdsc_get_pf_struct (struct pci_dev *vf_pdev);#endif /* _PDS_COMMON_H_ */ --2.17.1
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 33 lines · Shannon Nelson
Quoted 18 lines · Leon Romanovsky
Quoted 16 lines · Shannon Nelson
When the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to the clients.
The entry points
pdsc_register_notify()andpdsc_unregister_notify()are EXPORTed for other drivers that want to listen for these events.Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++ drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.h | 3 ++ include/linux/pds/pds_common.h | 2 ++ 4 files changed, 39 insertions(+)Quoted 13 lines
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 30
Quoted 9 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);Aren't you "resending" standard netdev event?
It will be better to send only custom, specific topds_coreevents, while leaving general ones to netdev.
Quoted 2 lines
We have no netdev in
pds_core, so we have to publish this to clients that might have a netdev or some other need to know.
I don't know netdev well enough if it is ok or not and maybe netdev will sent this LINK_CHANGE by itself anyway.
Jakub???
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_RESET , comp);
Quoted 7 lines
Quoted 2 lines
We can argue if clients should get this event. Once reset is detected, the
pds_coreshould close devices by deleting aux drivers.We can get a reset signal from the device when it has done a crash recovery or when it is preparing to do an update, and this allows clients to quiesce their operations when reset.state==0 and restart when they see reset.state==1
I don't think that it is safe behaviour from user POV. If FW resets itself under the hood, how can client be sure that nothing changes in its operation? Once FW reset occurs, it is much safer for the clients to reconfigure everything.
Thanks
Jakub Kicinski
maintainerRe: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 14 lines · On Thu, 13 Apr 2023 11:55:01 +0300 Leon Romanovsky wrote:
Quoted 14 lines
Quoted 13 lines
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 30
Quoted 9 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);Aren't you "resending" standard netdev event?
It will be better to send only custom, specific topds_coreevents, while leaving general ones to netdev.
Quoted 7 lines
Quoted 2 lines
We have no netdev in
pds_core, so we have to publish this to clients that might have a netdev or some other need to know.I don't know netdev well enough if it is ok or not and maybe netdev will sent this
LINK_CHANGEby itself anyway.Jakub???
I actually prefer for the driver to distribute the event via its own means than some random borderline proprietary stuff outside of netdev using netdev events.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
We can argue if clients should get this event. Once reset is detected, the
pds_coreshould close devices by deleting aux drivers.We can get a reset signal from the device when it has done a crash recovery or when it is preparing to do an update, and this allows clients to quiesce their operations when reset.state==0 and restart when they see reset.state==1
Quoted 4 lines
I don't think that it is safe behaviour from user POV. If FW resets itself under the hood, how can client be sure that nothing changes in its operation? Once FW reset occurs, it is much safer for the clients to reconfigure everything.
What's the argument exactly? We do have async resets including in mlx5, grep for enable_remote_dev_reset
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 18 lines · Jakub Kicinski
Quoted 17 lines · On Thu, 13 Apr 2023 11:55:01 +0300 Leon Romanovsky wrote:
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/adminq.c +0
-0@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)Quoted 8 lines
drivers/net/ethernet/amd/pds_core/adminq.c static int pdsc_process_notifyq(struct pdsc_qcq *qcq) · line 30
Quoted 4 lines
drivers/net/ethernet/amd/pds_core/adminq.c +1
-0case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (PDS_EVENT_LINK_CHANGE , comp);Aren't you "resending" standard netdev event?
It will be better to send only custom, specific topds_coreevents, while leaving general ones to netdev.We have no netdev in
pds_core, so we have to publish this to clients that might have a netdev or some other need to know.
Quoted 8 lines
Quoted 4 lines
I don't know netdev well enough if it is ok or not and maybe netdev will sent this
LINK_CHANGEby itself anyway.Jakub???
I actually prefer for the driver to distribute the event via its own means than some random borderline proprietary stuff outside of netdev using netdev events.
ok
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
We can argue if clients should get this event. Once reset is detected, the
pds_coreshould close devices by deleting aux drivers.We can get a reset signal from the device when it has done a crash recovery or when it is preparing to do an update, and this allows clients to quiesce their operations when reset.state==0 and restart when they see reset.state==1
Quoted 7 lines
Quoted 4 lines
I don't think that it is safe behaviour from user POV. If FW resets itself under the hood, how can client be sure that nothing changes in its operation? Once FW reset occurs, it is much safer for the clients to reconfigure everything.
What's the argument exactly? We do have async resets including in mlx5, grep for
enable_remote_dev_reset
I think that it is different. I'm complaining that during FW reset, auxiliary devices are not recreated and continue to be connected to physical device with a hope that everything will continue to work from kernel and FW perspective.
It is different from enable_remote_dev_reset, where someone externally resets device which will trigger mlx5_device_rescan() routine through mlx5_unload_one->mlx5_load_one sequence.
Thanks
Jakub Kicinski
maintainerRe: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 8 lines · On Thu, 13 Apr 2023 19:44:34 +0300 Leon Romanovsky wrote:
Quoted 7 lines
Quoted 4 lines
I don't think that it is safe behaviour from user POV. If FW resets itself under the hood, how can client be sure that nothing changes in its operation? Once FW reset occurs, it is much safer for the clients to reconfigure everything.
What's the argument exactly? We do have async resets including in mlx5, grep for
enable_remote_dev_reset
Quoted 8 lines
I think that it is different. I'm complaining that during FW reset, auxiliary devices are not recreated and continue to be connected to physical device with a hope that everything will continue to work from kernel and FW perspective.
It is different from
enable_remote_dev_reset, where someone externally resets device which will triggermlx5_device_rescan()routine throughmlx5_unload_one->mlx5_load_onesequence.
Hm, my memory may be incorrect and I didn't look at the code but I thought that knob came from the "hit-less upgrade" effort. And for "hit-less upgrade" not respawning the devices was the whole point.
Which is not to disagree with you. What I'm trying to get at is that there are different types of reset which deserve different treatment.
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 18 lines · Jakub Kicinski
Quoted 8 lines · On Thu, 13 Apr 2023 19:44:34 +0300 Leon Romanovsky wrote:
Quoted 7 lines
Quoted 4 lines
I don't think that it is safe behaviour from user POV. If FW resets itself under the hood, how can client be sure that nothing changes in its operation? Once FW reset occurs, it is much safer for the clients to reconfigure everything.
What's the argument exactly? We do have async resets including in mlx5, grep for
enable_remote_dev_resetQuoted 8 lines
I think that it is different. I'm complaining that during FW reset, auxiliary devices are not recreated and continue to be connected to physical device with a hope that everything will continue to work from kernel and FW perspective.
It is different from
enable_remote_dev_reset, where someone externally resets device which will triggermlx5_device_rescan()routine throughmlx5_unload_one->mlx5_load_onesequence.
Quoted 7 lines
Hm, my memory may be incorrect and I didn't look at the code but I thought that knob came from the "hit-less upgrade" effort. And for "hit-less upgrade" not respawning the devices was the whole point.
Which is not to disagree with you. What I'm trying to get at is that there are different types of reset which deserve different treatment.
I don't disagree with you either, just have a feeling that proposed behaviour is wrong.
Thanks
Jakub Kicinski
maintainerRe: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 10 lines · On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
Quoted 7 lines
Hm, my memory may be incorrect and I didn't look at the code but I thought that knob came from the "hit-less upgrade" effort. And for "hit-less upgrade" not respawning the devices was the whole point.
Which is not to disagree with you. What I'm trying to get at is that there are different types of reset which deserve different treatment.
I don't disagree with you either, just have a feeling that proposed behaviour is wrong.
Shannon, can you elaborate on what the impact of the reset is? What loss of state and/or configuration is possible?
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
Quoted 5 lines
Hm, my memory may be incorrect and I didn't look at the code but I thought that knob came from the "hit-less upgrade" effort. And for "hit-less upgrade" not respawning the devices was the whole point.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Which is not to disagree with you. What I'm trying to get at is that there are different types of reset which deserve different treatment.
I don't disagree with you either, just have a feeling that proposed behaviour is wrong.
Shannon, can you elaborate on what the impact of the reset is? What loss of state and/or configuration is possible?
The device has a couple different cases that might generate the RESET message:
- crashed and recovered, no state saved
- FW restarted, some or all state saved
There are some variations on this, but these are the two general cases.
We can see in the existing ionic driver there already is some handling of this where the driver sees the FW come back and is able to replay the Rx filters and Rx mode configurations. If and when we are able to add an Eth client through pds_core it will want this message so that it can replay configuration in the same way. This case will also want the Link Down event so that it can do the right thing with the netdev.
For the VFio/Migration support (pds_vfio) the RESET message is essentially a no-op if nothing is happening. But if the system is in the middle of a migration it offers the ability to "cleanly" fail the migration and let the system get ready to try again.
For the vDPA case (pds_vdpa) we can trigger the config_cb callback to get the attention of the stack above us to suggest it look at current status and respond as needed, whether that is a Link Change or a Reset.
sln
Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
Quoted 23 lines · Shannon Nelson
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
Quoted 5 lines
Hm, my memory may be incorrect and I didn't look at the code but I thought that knob came from the "hit-less upgrade" effort. And for "hit-less upgrade" not respawning the devices was the whole point.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Which is not to disagree with you. What I'm trying to get at is that there are different types of reset which deserve different treatment.
I don't disagree with you either, just have a feeling that proposed behaviour is wrong.
Shannon, can you elaborate on what the impact of the reset is? What loss of state and/or configuration is possible?
The device has a couple different cases that might generate the RESET message:
- crashed and recovered, no state saved
- FW restarted, some or all state saved
There are some variations on this, but these are the two general cases.
Quoted 6 lines
We can see in the existing ionic driver there already is some handling of this where the driver sees the FW come back and is able to replay the Rx filters and Rx mode configurations. If and when we are able to add an Eth client through
pds_coreit will want this message so that it can replay configuration in the same way. This case will also want the Link Down event so that it can do the right thing with the netdev.
I don't see how you can replay ALL (ethtool, devlink, ip, e.t.c) states without net core involvement. The real fun will be with many offloaded features, where you must preserve everything while keeping upper layer in-sync with HW.
Thanks
Quoted 12 lines
For the VFio/Migration support (
pds_vfio) the RESET message is essentially a no-op if nothing is happening. But if the system is in the middle of a migration it offers the ability to "cleanly" fail the migration and let the system get ready to try again.For the vDPA case (
pds_vdpa) we can trigger theconfig_cbcallback to get the attention of the stack above us to suggest it look at current status and respond as needed, whether that is a Link Change or a Reset.sln
PATCH v9 14/14 pds_core: Kconfig and pds_core.rst
In reply to Shannon NelsonDocumentation and Kconfig hook for building the driver.
--- .../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++ MAINTAINERS | 9 +++++++++ drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++ drivers/net/ethernet/amd/Makefile | 1 + 4 files changed, 38 insertions(+)
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +16 -0
@@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status:: # devlink health diagnose pci/0000:2b:00.0 reporter fw Status: healthy State: 1 Generation: 0 Recoveries: 0 +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> AMD devices + -> AMD/Pensando Ethernet PDS_CORE Support + Support =======
MAINTAINERS +9 -0
@@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h F: include/uapi/linux/kfd_ioctl.h F: include/uapi/linux/kfd_sysfs.h +AMD PDS CORE DRIVER +M: Shannon Nelson <[email protected]> +M: Brett Creeley <[email protected]> +M: [email protected] +L: [email protected] +S: Supported +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +F: drivers/net/ethernet/amd/pds_core/ + AMD SPI DRIVER M: Sanjay R Mehta <[email protected]> S: Maintained
drivers/net/ethernet/amd/Kconfig +12 -0
@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC bool default n +config PDS_CORE + tristate "AMD/Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the AMD/Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + endif # NET_VENDOR_AMD
drivers/net/ethernet/amd/Makefile +1 -0
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o obj-$(CONFIG_SUN3LANCE) += sun3lance.o obj-$(CONFIG_SUNLANCE) += sunlance.o obj-$(CONFIG_AMD_XGBE) += xgbe/ +obj-$(CONFIG_PDS_CORE) += pds_core/
Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst
Quoted 42 lines · Shannon Nelson
Documentation and
Kconfighook for building the driver.Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++ MAINTAINERS | 9 +++++++++ drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++ drivers/net/ethernet/amd/Makefile | 1 + 4 files changed, 38 insertions(+)Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +16
-0@@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status:: # devlink health diagnose pci/0000:2b:00.0 reporter fw Status: healthy State: 1 Generation: 0 Recoveries: 0 +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> AMD devices + -> AMD/Pensando Ethernet PDS_CORE Support + Support =======MAINTAINERS +0
-0
MAINTAINERS F: drivers/gpu/drm/amd/include/vi_structs.h · line 1047
Quoted 7 lines
MAINTAINERS +4
-0@@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h F: include/uapi/linux/kfd_ioctl.h F: include/uapi/linux/kfd_sysfs.h +AMD PDS CORE DRIVER +M: Shannon Nelson <[email protected]> +M: Brett Creeley <[email protected]> +M: [email protected]
I don't know if we have any policy here, but prefer if we won't add private distribution lists to MAINTAINERS file. It is very annoying to send emails to these lists and get responses from random people.
MAINTAINERS F: drivers/gpu/drm/amd/include/vi_structs.h · line 1051
Quoted 4 lines
MAINTAINERS +4
-0+L: [email protected] +S: Supported +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +F: drivers/net/ethernet/amd/pds_core/
You forgot to add includes to this list.
Thanks
Quoted 37 lines
MAINTAINERS +1
-0+ AMD SPI DRIVER M: Sanjay R Mehta <[email protected]> S: Maintaineddrivers/net/ethernet/amd/Kconfig +12
-0@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC bool default n +config PDS_CORE + tristate "AMD/Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the AMD/Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + endif # NET_VENDOR_AMDdrivers/net/ethernet/amd/Makefile +1
-0@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o obj-$(CONFIG_SUN3LANCE) += sun3lance.o obj-$(CONFIG_SUNLANCE) += sunlance.o obj-$(CONFIG_AMD_XGBE) += xgbe/ +obj-$(CONFIG_PDS_CORE) += pds_core/Signature · 1 line
2.17.1
Re: [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst
Quoted 47 lines · Leon Romanovsky
Quoted 45 lines · Shannon Nelson
Documentation and
Kconfighook for building the driver.Signed-off-by Shannon Nelson <[email protected]>--- .../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++ MAINTAINERS | 9 +++++++++ drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++ drivers/net/ethernet/amd/Makefile | 1 + 4 files changed, 38 insertions(+)Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +16
-0@@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status:: # devlink health diagnose pci/0000:2b:00.0 reporter fw Status: healthy State: 1 Generation: 0 Recoveries: 0 +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> AMD devices + -> AMD/Pensando Ethernet PDS_CORE Support + Support =======MAINTAINERS +0
-0@@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h F: include/uapi/linux/kfd_ioctl.h F: include/uapi/linux/kfd_sysfs.h
Quoted 8 lines
MAINTAINERS F: drivers/gpu/drm/amd/include/vi_structs.h · line 1047
Quoted 4 lines
MAINTAINERS +4
-0+AMD PDS CORE DRIVER +M: Shannon Nelson <[email protected]> +M: Brett Creeley <[email protected]> +M: [email protected]I don't know if we have any policy here, but prefer if we won't add private distribution lists to
MAINTAINERSfile. It is very annoying to send emails to these lists and get responses from random people.
will fix
Quoted 7 lines
MAINTAINERS F: drivers/gpu/drm/amd/include/vi_structs.h · line 1051
Quoted 5 lines
MAINTAINERS +4
-0+L: [email protected] +S: Supported +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +F: drivers/net/ethernet/amd/pds_core/You forgot to add includes to this list.
will fix
Quoted 40 lines
Thanks
Quoted 37 lines
MAINTAINERS +1
-0+ AMD SPI DRIVER M: Sanjay R Mehta <[email protected]> S: Maintaineddrivers/net/ethernet/amd/Kconfig +12
-0@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC bool default n +config PDS_CORE + tristate "AMD/Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the AMD/Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + endif # NET_VENDOR_AMDdrivers/net/ethernet/amd/Makefile +1
-1@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o obj-$(CONFIG_SUN3LANCE) += sun3lance.o obj-$(CONFIG_SUNLANCE) += sunlance.o obj-$(CONFIG_AMD_XGBE) += xgbe/ +obj-$(CONFIG_PDS_CORE) += pds_core/ --2.17.1
Quoted 8 lines · On Thu, 6 Apr 2023 16:41:29 -0700 Shannon Nelson wrote:
This patchset implements a new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the
auxiliary_busand through a couple of EXPORTed functions for use initially in VFio and vDPA feature specific drivers.To keep this patchset to a manageable size, the
pds_vdpaandpds_vfiodrivers have been split out into their own patchsets to be reviewed separately.
Let's hear from auxdev folks, tho.
Quoted 10 lines · On Thu, 6 Apr 2023 16:41:29 -0700 Shannon Nelson <[email protected]> wrote:
Summary:
--------
This patchset implements a new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through theauxiliary_busand through a couple of EXPORTed functions for use initially in VFio and vDPA feature specific drivers.To keep this patchset to a manageable size, the
pds_vdpaandpds_vfiodrivers have been split out into their own patchsets to be reviewed separately.
FYI, this fails to build w/o DYNAMIC_DEBUG set due to implicit declaration of dynamic_hex_dump(). Seems the ionic driver avoids this by including linux/dynamic_debug.h. Other use cases have #ifdef around their uses. Thanks,
Alex
Quoted 15 lines · Alex Williamson
Quoted 10 lines · On Thu, 6 Apr 2023 16:41:29 -0700 Shannon Nelson <[email protected]> wrote:
Summary:
--------
This patchset implements a new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through theauxiliary_busand through a couple of EXPORTed functions for use initially in VFio and vDPA feature specific drivers.To keep this patchset to a manageable size, the
pds_vdpaandpds_vfiodrivers have been split out into their own patchsets to be reviewed separately.
Quoted 7 lines
FYI, this fails to build w/o
DYNAMIC_DEBUGset due to implicit declaration ofdynamic_hex_dump(). Seems the ionic driver avoids this by includinglinux/dynamic_debug.h. Other use cases have #ifdef around their uses. Thanks,Alex
Oh, good catch, thanks.
sln