pds_core driver
Subsystems (4)
v4 cover pds_core driver
Note: After comments on our previous patchset versions about the
suitability of mixing auxiliary and PCI drivers, this patchset has been re-designed to better follow the existing examples of one core PCI driver for both PF and VF use, a vDPA client that is auxiliary driver only, and a VFio client that is PCI only. See the changes notes below for more details.
Summary:
--------
This patchset implements a new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus and through a couple of EXPORTed functions for use initially in VFio and vDPA feature specific drivers.
To keep this patchset to a manageable size, the pds_vdpa and pds_vfio drivers have been split out into their own patchsets to be reviewed separately.
Detail:
-------
AMD/Pensando is making available a new set of devices for supporting vDPA, VFio, and potentially other features in the Distributed Services Card (DSC). These features are implemented through a PF that serves as a Core device for controlling and configuring its VF devices. These VF devices have separate drivers that use the auxiliary_bus to work through the Core device as the control path.
Currently, the DSC supports standard ethernet operations using the ionic driver. This is not replaced by the Core-based devices - these new devices are in addition to the existing Ethernet device. Typical DSC configurations will include both PDS devices and Ionic Eth devices. However, there is a potential future path for ethernet services to come through this device as well.
The Core device is a new PCI PF/VF device managed by a new driver 'pds_core'. The PF device has access to an admin queue for configuring the services used by the VFs, and sets up auxiliary_bus devices for each vDPA VF for communicating with the drivers for the vDPA devices. The VFs may be for VFio or vDPA, and other services in the future; these VF types are selected as part of the DSC internal FW configurations, which is out of the scope of this patchset.
When the vDPA support set is enabled in the core PF through its devlink param, auxiliary_bus devices are created for each VF that supports the feature. The vDPA driver then connects to and uses this auxiliary_device to do control path configuration through the PF device. This can then be used with the vdpa kernel module to provide devices for virtio_vdpa kernel module for host interfaces, or vhost_vdpa kernel module for interfaces exported into your favorite VM.
A cheap ASCII diagram of a vDPA instance looks something like this:
,----------.
| vdpa |
'----------'
| ||
ctl data
| ||
.----------. ||
| pds_vdpa | ||
'----------' ||
| ||
pds_core.vDPA.1 ||
| ||
.---------------. ||
| pds_core | ||
'---------------' ||
|| || ||
09:00.0 09:00.1
== PCI ============================================
|| ||
.----------. .----------.
,------| PF |---| VF |-------,
| '----------' '----------' |
| DSC |
| |
------------------------------------------Changes:
v4:
- reworked to attach to both Core PF and vDPA VF PCI devices
- now creates auxiliary_device as part of each VF PCI probe, removes them on PCI remove
- auxiliary devices now use simple unique id rather than PCI address for identifier
- replaced home-grown event publishing with kernel-based notifier service
- dropped live_migration parameter, not needed when not creating aux device for it
- replaced devm_* functions with traditional interfaces
- added MAINTAINERS entry
- removed lingering traces of set/get_vf attribute adminq commands
- trimmed some include lists
- cleaned a kernel test robot complaint about a stray unused variable
Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
v3:
https://lore.kernel.org/netdev/[email protected]/
- changed names from "pensando" to "amd" and updated copyright strings
- dropped the DEVLINK_PARAM_GENERIC_ID_FW_BANK for future development
- changed the auxiliary device creation to be triggered by the PCI bus event BOUND_DRIVER, and torn down at UNBIND_DRIVER in order to properly handle users using the sysfs bind/unbind functions
- dropped some noisy log messages
- rebased to current net-next
RFC to v2:
https://lore.kernel.org/netdev/[email protected]/
- added separate devlink param patches for DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION and DEVLINK_PARAM_GENERIC_ID_FW_BANK, and dropped the driver specific implementations
- updated descriptions for the new devlink parameters
- dropped netdev support
- dropped vDPA patches, will followup later
- separated fw update and fw bank select into their own patches
RFC:
https://lore.kernel.org/netdev/[email protected]/
Shannon Nelson (13):
pds_core: initial framework for pds_core PF driver
pds_core: add devcmd device interfaces
pds_core: health timer and workqueue
pds_core: set up device and adminq
pds_core: Add adminq processing and commands
pds_core: add FW update feature to devlink
pds_core: set up the VIF definitions and defaults
pds_core: add initial VF device handling
pds_core: add auxiliary_bus devices
pds_core: devlink params for enabling VIF support
pds_core: add the aux client API
pds_core: publish events to the clients
pds_core: Kconfig and pds_core.rst
.../device_drivers/ethernet/amd/pds_core.rst | 133 ++++ .../device_drivers/ethernet/index.rst | 1 + drivers/net/ethernet/amd/Kconfig | 12 + drivers/net/ethernet/amd/Makefile | 1 + drivers/net/ethernet/amd/pds_core/Makefile | 14 + drivers/net/ethernet/amd/pds_core/adminq.c | 283 ++++++++ drivers/net/ethernet/amd/pds_core/auxbus.c | 268 ++++++++ drivers/net/ethernet/amd/pds_core/core.c | 605 +++++++++++++++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 243 +++++++ drivers/net/ethernet/amd/pds_core/dev.c | 353 ++++++++++ drivers/net/ethernet/amd/pds_core/devlink.c | 202 ++++++ drivers/net/ethernet/amd/pds_core/fw.c | 187 ++++++ drivers/net/ethernet/amd/pds_core/main.c | 442 ++++++++++++ include/linux/pds/pds_adminq.h | 635 ++++++++++++++++++ include/linux/pds/pds_auxbus.h | 58 ++ include/linux/pds/pds_common.h | 93 +++ include/linux/pds/pds_core.h | 320 +++++++++ include/linux/pds/pds_core_if.h | 406 +++++++++++ include/linux/pds/pds_intr.h | 161 +++++ 19 files changed, 4417 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c create mode 100644 drivers/net/ethernet/amd/pds_core/core.c create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c create mode 100644 drivers/net/ethernet/amd/pds_core/main.c create mode 100644 include/linux/pds/pds_adminq.h create mode 100644 include/linux/pds/pds_auxbus.h create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core.h create mode 100644 include/linux/pds/pds_core_if.h create mode 100644 include/linux/pds/pds_intr.h
Signature · 1 line
2.17.1
RFC PATCH v4 1/13 pds_core: initial framework for pds_core PF driver
This is the initial PCI driver framework for the new pds_core device driver and its family of devices. This does the very basics of registering for the new PF PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.
--- drivers/net/ethernet/amd/pds_core/Makefile | 9 + drivers/net/ethernet/amd/pds_core/debugfs.c | 47 +++ drivers/net/ethernet/amd/pds_core/devlink.c | 51 +++ drivers/net/ethernet/amd/pds_core/main.c | 274 +++++++++++++ include/linux/pds/pds_common.h | 13 + include/linux/pds/pds_core.h | 68 ++++ include/linux/pds/pds_core_if.h | 405 ++++++++++++++++++++ 7 files changed, 867 insertions(+) create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c create mode 100644 drivers/net/ethernet/amd/pds_core/main.c create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core.h create mode 100644 include/linux/pds/pds_core_if.h
drivers/net/ethernet/amd/pds_core/Makefile new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Advanced Micro Devices, Inc. + +obj-$(CONFIG_PDS_CORE) := pds_core.o + +pds_core-y := main.o \ + devlink.o + +pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/debugfs.c new file +47 -0
47 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/devlink.c new file +51 -0
51 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c new file +274 -0
274 lines not on this page · raw message
include/linux/pds/pds_common.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright (c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core" + +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) + +#endif /* _PDS_COMMON_H_ */
include/linux/pds/pds_core.h new file +68 -0
68 lines not on this page · raw message
include/linux/pds/pds_core_if.h new file +405 -0
405 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v4 2/13 pds_core: add devcmd device interfaces
In reply to Shannon NelsonThe devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
--- drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 67 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 350 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 35 +- include/linux/pds/pds_common.h | 63 ++++ include/linux/pds/pds_core.h | 52 +++ include/linux/pds/pds_intr.h | 161 +++++++++ 8 files changed, 764 insertions(+), 4 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/core.c create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c create mode 100644 include/linux/pds/pds_intr.h
drivers/net/ethernet/amd/pds_core/Makefile +3 -1
@@ -4,6 +4,8 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ - devlink.o + devlink.o \ + dev.o \ + core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pds/pds_core.h> + +int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ; + +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +} + +void pdsc_teardown (struct pdsc *pdsc,bool removing) +{ +pdsc_devcmd_reset (pdsc); + +if (removing) { +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + } + +if (pdsc->kern_dbpage) { +iounmap (pdsc->kern_dbpage); + pdsc->kern_dbpage =NULL ; + } + +set_bit (PDSC_S_FW_DEAD , &pdsc->state); +}
drivers/net/ethernet/amd/pds_core/debugfs.c +67 -0
71 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/dev.c new file +350 -0
350 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +32 -3
@@ -151,13 +151,31 @@ static int pdsc_init_pf(struct pdsc *pdsc)if (err) goto err_out_release_regions; - err =pdsc_dl_register (pdsc); +mutex_init (&pdsc->devcmd_lock); +mutex_init (&pdsc->config_lock); + +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_FW_DEAD , &pdsc->state); + + err =pdsc_setup (pdsc,PDSC_SETUP_INIT );if (err) goto err_out_unmap_bars; + err =pdsc_dl_register (pdsc); +if (err) + goto err_out_teardown; + +mutex_unlock (&pdsc->config_lock); +return 0 ; +err_out_teardown: +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); err_out_unmap_bars: +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); +pci_free_irq_vectors (pdsc->pdev);pdsc_unmap_bars (pdsc); err_out_release_regions:pci_release_regions (pdsc->pdev);
@@ -240,8 +258,19 @@ static void pdsc_remove(struct pci_dev *pdev)*/ pdsc_dl_unregister (pdsc); -pdsc_unmap_bars (pdsc); -pci_release_regions (pdev); +if (!pdev->is_virtfn) { +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); + +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); + +pci_free_irq_vectors (pdev); +pdsc_unmap_bars (pdsc); +pci_release_regions (pdev); + }pci_clear_master (pdev);pci_disable_device (pdev);
include/linux/pds/pds_common.h +63 -0
67 lines not on this page · raw message
include/linux/pds/pds_core.h +52 -0
84 lines not on this page · raw message
include/linux/pds/pds_intr.h new file +161 -0
161 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 02/13] pds_core: add devcmd device interfaces
Quoted 87 lines · Wed, Mar 08, 2023 at 06:12:59AM CET, [email protected] wrote:
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>---
drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 67 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 350 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 35 +- include/linux/pds/pds_common.h | 63 ++++ include/linux/pds/pds_core.h | 52 +++ include/linux/pds/pds_intr.h | 161 +++++++++ 8 files changed, 764 insertions(+), 4 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/core.ccreate mode 100644drivers/net/ethernet/amd/pds_core/dev.ccreate mode 100644include/linux/pds/pds_intr.hdrivers/net/ethernet/amd/pds_core/Makefile +0
-0@@ -4,6 +4,8 @@obj-$(CONFIG_PDS_CORE) := pds_core.o
pds_core-y := main.o \
drivers/net/ethernet/amd/pds_core/Makefile +3
-1- devlink.o + devlink.o \ + dev.o \ + core.opds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.c new file +36
-0@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pds/pds_core.h> + +int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ; + +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +} + +void pdsc_teardown (struct pdsc *pdsc,bool removing) +{ +pdsc_devcmd_reset (pdsc); + +if (removing) { +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + } + +if (pdsc->kern_dbpage) { +iounmap (pdsc->kern_dbpage); + pdsc->kern_dbpage =NULL ; + } + +set_bit (PDSC_S_FW_DEAD , &pdsc->state); +}drivers/net/ethernet/amd/pds_core/debugfs.c +3
-0@@ -44,4 +44,71 @@ void pdsc_debugfs_del_dev(struct pdsc *pdsc)debugfs_remove_recursive (pdsc->dentry); pdsc->dentry =NULL ; } + +static int identity_show (struct seq_file *seq,void *v) +{
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 59
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +10
-0+struct pdsc *pdsc = seq->private; +struct pds_core_dev_identity *ident; +int vt; + + ident = &pdsc->dev_ident; + +seq_printf (seq,"asic_type: 0x%x\n" , pdsc->dev_info.asic_type); +seq_printf (seq,"asic_rev: 0x%x\n" , pdsc->dev_info.asic_rev); +seq_printf (seq,"serial_num: %s\n" , pdsc->dev_info.serial_num); +seq_printf (seq,"fw_version: %s\n" , pdsc->dev_info.fw_version);
What is the exact reason of exposing this here and not trought well defined devlink info interface?
Quoted 308 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +54
-0+seq_printf (seq,"fw_status: 0x%x\n" , +ioread8 (&pdsc->info_regs->fw_status)); +seq_printf (seq,"fw_heartbeat: 0x%x\n" , +ioread32 (&pdsc->info_regs->fw_heartbeat)); + +seq_printf (seq,"nlifs: %d\n" ,le32_to_cpu (ident->nlifs)); +seq_printf (seq,"nintrs: %d\n" ,le32_to_cpu (ident->nintrs)); +seq_printf (seq,"ndbpgs_per_lif: %d\n" ,le32_to_cpu (ident->ndbpgs_per_lif)); +seq_printf (seq,"intr_coal_mult: %d\n" ,le32_to_cpu (ident->intr_coal_mult)); +seq_printf (seq,"intr_coal_div: %d\n" ,le32_to_cpu (ident->intr_coal_div)); + +seq_puts (seq,"vif_types: " ); +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) +seq_printf (seq,"%d " ,le16_to_cpu (pdsc->dev_ident.vif_types[vt])); +seq_puts (seq,"\n" ); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (identity); + +void pdsc_debugfs_add_ident (struct pdsc *pdsc) +{ +debugfs_create_file ("identity" ,0400 , pdsc->dentry, pdsc, &identity_fops); +} + +static int irqs_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pdsc_intr_info *intr_info; +int i; + +seq_printf (seq,"index vector name (nintrs %d)\n" , pdsc->nintrs); + +if (!pdsc->intr_info) +return 0 ; + +for (i =0 ; i < pdsc->nintrs; i++) { + intr_info = &pdsc->intr_info[i]; +if (!intr_info->vector) +continue ; + +seq_printf (seq,"% 3d % 3d %s\n" , + i, intr_info->vector, intr_info->name); + } + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (irqs); + +void pdsc_debugfs_add_irqs (struct pdsc *pdsc) +{ +debugfs_create_file ("irqs" ,0400 , pdsc->dentry, pdsc, &irqs_fops); +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/dev.c new file +247
-0@@ -0,0 +1,350 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/version.h> +#include <linux/errno.h> +#include <linux/pci.h> +#include <linux/utsname.h> + +#include <linux/pds/pds_core.h> + +int pdsc_err_to_errno (enum pds_core_status_code code) +{ +switch (code) { +case PDS_RC_SUCCESS : +return 0 ; +case PDS_RC_EVERSION : +case PDS_RC_EQTYPE : +case PDS_RC_EQID : +case PDS_RC_EINVAL : +case PDS_RC_ENOSUPP : +return -EINVAL ; +case PDS_RC_EPERM : +return -EPERM ; +case PDS_RC_ENOENT : +return -ENOENT ; +case PDS_RC_EAGAIN : +return -EAGAIN ; +case PDS_RC_ENOMEM : +return -ENOMEM ; +case PDS_RC_EFAULT : +return -EFAULT ; +case PDS_RC_EBUSY : +return -EBUSY ; +case PDS_RC_EEXIST : +return -EEXIST ; +case PDS_RC_EVFID : +return -ENODEV ; +case PDS_RC_ECLIENT : +return -ECHILD ; +case PDS_RC_ENOSPC : +return -ENOSPC ; +case PDS_RC_ERANGE : +return -ERANGE ; +case PDS_RC_BAD_ADDR : +return -EFAULT ; +case PDS_RC_EOPCODE : +case PDS_RC_EINTR : +case PDS_RC_DEV_CMD : +case PDS_RC_ERROR : +case PDS_RC_ERDMA : +case PDS_RC_EIO : +default : +return -EIO ; + } +} + +bool pdsc_is_fw_running (struct pdsc *pdsc) +{ + pdsc->fw_status =ioread8 (&pdsc->info_regs->fw_status); + pdsc->last_fw_time = jiffies; + pdsc->last_hb =ioread32 (&pdsc->info_regs->fw_heartbeat); + +/* Firmware is useful only if the running bit is set and +* fw_status != 0xff (bad PCI read) +*/ +return (pdsc->fw_status !=0xff ) && + (pdsc->fw_status &PDS_CORE_FW_STS_F_RUNNING ); +} + +bool pdsc_is_fw_good (struct pdsc *pdsc) +{ +return pdsc_is_fw_running (pdsc) && + (pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ) == pdsc->fw_generation; +} + +static u8 pdsc_devcmd_status (struct pdsc *pdsc) +{ +return ioread8 (&pdsc->cmd_regs->comp.status); +} + +static bool pdsc_devcmd_done (struct pdsc *pdsc) +{ +return ioread32 (&pdsc->cmd_regs->done) &PDS_CORE_DEV_CMD_DONE ; +} + +static void pdsc_devcmd_dbell (struct pdsc *pdsc) +{ +iowrite32 (0 , &pdsc->cmd_regs->done); +iowrite32 (1 , &pdsc->cmd_regs->doorbell); +} + +static void pdsc_devcmd_clean (struct pdsc *pdsc) +{ +iowrite32 (0 , &pdsc->cmd_regs->doorbell); +memset_io (&pdsc->cmd_regs->cmd,0 ,sizeof (pdsc->cmd_regs->cmd)); +} + +static const char *pdsc_devcmd_str (int opcode) +{ +switch (opcode) { +case PDS_CORE_CMD_NOP : +return "PDS_CORE_CMD_NOP" ; +case PDS_CORE_CMD_IDENTIFY : +return "PDS_CORE_CMD_IDENTIFY" ; +case PDS_CORE_CMD_RESET : +return "PDS_CORE_CMD_RESET" ; +case PDS_CORE_CMD_INIT : +return "PDS_CORE_CMD_INIT" ; +case PDS_CORE_CMD_FW_DOWNLOAD : +return "PDS_CORE_CMD_FW_DOWNLOAD" ; +case PDS_CORE_CMD_FW_CONTROL : +return "PDS_CORE_CMD_FW_CONTROL" ; +default : +return "PDS_CORE_CMD_UNKNOWN" ; + } +} + +static int pdsc_devcmd_wait (struct pdsc *pdsc,int max_seconds) +{ +struct device *dev = pdsc->dev; +unsigned long start_time; +unsigned long max_wait; +unsigned long duration; +int timeout =0 ; +int status =0 ; +int done =0 ; +int err =0 ; +int opcode; + + opcode =ioread8 (&pdsc->cmd_regs->cmd.opcode); + + start_time = jiffies; + max_wait = start_time + (max_seconds *HZ ); + +while (!done && !timeout) { + done =pdsc_devcmd_done (pdsc); +if (done) +break ; + + timeout =time_after (jiffies, max_wait); +if (timeout) +break ; + +usleep_range (100 ,200 ); + } + duration = jiffies - start_time; + +if (done && duration >HZ ) +dev_dbg (dev,"DEVCMD %d %s after %ld secs\n" , + opcode,pdsc_devcmd_str (opcode), duration /HZ ); + +if (!done || timeout) { +dev_err (dev,"DEVCMD %d %s timeout, done %d timeout %d max_seconds=%d\n" , + opcode,pdsc_devcmd_str (opcode), done, timeout, + max_seconds); + err = -ETIMEDOUT ; +pdsc_devcmd_clean (pdsc); + } + + status =pdsc_devcmd_status (pdsc); + err =pdsc_err_to_errno (status); +if (status !=PDS_RC_SUCCESS && status !=PDS_RC_EAGAIN ) +dev_err (dev,"DEVCMD %d %s failed, status=%d err %d %pe\n" , + opcode,pdsc_devcmd_str (opcode), status, err, +ERR_PTR (err)); + +return err; +} + +int pdsc_devcmd_locked (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds) +{ +int err; + +memcpy_toio (&pdsc->cmd_regs->cmd, cmd,sizeof (*cmd)); +pdsc_devcmd_dbell (pdsc); + err =pdsc_devcmd_wait (pdsc, max_seconds); +memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); + +return err; +} + +int pdsc_devcmd (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds) +{ +int err; + +mutex_lock (&pdsc->devcmd_lock); + err =pdsc_devcmd_locked (pdsc, cmd, comp, max_seconds); +mutex_unlock (&pdsc->devcmd_lock); + +return err; +} + +int pdsc_devcmd_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .opcode =PDS_CORE_CMD_INIT , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +int pdsc_devcmd_reset (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .reset.opcode =PDS_CORE_CMD_RESET , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static int pdsc_devcmd_identify_locked (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .identify.opcode =PDS_CORE_CMD_IDENTIFY , + .identify.ver =PDS_CORE_IDENTITY_VERSION_1 , + }; + +return pdsc_devcmd_locked (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static void pdsc_init_devinfo (struct pdsc *pdsc) +{ + pdsc->dev_info.asic_type =ioread8 (&pdsc->info_regs->asic_type); + pdsc->dev_info.asic_rev =ioread8 (&pdsc->info_regs->asic_rev); + pdsc->fw_generation =PDS_CORE_FW_STS_F_GENERATION & +ioread8 (&pdsc->info_regs->fw_status); + +memcpy_fromio (pdsc->dev_info.fw_version, + pdsc->info_regs->fw_version, +PDS_CORE_DEVINFO_FWVERS_BUFLEN ); + pdsc->dev_info.fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN ] =0 ; + +memcpy_fromio (pdsc->dev_info.serial_num, + pdsc->info_regs->serial_num, +PDS_CORE_DEVINFO_SERIAL_BUFLEN ); + pdsc->dev_info.serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN ] =0 ; + +dev_dbg (pdsc->dev,"fw_version %s\n" , pdsc->dev_info.fw_version); +} + +static int pdsc_identify (struct pdsc *pdsc) +{
drivers/net/ethernet/amd/pds_core/dev.c line 257
Quoted 10 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +10
-0+struct pds_core_drv_identity drv = {0 }; +size_t sz; +int err; + + drv.drv_type =cpu_to_le32 (PDS_DRIVER_LINUX ); + drv.kernel_ver =cpu_to_le32 (LINUX_VERSION_CODE ); +snprintf (drv.kernel_ver_str,sizeof (drv.kernel_ver_str), +"%s %s" ,utsname ()->release,utsname ()->version); +snprintf (drv.driver_ver_str,sizeof (drv.driver_ver_str), +"%s %s" ,PDS_CORE_DRV_NAME ,utsname ()->release);
Why exactly are you doing this? Looks very wrong.
Quoted 486 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +93
-0+ +/* Next let's get some info about the device +* We use the devcmd_lock at this level in order to +* get safe access to the cmd_regs->data before anyone +* else can mess it up +*/ +mutex_lock (&pdsc->devcmd_lock); + + sz =min_t (size_t,sizeof (drv),sizeof (pdsc->cmd_regs->data)); +memcpy_toio (&pdsc->cmd_regs->data, &drv, sz); + + err =pdsc_devcmd_identify_locked (pdsc); +if (!err) { + sz =min_t (size_t,sizeof (pdsc->dev_ident),sizeof (pdsc->cmd_regs->data)); +memcpy_fromio (&pdsc->dev_ident, &pdsc->cmd_regs->data, sz); + } +mutex_unlock (&pdsc->devcmd_lock); + +if (err) { +dev_err (pdsc->dev,"Cannot identify device: %pe\n" ,ERR_PTR (err)); +return err; + } + +if (isprint (pdsc->dev_info.fw_version[0 ]) && +isascii (pdsc->dev_info.fw_version[0 ])) +dev_info (pdsc->dev,"FW: %.*s\n" , + (int )(sizeof (pdsc->dev_info.fw_version) -1 ), + pdsc->dev_info.fw_version); +else +dev_info (pdsc->dev,"FW: (invalid string) 0x%02x 0x%02x 0x%02x 0x%02x ...\n" , + (u8 )pdsc->dev_info.fw_version[0 ], + (u8 )pdsc->dev_info.fw_version[1 ], + (u8 )pdsc->dev_info.fw_version[2 ], + (u8 )pdsc->dev_info.fw_version[3 ]); + +return 0 ; +} + +int pdsc_dev_reinit (struct pdsc *pdsc) +{ +pdsc_init_devinfo (pdsc); + +return pdsc_identify (pdsc); +} + +int pdsc_dev_init (struct pdsc *pdsc) +{ +unsigned int nintrs; +int err; + +/* Initial init and reset of device */ +pdsc_init_devinfo (pdsc); + pdsc->devcmd_timeout =PDS_CORE_DEVCMD_TIMEOUT ; + + err =pdsc_devcmd_reset (pdsc); +if (err) +return err; + + err =pdsc_identify (pdsc); +if (err) +return err; + +pdsc_debugfs_add_ident (pdsc); + +/* Now we can reserve interrupts */ + nintrs =le32_to_cpu (pdsc->dev_ident.nintrs); + nintrs = min_t(unsigned int, num_online_cpus(), nintrs); + +/* Get intr_info struct array for tracking */ + pdsc->intr_info =kcalloc (nintrs,sizeof (*pdsc->intr_info),GFP_KERNEL ); +if (!pdsc->intr_info) { + err = -ENOSPC ; + goto err_out; + } + + err =pci_alloc_irq_vectors (pdsc->pdev, nintrs, nintrs,PCI_IRQ_MSIX ); +if (err != nintrs) { +dev_err (pdsc->dev,"Can't get %d intrs from OS: %pe\n" , + nintrs,ERR_PTR (err)); + err = -ENOSPC ; + goto err_out; + } + pdsc->nintrs = nintrs; +pdsc_debugfs_add_irqs (pdsc); + +return 0 ; + +err_out: +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + +return err; +}drivers/net/ethernet/amd/pds_core/main.c +15
-1@@ -151,13 +151,31 @@ static int pdsc_init_pf(struct pdsc *pdsc)if (err) goto err_out_release_regions; - err =pdsc_dl_register (pdsc); +mutex_init (&pdsc->devcmd_lock); +mutex_init (&pdsc->config_lock); + +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_FW_DEAD , &pdsc->state); + + err =pdsc_setup (pdsc,PDSC_SETUP_INIT );if (err) goto err_out_unmap_bars; + err =pdsc_dl_register (pdsc); +if (err) + goto err_out_teardown; + +mutex_unlock (&pdsc->config_lock); +return 0 ; +err_out_teardown: +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );
err_out_unmap_bars:drivers/net/ethernet/amd/pds_core/main.c +4
-0+mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); +pci_free_irq_vectors (pdsc->pdev);pdsc_unmap_bars(pdsc); err_out_release_regions: pci_release_regions(pdsc->pdev);drivers/net/ethernet/amd/pds_core/main.c +13
-2@@ -240,8 +258,19 @@ static void pdsc_remove(struct pci_dev *pdev)*/ pdsc_dl_unregister (pdsc); -pdsc_unmap_bars (pdsc); -pci_release_regions (pdev); +if (!pdev->is_virtfn) { +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); + +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); + +pci_free_irq_vectors (pdev); +pdsc_unmap_bars (pdsc); +pci_release_regions (pdev); + }pci_clear_master (pdev);pci_disable_device (pdev);include/linux/pds/pds_common.h +63
-0@@ -10,4 +10,67 @@#define PDS_CORE_ADDR_LEN 52#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) +/* +* enum pds_core_status_code - Device command return codes +*/ +enum pds_core_status_code { +PDS_RC_SUCCESS =0 ,/* Success */ +PDS_RC_EVERSION =1 ,/* Incorrect version for request */ +PDS_RC_EOPCODE =2 ,/* Invalid cmd opcode */ +PDS_RC_EIO =3 ,/* I/O error */ +PDS_RC_EPERM =4 ,/* Permission denied */ +PDS_RC_EQID =5 ,/* Bad qid */ +PDS_RC_EQTYPE =6 ,/* Bad qtype */ +PDS_RC_ENOENT =7 ,/* No such element */ +PDS_RC_EINTR =8 ,/* operation interrupted */ +PDS_RC_EAGAIN =9 ,/* Try again */ +PDS_RC_ENOMEM =10 ,/* Out of memory */ +PDS_RC_EFAULT =11 ,/* Bad address */ +PDS_RC_EBUSY =12 ,/* Device or resource busy */ +PDS_RC_EEXIST =13 ,/* object already exists */ +PDS_RC_EINVAL =14 ,/* Invalid argument */ +PDS_RC_ENOSPC =15 ,/* No space left or alloc failure */ +PDS_RC_ERANGE =16 ,/* Parameter out of range */ +PDS_RC_BAD_ADDR =17 ,/* Descriptor contains a bad ptr */ +PDS_RC_DEV_CMD =18 ,/* Device cmd attempted on AdminQ */ +PDS_RC_ENOSUPP =19 ,/* Operation not supported */ +PDS_RC_ERROR =29 ,/* Generic error */ +PDS_RC_ERDMA =30 ,/* Generic RDMA error */ +PDS_RC_EVFID =31 ,/* VF ID does not exist */ +PDS_RC_BAD_FW =32 ,/* FW file is invalid or corrupted */ +PDS_RC_ECLIENT =33 ,/* No such client id */ +}; + +enum pds_core_driver_type { +PDS_DRIVER_LINUX =1 , +PDS_DRIVER_WIN =2 , +PDS_DRIVER_DPDK =3 , +PDS_DRIVER_FREEBSD =4 , +PDS_DRIVER_IPXE =5 , +PDS_DRIVER_ESXI =6 , +}; + +#define PDS_CORE_IFNAMSIZ 16 + +/** +* enum pds_core_logical_qtype - Logical Queue Types +* @PDS_CORE_QTYPE_ADMINQ: Administrative Queue +* @PDS_CORE_QTYPE_NOTIFYQ: Notify Queue +* @PDS_CORE_QTYPE_RXQ: Receive Queue +* @PDS_CORE_QTYPE_TXQ: Transmit Queue +* @PDS_CORE_QTYPE_EQ: Event Queue +* @PDS_CORE_QTYPE_MAX: Max queue type supported +*/ +enum pds_core_logical_qtype { +PDS_CORE_QTYPE_ADMINQ =0 , +PDS_CORE_QTYPE_NOTIFYQ =1 , +PDS_CORE_QTYPE_RXQ =2 , +PDS_CORE_QTYPE_TXQ =3 , +PDS_CORE_QTYPE_EQ =4 , + +PDS_CORE_QTYPE_MAX =16 /* don't change - used in struct size */ +}; + +typedef void (*pds_core_cb )(void *cb_arg); +#endif /* _PDS_COMMON_H_ */ include/linux/pds/pds_core.h +52
-0@@ -9,8 +9,13 @@#include <linux/pds/pds_common.h> #include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_intr.h> #define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver" +#define PDSC_TEARDOWN_RECOVERY false +#define PDSC_TEARDOWN_REMOVING true +#define PDSC_SETUP_RECOVERY false +#define PDSC_SETUP_INIT truestruct pdsc_dev_bar {void __iomem *vaddr;@@ -19,6 +24,22 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc_devinfo { +u8 asic_type; +u8 asic_rev; +char fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN +1 ]; +char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN +1 ]; +}; + +#define PDSC_INTR_NAME_MAX_SZ 32 + +struct pdsc_intr_info { +char name[PDSC_INTR_NAME_MAX_SZ ]; +unsigned int index; +unsigned int vector; +void *data; +}; +/* No state flags set means we are in a steady running state */ enum pdsc_state_flags {PDSC_S_FW_DEAD ,/* fw stopped, waiting for startup or recovery */ @@ -38,7 +59,19 @@ struct pdsc {int id;unsigned long state; +u8 fw_status; +u8 fw_generation; +unsigned long last_fw_time; +u32 last_hb; + +struct pdsc_devinfo dev_info; +struct pds_core_dev_identity dev_ident; +unsigned int nintrs; +struct pdsc_intr_info *intr_info;/* array of nintrs elements */ +unsigned int devcmd_timeout; +struct mutex devcmd_lock;/* lock for dev_cmd operations */ +struct mutex config_lock;/* lock for configuration operations */ struct pds_core_dev_info_regs __iomem *info_regs;struct pds_core_dev_cmd_regs __iomem *cmd_regs;struct pds_core_intr __iomem *intr_ctrl;@@ -58,11 +91,30 @@ void pdsc_debugfs_create(void);void pdsc_debugfs_destroy (void );void pdsc_debugfs_add_dev (struct pdsc *pdsc);void pdsc_debugfs_del_dev (struct pdsc *pdsc); +void pdsc_debugfs_add_ident (struct pdsc *pdsc); +void pdsc_debugfs_add_irqs (struct pdsc *pdsc);#else static inline void pdsc_debugfs_create (void ) { }static inline void pdsc_debugfs_destroy (void ) { }static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_add_ident (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_add_irqs (struct pdsc *pdsc) { }#endif +int pdsc_err_to_errno (enum pds_core_status_code code); +bool pdsc_is_fw_running (struct pdsc *pdsc); +bool pdsc_is_fw_good (struct pdsc *pdsc); +int pdsc_devcmd (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds); +int pdsc_devcmd_locked (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds); +int pdsc_devcmd_init (struct pdsc *pdsc); +int pdsc_devcmd_reset (struct pdsc *pdsc); +int pdsc_dev_reinit (struct pdsc *pdsc); +int pdsc_dev_init (struct pdsc *pdsc); + +int pdsc_setup (struct pdsc *pdsc,bool init); +void pdsc_teardown (struct pdsc *pdsc,bool removing); +#endif /* _PDSC_H_ */ include/linux/pds/pds_intr.h new file +161
-0@@ -0,0 +1,161 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright (c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_INTR_H_ +#define _PDS_INTR_H_ + +/* +* Interrupt control register +* @coal_init: Coalescing timer initial value, in +* device units. Use @identity->intr_coal_mult +* and @identity->intr_coal_div to convert from +* usecs to device units: +* +* coal_init = coal_usecs * coal_mutl / coal_div +* +* When an interrupt is sent the interrupt +* coalescing timer current value +* (@coalescing_curr) is initialized with this +* value and begins counting down. No more +* interrupts are sent until the coalescing +* timer reaches 0. When @coalescing_init=0 +* interrupt coalescing is effectively disabled +* and every interrupt assert results in an +* interrupt. Reset value: 0 +* @mask: Interrupt mask. When @mask=1 the interrupt +* resource will not send an interrupt. When +* @mask=0 the interrupt resource will send an +* interrupt if an interrupt event is pending +* or on the next interrupt assertion event. +* Reset value: 1 +* @credits: Interrupt credits. This register indicates +* how many interrupt events the hardware has +* sent. When written by software this +* register atomically decrements @int_credits +* by the value written. When @int_credits +* becomes 0 then the "pending interrupt" bit +* in the Interrupt Status register is cleared +* by the hardware and any pending but unsent +* interrupts are cleared. +* !!!IMPORTANT!!! This is a signed register. +* @flags: Interrupt control flags +* @unmask -- When this bit is written with a 1 +* the interrupt resource will set mask=0. +* @coal_timer_reset -- When this +* bit is written with a 1 the +* @coalescing_curr will be reloaded with +* @coalescing_init to reset the coalescing +* timer. +* @mask_on_assert: Automatically mask on assertion. When +* @mask_on_assert=1 the interrupt resource +* will set @mask=1 whenever an interrupt is +* sent. When using interrupts in Legacy +* Interrupt mode the driver must select +* @mask_on_assert=0 for proper interrupt +* operation. +* @coalescing_curr: Coalescing timer current value, in +* microseconds. When this value reaches 0 +* the interrupt resource is again eligible to +* send an interrupt. If an interrupt event +* is already pending when @coalescing_curr +* reaches 0 the pending interrupt will be +* sent, otherwise an interrupt will be sent +* on the next interrupt assertion event. +*/ +struct pds_core_intr { +u32 coal_init; +u32 mask; +u16 credits; +u16 flags; +#define PDS_CORE_INTR_F_UNMASK 0x0001 +#define PDS_CORE_INTR_F_TIMER_RESET 0x0002 +u32 mask_on_assert; +u32 coalescing_curr; +u32 rsvd6[3 ]; +}; + +#ifndef __CHECKER__ +static_assert (sizeof (struct pds_core_intr ) ==32 ); +#endif /* __CHECKER__ */ + +#define PDS_CORE_INTR_CTRL_REGS_MAX 2048 +#define PDS_CORE_INTR_CTRL_COAL_MAX 0x3F +#define PDS_CORE_INTR_INDEX_NOT_ASSIGNED -1 + +struct pds_core_intr_status { +u32 status[2 ]; +}; + +/** +* enum pds_core_intr_mask_vals - valid values for mask and mask_assert. +* @PDS_CORE_INTR_MASK_CLEAR: unmask interrupt. +* @PDS_CORE_INTR_MASK_SET: mask interrupt. +*/ +enum pds_core_intr_mask_vals { +PDS_CORE_INTR_MASK_CLEAR =0 , +PDS_CORE_INTR_MASK_SET =1 , +}; + +/** +* enum pds_core_intr_credits_bits - Bitwise composition of credits values. +* @PDS_CORE_INTR_CRED_COUNT: bit mask of credit count, no shift needed. +* @PDS_CORE_INTR_CRED_COUNT_SIGNED: bit mask of credit count, including sign bit. +* @PDS_CORE_INTR_CRED_UNMASK: unmask the interrupt. +* @PDS_CORE_INTR_CRED_RESET_COALESCE: reset the coalesce timer. +* @PDS_CORE_INTR_CRED_REARM: unmask the and reset the timer. +*/ +enum pds_core_intr_credits_bits { +PDS_CORE_INTR_CRED_COUNT =0x7fffu , +PDS_CORE_INTR_CRED_COUNT_SIGNED =0xffffu , +PDS_CORE_INTR_CRED_UNMASK =0x10000u , +PDS_CORE_INTR_CRED_RESET_COALESCE =0x20000u , +PDS_CORE_INTR_CRED_REARM = (PDS_CORE_INTR_CRED_UNMASK | +PDS_CORE_INTR_CRED_RESET_COALESCE ), +}; + +static inline void pds_core_intr_coal_init (struct pds_core_intr __iomem *intr_ctrl, +u32 coal) +{ +iowrite32 (coal, &intr_ctrl->coal_init); +} + +static inline void pds_core_intr_mask (struct pds_core_intr __iomem *intr_ctrl, +u32 mask) +{ +iowrite32 (mask, &intr_ctrl->mask); +} + +static inline void pds_core_intr_credits (struct pds_core_intr __iomem *intr_ctrl, +u32 cred,u32 flags) +{ +if (WARN_ON_ONCE (cred >PDS_CORE_INTR_CRED_COUNT )) { + cred =ioread32 (&intr_ctrl->credits); + cred &=PDS_CORE_INTR_CRED_COUNT_SIGNED ; + } + +iowrite32 (cred | flags, &intr_ctrl->credits); +} + +static inline void pds_core_intr_clean_flags (struct pds_core_intr __iomem *intr_ctrl, +u32 flags) +{ +u32 cred; + + cred =ioread32 (&intr_ctrl->credits); + cred &=PDS_CORE_INTR_CRED_COUNT_SIGNED ; + cred |= flags; +iowrite32 (cred, &intr_ctrl->credits); +} + +static inline void pds_core_intr_clean (struct pds_core_intr __iomem *intr_ctrl) +{ +pds_core_intr_clean_flags (intr_ctrl,PDS_CORE_INTR_CRED_RESET_COALESCE ); +} + +static inline void pds_core_intr_mask_assert (struct pds_core_intr __iomem *intr_ctrl, +u32 mask) +{ +iowrite32 (mask, &intr_ctrl->mask_on_assert); +} + +#endif /* _PDS_INTR_H_ */ Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 02/13] pds_core: add devcmd device interfaces
Quoted 93 lines · Jiri Pirko
Quoted 92 lines · Wed, Mar 08, 2023 at 06:12:59AM CET, [email protected] wrote:
The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
Signed-off-by Shannon Nelson <[email protected]>---
drivers/net/ethernet/amd/pds_core/Makefile | 4 +- drivers/net/ethernet/amd/pds_core/core.c | 36 ++ drivers/net/ethernet/amd/pds_core/debugfs.c | 67 ++++ drivers/net/ethernet/amd/pds_core/dev.c | 350 ++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 35 +- include/linux/pds/pds_common.h | 63 ++++ include/linux/pds/pds_core.h | 52 +++ include/linux/pds/pds_intr.h | 161 +++++++++ 8 files changed, 764 insertions(+), 4 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/core.ccreate mode 100644drivers/net/ethernet/amd/pds_core/dev.ccreate mode 100644include/linux/pds/pds_intr.hdrivers/net/ethernet/amd/pds_core/Makefile +0
-0@@ -4,6 +4,8 @@obj-$(CONFIG_PDS_CORE) := pds_core.o
pds_core-y := main.o \
drivers/net/ethernet/amd/pds_core/Makefile +3
-1- devlink.o + devlink.o \ + dev.o \ + core.opds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.c new file +36
-0@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pds/pds_core.h> + +int pdsc_setup (struct pdsc *pdsc,bool init) +{ +int err =0 ; + +if (init) + err =pdsc_dev_init (pdsc); +else + err =pdsc_dev_reinit (pdsc); +if (err) +return err; + +clear_bit (PDSC_S_FW_DEAD , &pdsc->state); +return 0 ; +} + +void pdsc_teardown (struct pdsc *pdsc,bool removing) +{ +pdsc_devcmd_reset (pdsc); + +if (removing) { +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + } + +if (pdsc->kern_dbpage) { +iounmap (pdsc->kern_dbpage); + pdsc->kern_dbpage =NULL ; + } + +set_bit (PDSC_S_FW_DEAD , &pdsc->state); +}drivers/net/ethernet/amd/pds_core/debugfs.c +8
-0@@ -44,4 +44,71 @@ void pdsc_debugfs_del_dev(struct pdsc *pdsc)debugfs_remove_recursive (pdsc->dentry); pdsc->dentry =NULL ; } + +static int identity_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pds_core_dev_identity *ident; +int vt; + + ident = &pdsc->dev_ident;
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 59
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +5
-0+ +seq_printf (seq,"asic_type: 0x%x\n" , pdsc->dev_info.asic_type); +seq_printf (seq,"asic_rev: 0x%x\n" , pdsc->dev_info.asic_rev); +seq_printf (seq,"serial_num: %s\n" , pdsc->dev_info.serial_num); +seq_printf (seq,"fw_version: %s\n" , pdsc->dev_info.fw_version);What is the exact reason of exposing this here and not trought well defined devlink info interface?
These do show up in devlink dev info eventually, but that isn't for another couple of patches. This gives us info here for debugging the earlier patches if needed.
Quoted 314 lines
Quoted 314 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +54
-0+seq_printf (seq,"fw_status: 0x%x\n" , +ioread8 (&pdsc->info_regs->fw_status)); +seq_printf (seq,"fw_heartbeat: 0x%x\n" , +ioread32 (&pdsc->info_regs->fw_heartbeat)); + +seq_printf (seq,"nlifs: %d\n" ,le32_to_cpu (ident->nlifs)); +seq_printf (seq,"nintrs: %d\n" ,le32_to_cpu (ident->nintrs)); +seq_printf (seq,"ndbpgs_per_lif: %d\n" ,le32_to_cpu (ident->ndbpgs_per_lif)); +seq_printf (seq,"intr_coal_mult: %d\n" ,le32_to_cpu (ident->intr_coal_mult)); +seq_printf (seq,"intr_coal_div: %d\n" ,le32_to_cpu (ident->intr_coal_div)); + +seq_puts (seq,"vif_types: " ); +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) +seq_printf (seq,"%d " ,le16_to_cpu (pdsc->dev_ident.vif_types[vt])); +seq_puts (seq,"\n" ); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (identity); + +void pdsc_debugfs_add_ident (struct pdsc *pdsc) +{ +debugfs_create_file ("identity" ,0400 , pdsc->dentry, pdsc, &identity_fops); +} + +static int irqs_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pdsc_intr_info *intr_info; +int i; + +seq_printf (seq,"index vector name (nintrs %d)\n" , pdsc->nintrs); + +if (!pdsc->intr_info) +return 0 ; + +for (i =0 ; i < pdsc->nintrs; i++) { + intr_info = &pdsc->intr_info[i]; +if (!intr_info->vector) +continue ; + +seq_printf (seq,"% 3d % 3d %s\n" , + i, intr_info->vector, intr_info->name); + } + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (irqs); + +void pdsc_debugfs_add_irqs (struct pdsc *pdsc) +{ +debugfs_create_file ("irqs" ,0400 , pdsc->dentry, pdsc, &irqs_fops); +} +#endif /* CONFIG_DEBUG_FS */ drivers/net/ethernet/amd/pds_core/dev.c new file +251
-0@@ -0,0 +1,350 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/version.h> +#include <linux/errno.h> +#include <linux/pci.h> +#include <linux/utsname.h> + +#include <linux/pds/pds_core.h> + +int pdsc_err_to_errno (enum pds_core_status_code code) +{ +switch (code) { +case PDS_RC_SUCCESS : +return 0 ; +case PDS_RC_EVERSION : +case PDS_RC_EQTYPE : +case PDS_RC_EQID : +case PDS_RC_EINVAL : +case PDS_RC_ENOSUPP : +return -EINVAL ; +case PDS_RC_EPERM : +return -EPERM ; +case PDS_RC_ENOENT : +return -ENOENT ; +case PDS_RC_EAGAIN : +return -EAGAIN ; +case PDS_RC_ENOMEM : +return -ENOMEM ; +case PDS_RC_EFAULT : +return -EFAULT ; +case PDS_RC_EBUSY : +return -EBUSY ; +case PDS_RC_EEXIST : +return -EEXIST ; +case PDS_RC_EVFID : +return -ENODEV ; +case PDS_RC_ECLIENT : +return -ECHILD ; +case PDS_RC_ENOSPC : +return -ENOSPC ; +case PDS_RC_ERANGE : +return -ERANGE ; +case PDS_RC_BAD_ADDR : +return -EFAULT ; +case PDS_RC_EOPCODE : +case PDS_RC_EINTR : +case PDS_RC_DEV_CMD : +case PDS_RC_ERROR : +case PDS_RC_ERDMA : +case PDS_RC_EIO : +default : +return -EIO ; + } +} + +bool pdsc_is_fw_running (struct pdsc *pdsc) +{ + pdsc->fw_status =ioread8 (&pdsc->info_regs->fw_status); + pdsc->last_fw_time = jiffies; + pdsc->last_hb =ioread32 (&pdsc->info_regs->fw_heartbeat); + +/* Firmware is useful only if the running bit is set and +* fw_status != 0xff (bad PCI read) +*/ +return (pdsc->fw_status !=0xff ) && + (pdsc->fw_status &PDS_CORE_FW_STS_F_RUNNING ); +} + +bool pdsc_is_fw_good (struct pdsc *pdsc) +{ +return pdsc_is_fw_running (pdsc) && + (pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ) == pdsc->fw_generation; +} + +static u8 pdsc_devcmd_status (struct pdsc *pdsc) +{ +return ioread8 (&pdsc->cmd_regs->comp.status); +} + +static bool pdsc_devcmd_done (struct pdsc *pdsc) +{ +return ioread32 (&pdsc->cmd_regs->done) &PDS_CORE_DEV_CMD_DONE ; +} + +static void pdsc_devcmd_dbell (struct pdsc *pdsc) +{ +iowrite32 (0 , &pdsc->cmd_regs->done); +iowrite32 (1 , &pdsc->cmd_regs->doorbell); +} + +static void pdsc_devcmd_clean (struct pdsc *pdsc) +{ +iowrite32 (0 , &pdsc->cmd_regs->doorbell); +memset_io (&pdsc->cmd_regs->cmd,0 ,sizeof (pdsc->cmd_regs->cmd)); +} + +static const char *pdsc_devcmd_str (int opcode) +{ +switch (opcode) { +case PDS_CORE_CMD_NOP : +return "PDS_CORE_CMD_NOP" ; +case PDS_CORE_CMD_IDENTIFY : +return "PDS_CORE_CMD_IDENTIFY" ; +case PDS_CORE_CMD_RESET : +return "PDS_CORE_CMD_RESET" ; +case PDS_CORE_CMD_INIT : +return "PDS_CORE_CMD_INIT" ; +case PDS_CORE_CMD_FW_DOWNLOAD : +return "PDS_CORE_CMD_FW_DOWNLOAD" ; +case PDS_CORE_CMD_FW_CONTROL : +return "PDS_CORE_CMD_FW_CONTROL" ; +default : +return "PDS_CORE_CMD_UNKNOWN" ; + } +} + +static int pdsc_devcmd_wait (struct pdsc *pdsc,int max_seconds) +{ +struct device *dev = pdsc->dev; +unsigned long start_time; +unsigned long max_wait; +unsigned long duration; +int timeout =0 ; +int status =0 ; +int done =0 ; +int err =0 ; +int opcode; + + opcode =ioread8 (&pdsc->cmd_regs->cmd.opcode); + + start_time = jiffies; + max_wait = start_time + (max_seconds *HZ ); + +while (!done && !timeout) { + done =pdsc_devcmd_done (pdsc); +if (done) +break ; + + timeout =time_after (jiffies, max_wait); +if (timeout) +break ; + +usleep_range (100 ,200 ); + } + duration = jiffies - start_time; + +if (done && duration >HZ ) +dev_dbg (dev,"DEVCMD %d %s after %ld secs\n" , + opcode,pdsc_devcmd_str (opcode), duration /HZ ); + +if (!done || timeout) { +dev_err (dev,"DEVCMD %d %s timeout, done %d timeout %d max_seconds=%d\n" , + opcode,pdsc_devcmd_str (opcode), done, timeout, + max_seconds); + err = -ETIMEDOUT ; +pdsc_devcmd_clean (pdsc); + } + + status =pdsc_devcmd_status (pdsc); + err =pdsc_err_to_errno (status); +if (status !=PDS_RC_SUCCESS && status !=PDS_RC_EAGAIN ) +dev_err (dev,"DEVCMD %d %s failed, status=%d err %d %pe\n" , + opcode,pdsc_devcmd_str (opcode), status, err, +ERR_PTR (err)); + +return err; +} + +int pdsc_devcmd_locked (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds) +{ +int err; + +memcpy_toio (&pdsc->cmd_regs->cmd, cmd,sizeof (*cmd)); +pdsc_devcmd_dbell (pdsc); + err =pdsc_devcmd_wait (pdsc, max_seconds); +memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); + +return err; +} + +int pdsc_devcmd (struct pdsc *pdsc,union pds_core_dev_cmd *cmd, +union pds_core_dev_comp *comp,int max_seconds) +{ +int err; + +mutex_lock (&pdsc->devcmd_lock); + err =pdsc_devcmd_locked (pdsc, cmd, comp, max_seconds); +mutex_unlock (&pdsc->devcmd_lock); + +return err; +} + +int pdsc_devcmd_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .opcode =PDS_CORE_CMD_INIT , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +int pdsc_devcmd_reset (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .reset.opcode =PDS_CORE_CMD_RESET , + }; + +return pdsc_devcmd (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static int pdsc_devcmd_identify_locked (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .identify.opcode =PDS_CORE_CMD_IDENTIFY , + .identify.ver =PDS_CORE_IDENTITY_VERSION_1 , + }; + +return pdsc_devcmd_locked (pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static void pdsc_init_devinfo (struct pdsc *pdsc) +{ + pdsc->dev_info.asic_type =ioread8 (&pdsc->info_regs->asic_type); + pdsc->dev_info.asic_rev =ioread8 (&pdsc->info_regs->asic_rev); + pdsc->fw_generation =PDS_CORE_FW_STS_F_GENERATION & +ioread8 (&pdsc->info_regs->fw_status); + +memcpy_fromio (pdsc->dev_info.fw_version, + pdsc->info_regs->fw_version, +PDS_CORE_DEVINFO_FWVERS_BUFLEN ); + pdsc->dev_info.fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN ] =0 ; + +memcpy_fromio (pdsc->dev_info.serial_num, + pdsc->info_regs->serial_num, +PDS_CORE_DEVINFO_SERIAL_BUFLEN ); + pdsc->dev_info.serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN ] =0 ; + +dev_dbg (pdsc->dev,"fw_version %s\n" , pdsc->dev_info.fw_version); +} + +static int pdsc_identify (struct pdsc *pdsc) +{ +struct pds_core_drv_identity drv = {0 }; +size_t sz; +int err; +
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c line 257
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+ drv.drv_type =cpu_to_le32 (PDS_DRIVER_LINUX ); + drv.kernel_ver =cpu_to_le32 (LINUX_VERSION_CODE ); +snprintf (drv.kernel_ver_str,sizeof (drv.kernel_ver_str), +"%s %s" ,utsname ()->release,utsname ()->version); +snprintf (drv.driver_ver_str,sizeof (drv.driver_ver_str), +"%s %s" ,PDS_CORE_DRV_NAME ,utsname ()->release);Why exactly are you doing this? Looks very wrong.
This helps us when debugging on the DSC side - we can see which host driver is using the interface (PXE, Linux, etc). This is modeled from what we have in our out-of-tree ionic driver interface, but I need to trim it down to be less snoopy.
sln
Re: [PATCH RFC v4 net-next 02/13] pds_core: add devcmd device interfaces
Quoted 2 lines · Thu, Mar 09, 2023 at 03:05:13AM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Wed, Mar 08, 2023 at 06:12:59AM CET, [email protected] wrote:
[..]
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +7
-0+static int identity_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pds_core_dev_identity *ident; +int vt; + + ident = &pdsc->dev_ident;Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 59
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +5
-0+ +seq_printf (seq,"asic_type: 0x%x\n" , pdsc->dev_info.asic_type); +seq_printf (seq,"asic_rev: 0x%x\n" , pdsc->dev_info.asic_rev); +seq_printf (seq,"serial_num: %s\n" , pdsc->dev_info.serial_num); +seq_printf (seq,"fw_version: %s\n" , pdsc->dev_info.fw_version);What is the exact reason of exposing this here and not trought well defined devlink info interface?
Quoted 3 lines
These do show up in devlink dev info eventually, but that isn't for another couple of patches. This gives us info here for debugging the earlier patches if needed.
Implement it properly from the start and avoid these, please.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +5
-0+seq_printf (seq,"fw_status: 0x%x\n" , +ioread8 (&pdsc->info_regs->fw_status)); +seq_printf (seq,"fw_heartbeat: 0x%x\n" , +ioread32 (&pdsc->info_regs->fw_heartbeat)); +
[..]
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+static int pdsc_identify (struct pdsc *pdsc) +{ +struct pds_core_drv_identity drv = {0 }; +size_t sz; +int err; +Quoted 8 lines
drivers/net/ethernet/amd/pds_core/dev.c line 257
Quoted 6 lines
drivers/net/ethernet/amd/pds_core/dev.c new file +6
-0+ drv.drv_type =cpu_to_le32 (PDS_DRIVER_LINUX ); + drv.kernel_ver =cpu_to_le32 (LINUX_VERSION_CODE ); +snprintf (drv.kernel_ver_str,sizeof (drv.kernel_ver_str), +"%s %s" ,utsname ()->release,utsname ()->version); +snprintf (drv.driver_ver_str,sizeof (drv.driver_ver_str), +"%s %s" ,PDS_CORE_DRV_NAME ,utsname ()->release);Why exactly are you doing this? Looks very wrong.
Quoted 4 lines
This helps us when debugging on the DSC side - we can see which host driver is using the interface (PXE, Linux, etc). This is modeled from what we have in our out-of-tree ionic driver interface, but I need to trim it down to be less snoopy.
Device should not care who is on the other side in my opinion. Jakub, any thoughts?
Quoted 2 lines
sln
Re: [PATCH RFC v4 net-next 02/13] pds_core: add devcmd device interfaces
Quoted 23 lines · Jiri Pirko
Quoted 2 lines · Thu, Mar 09, 2023 at 03:05:13AM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Wed, Mar 08, 2023 at 06:12:59AM CET, [email protected] wrote:
[..]
Quoted 16 lines
Quoted 15 lines
drivers/net/ethernet/amd/pds_core/debugfs.c void pdsc_debugfs_del_dev(struct pdsc *pdsc) · line 59
Quoted 12 lines
drivers/net/ethernet/amd/pds_core/debugfs.c +12
-0+static int identity_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +struct pds_core_dev_identity *ident; +int vt; + + ident = &pdsc->dev_ident; + +seq_printf (seq,"asic_type: 0x%x\n" , pdsc->dev_info.asic_type); +seq_printf (seq,"asic_rev: 0x%x\n" , pdsc->dev_info.asic_rev); +seq_printf (seq,"serial_num: %s\n" , pdsc->dev_info.serial_num); +seq_printf (seq,"fw_version: %s\n" , pdsc->dev_info.fw_version);What is the exact reason of exposing this here and not trought well defined devlink info interface?
Quoted 5 lines
Quoted 3 lines
These do show up in devlink dev info eventually, but that isn't for another couple of patches. This gives us info here for debugging the earlier patches if needed.
Implement it properly from the start and avoid these, please.
Sure, I'll drop these in the next rev.
sln
RFC PATCH v4 3/13 pds_core: health timer and workqueue
In reply to Shannon NelsonAdd in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
--- drivers/net/ethernet/amd/pds_core/core.c | 60 ++++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/dev.c | 3 ++ drivers/net/ethernet/amd/pds_core/main.c | 51 ++++++++++++++++++++ include/linux/pds/pds_core.h | 10 ++++ 4 files changed, 124 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +60 -0
@@ -34,3 +34,63 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } + +static void pdsc_fw_down (struct pdsc *pdsc) +{ +mutex_lock (&pdsc->config_lock); + +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: already happening\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); + +mutex_unlock (&pdsc->config_lock); +} + +static void pdsc_fw_up (struct pdsc *pdsc) +{ +int err; + +mutex_lock (&pdsc->config_lock); + +if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: fw not dead\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + + err =pdsc_setup (pdsc,PDSC_SETUP_RECOVERY ); +if (err) + goto err_out; + +mutex_unlock (&pdsc->config_lock); + +return ; + +err_out: +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); +mutex_unlock (&pdsc->config_lock); +} + +void pdsc_health_thread (struct work_struct *work) +{ + struct pdsc *pdsc = container_of(work, struct pdsc, health_work); +bool healthy; + + healthy =pdsc_is_fw_good (pdsc); +dev_dbg (pdsc->dev,"%s: health %d fw_status %#02x fw_heartbeat %d\n" , + __func__, healthy, pdsc->fw_status, pdsc->last_hb); + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +if (healthy) +pdsc_fw_up (pdsc); + }else { +if (!healthy) +pdsc_fw_down (pdsc); + } + + pdsc->fw_generation = pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ; +}
drivers/net/ethernet/amd/pds_core/dev.c +3 -0
@@ -177,6 +177,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, err =pdsc_devcmd_wait (pdsc, max_seconds);memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); +if (err == -ENXIO || err == -ETIMEDOUT ) +pdsc_queue_health_check (pdsc); +return err; }
drivers/net/ethernet/amd/pds_core/main.c +51 -0
88 lines not on this page · raw message
include/linux/pds/pds_core.h +10 -0
@@ -12,6 +12,8 @@#include <linux/pds/pds_intr.h> #define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver" + +#define PDSC_WATCHDOG_SECS 5#define PDSC_TEARDOWN_RECOVERY false#define PDSC_TEARDOWN_REMOVING true#define PDSC_SETUP_RECOVERY false
@@ -63,12 +65,17 @@ struct pdsc {u8 fw_generation;unsigned long last_fw_time;u32 last_hb; +struct timer_list wdtimer; +unsigned int wdtimer_period; +struct work_struct health_work;struct pdsc_devinfo dev_info;struct pds_core_dev_identity dev_ident;unsigned int nintrs;struct pdsc_intr_info *intr_info;/* array of nintrs elements */ +struct workqueue_struct *wq; +unsigned int devcmd_timeout;struct mutex devcmd_lock;/* lock for dev_cmd operations */ struct mutex config_lock;/* lock for configuration operations */
@@ -81,6 +88,8 @@ struct pdsc {u64 __iomem *kern_dbpage; }; +void pdsc_queue_health_check (struct pdsc *pdsc); +struct pdsc *pdsc_dl_alloc (struct device *dev,bool is_pf);void pdsc_dl_free (struct pdsc *pdsc);int pdsc_dl_register (struct pdsc *pdsc);
@@ -116,5 +125,6 @@ int pdsc_dev_init(struct pdsc *pdsc);int pdsc_setup (struct pdsc *pdsc,bool init);void pdsc_teardown (struct pdsc *pdsc,bool removing); +void pdsc_health_thread (struct work_struct *work);#endif /* _PDSC_H_ */
Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 03/13] pds_core: health timer and workqueue
Quoted 2 lines · Wed, Mar 08, 2023 at 06:13:00AM CET, [email protected] wrote:
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
Why don't you use devlink health to let the user know that something odd happened with HW?
Quoted 238 lines
Signed-off-by Shannon Nelson <[email protected]>---
drivers/net/ethernet/amd/pds_core/core.c | 60 ++++++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/dev.c | 3 ++ drivers/net/ethernet/amd/pds_core/main.c | 51 ++++++++++++++++++++ include/linux/pds/pds_core.h | 10 ++++ 4 files changed, 124 insertions(+)drivers/net/ethernet/amd/pds_core/core.c +60
-0@@ -34,3 +34,63 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } + +static void pdsc_fw_down (struct pdsc *pdsc) +{ +mutex_lock (&pdsc->config_lock); + +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: already happening\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); + +mutex_unlock (&pdsc->config_lock); +} + +static void pdsc_fw_up (struct pdsc *pdsc) +{ +int err; + +mutex_lock (&pdsc->config_lock); + +if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: fw not dead\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + + err =pdsc_setup (pdsc,PDSC_SETUP_RECOVERY ); +if (err) + goto err_out; + +mutex_unlock (&pdsc->config_lock); + +return ; + +err_out: +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); +mutex_unlock (&pdsc->config_lock); +} + +void pdsc_health_thread (struct work_struct *work) +{ + struct pdsc *pdsc = container_of(work, struct pdsc, health_work); +bool healthy; + + healthy =pdsc_is_fw_good (pdsc); +dev_dbg (pdsc->dev,"%s: health %d fw_status %#02x fw_heartbeat %d\n" , + __func__, healthy, pdsc->fw_status, pdsc->last_hb); + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +if (healthy) +pdsc_fw_up (pdsc); + }else { +if (!healthy) +pdsc_fw_down (pdsc); + } + + pdsc->fw_generation = pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ; +}drivers/net/ethernet/amd/pds_core/dev.c +3
-0@@ -177,6 +177,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, err =pdsc_devcmd_wait (pdsc, max_seconds);memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); +if (err == -ENXIO || err == -ETIMEDOUT ) +pdsc_queue_health_check (pdsc); +return err; }drivers/net/ethernet/amd/pds_core/main.c +38
-0@@ -25,6 +25,31 @@ static const struct pci_device_id pdsc_id_table[] = { };MODULE_DEVICE_TABLE (pci, pdsc_id_table); +void pdsc_queue_health_check (struct pdsc *pdsc) +{ +unsigned long mask; + +/* Don't do a check when in a transition state */ + mask =BIT_ULL (PDSC_S_INITING_DRIVER ) | +BIT_ULL (PDSC_S_STOPPING_DRIVER ); +if (pdsc->state & mask) +return ; + +/* Queue a new health check if one isn't already queued */ +queue_work (pdsc->wq, &pdsc->health_work); +} + +static void pdsc_wdtimer_cb (struct timer_list *t) +{ +struct pdsc *pdsc =from_timer (pdsc, t, wdtimer); + +dev_dbg (pdsc->dev,"%s: jiffies %ld\n" , __func__, jiffies); +mod_timer (&pdsc->wdtimer, +round_jiffies (jiffies + pdsc->wdtimer_period)); + +pdsc_queue_health_check (pdsc); +} +static void pdsc_unmap_bars (struct pdsc *pdsc) {struct pdsc_dev_bar *bars = pdsc->bars;@@ -135,8 +160,11 @@ static int pdsc_init_vf(struct pdsc *pdsc)return -1 ; } +#define PDSC_WQ_NAME_LEN 24 +static int pdsc_init_pf (struct pdsc *pdsc) { +char wq_name[PDSC_WQ_NAME_LEN ];int err;pcie_print_link_status (pdsc->pdev);@@ -151,6 +179,13 @@ static int pdsc_init_pf(struct pdsc *pdsc)if (err) goto err_out_release_regions; +/* General workqueue and timer, but don't start timer yet */ +snprintf (wq_name,sizeof (wq_name),"%s.%d" ,PDS_CORE_DRV_NAME , pdsc->id); + pdsc->wq =create_singlethread_workqueue (wq_name); +INIT_WORK (&pdsc->health_work, pdsc_health_thread); +timer_setup (&pdsc->wdtimer, pdsc_wdtimer_cb,0 ); + pdsc->wdtimer_period =PDSC_WATCHDOG_SECS *HZ ; +mutex_init (&pdsc->devcmd_lock);mutex_init (&pdsc->config_lock);@@ -167,11 +202,20 @@ static int pdsc_init_pf(struct pdsc *pdsc)mutex_unlock (&pdsc->config_lock); +/* Lastly, start the health check timer */ +mod_timer (&pdsc->wdtimer,round_jiffies (jiffies + pdsc->wdtimer_period)); +return 0 ;
err_out_teardown:pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
err_out_unmap_bars:drivers/net/ethernet/amd/pds_core/main.c +6
-0+del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq); + pdsc->wq =NULL ; + }mutex_unlock(&pdsc->config_lock); mutex_destroy(&pdsc->config_lock); mutex_destroy(&pdsc->devcmd_lock);drivers/net/ethernet/amd/pds_core/main.c +7
-0@@ -262,6 +306,13 @@ static void pdsc_remove(struct pci_dev *pdev)mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); +del_timer_sync (&pdsc->wdtimer); +if (pdsc->wq) { +flush_workqueue (pdsc->wq); +destroy_workqueue (pdsc->wq); + pdsc->wq =NULL ; + } +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );mutex_unlock (&pdsc->config_lock);mutex_destroy (&pdsc->config_lock);include/linux/pds/pds_core.h +10
-0@@ -12,6 +12,8 @@#include <linux/pds/pds_intr.h> #define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver" + +#define PDSC_WATCHDOG_SECS 5#define PDSC_TEARDOWN_RECOVERY false#define PDSC_TEARDOWN_REMOVING true#define PDSC_SETUP_RECOVERY false@@ -63,12 +65,17 @@ struct pdsc {u8 fw_generation;unsigned long last_fw_time;u32 last_hb; +struct timer_list wdtimer; +unsigned int wdtimer_period; +struct work_struct health_work;struct pdsc_devinfo dev_info;struct pds_core_dev_identity dev_ident;unsigned int nintrs;struct pdsc_intr_info *intr_info;/* array of nintrs elements */ +struct workqueue_struct *wq; +unsigned int devcmd_timeout;struct mutex devcmd_lock;/* lock for dev_cmd operations */ struct mutex config_lock;/* lock for configuration operations */ @@ -81,6 +88,8 @@ struct pdsc {u64 __iomem *kern_dbpage; }; +void pdsc_queue_health_check (struct pdsc *pdsc); +struct pdsc *pdsc_dl_alloc (struct device *dev,bool is_pf);void pdsc_dl_free (struct pdsc *pdsc);int pdsc_dl_register (struct pdsc *pdsc);@@ -116,5 +125,6 @@ int pdsc_dev_init(struct pdsc *pdsc);int pdsc_setup (struct pdsc *pdsc,bool init);void pdsc_teardown (struct pdsc *pdsc,bool removing); +void pdsc_health_thread (struct work_struct *work);#endif /* _PDSC_H_ */ Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 03/13] pds_core: health timer and workqueue
Quoted 6 lines · Jiri Pirko
Quoted 2 lines · Wed, Mar 08, 2023 at 06:13:00AM CET, [email protected] wrote:
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
Why don't you use devlink health to let the user know that something odd happened with HW?
Just haven't gotten to that yet.
sln
Re: [PATCH RFC v4 net-next 03/13] pds_core: health timer and workqueue
Quoted 9 lines · Thu, Mar 09, 2023 at 03:08:44AM CET, [email protected] wrote:
Quoted 6 lines · Jiri Pirko
Quoted 2 lines · Wed, Mar 08, 2023 at 06:13:00AM CET, [email protected] wrote:
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
Why don't you use devlink health to let the user know that something odd happened with HW?
Just haven't gotten to that yet.
Please do it from start, this is exactly why devlink health was introduced.
Quoted 2 lines
sln
Re: [PATCH RFC v4 net-next 03/13] pds_core: health timer and workqueue
Quoted 6 lines · Jiri Pirko
Quoted 5 lines · Thu, Mar 09, 2023 at 03:08:44AM CET, [email protected] wrote:
Quoted 4 lines · Jiri Pirko
Quoted 2 lines · Wed, Mar 08, 2023 at 06:13:00AM CET, [email protected] wrote:
Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Why don't you use devlink health to let the user know that something odd happened with HW?
Just haven't gotten to that yet.
Please do it from start, this is exactly why devlink health was introduced.
I'll add it in the next rev.
sln
RFC PATCH v4 4/13 pds_core: set up device and adminq
In reply to Shannon NelsonSet up the basic adminq and notifyq services. These are used mostly by the client drivers for feature configuration.
--- drivers/net/ethernet/amd/pds_core/core.c | 445 +++++++++++++- drivers/net/ethernet/amd/pds_core/debugfs.c | 106 ++++ drivers/net/ethernet/amd/pds_core/devlink.c | 53 ++ drivers/net/ethernet/amd/pds_core/main.c | 16 +- include/linux/pds/pds_adminq.h | 633 ++++++++++++++++++++ include/linux/pds/pds_core.h | 150 +++++ 6 files changed, 1399 insertions(+), 4 deletions(-) create mode 100644 include/linux/pds/pds_adminq.h
drivers/net/ethernet/amd/pds_core/core.c +442 -3
@@ -1,10 +1,371 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2023 Advanced Micro Devices, Inc */ +#include <linux/pci.h> +#include <linux/pds/pds_core.h> +#include <linux/pds/pds_adminq.h> + +void pdsc_work_thread (struct work_struct *work) +{ +/* stub */ +} + +irqreturn_t pdsc_adminq_isr (int irq,void *data) +{ +/* stub */ +return IRQ_HANDLED ; +} + +void pdsc_intr_free (struct pdsc *pdsc,int index) +{ +struct pdsc_intr_info *intr_info; + +if (index >= pdsc->nintrs || index <0 ) { +WARN (true,"bad intr index %d\n" , index); +return ; + } + + intr_info = &pdsc->intr_info[index]; +if (!intr_info->vector) +return ; +dev_dbg (pdsc->dev,"%s: idx %d vec %d name %s\n" , + __func__, index, intr_info->vector, intr_info->name); + +pds_core_intr_mask (&pdsc->intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); +pds_core_intr_clean (&pdsc->intr_ctrl[index]); + +free_irq (intr_info->vector, intr_info->data); + +memset (intr_info,0 ,sizeof (*intr_info)); +} + +int pdsc_intr_alloc (struct pdsc *pdsc,char *name, +irq_handler_t handler,void *data) +{ +struct pdsc_intr_info *intr_info; +unsigned int index; +int err; + +/* Find the first available interrupt */ +for (index =0 ; index < pdsc->nintrs; index++) +if (!pdsc->intr_info[index].vector) +break ; +if (index >= pdsc->nintrs) { +dev_warn (pdsc->dev,"%s: no intr, index=%d nintrs=%d\n" , + __func__, index, pdsc->nintrs); +return -ENOSPC ; + } + +pds_core_intr_clean_flags (&pdsc->intr_ctrl[index], +PDS_CORE_INTR_CRED_RESET_COALESCE ); + + intr_info = &pdsc->intr_info[index]; + + intr_info->index = index; + intr_info->data = data; +strscpy (intr_info->name, name,sizeof (intr_info->name)); + +/* Get the OS vector number for the interrupt */ + err =pci_irq_vector (pdsc->pdev, index); +if (err <0 ) { +dev_err (pdsc->dev,"failed to get intr vector index %d: %pe\n" , + index,ERR_PTR (err)); + goto err_out_free_intr; + } + intr_info->vector = err; + +/* Init the device's intr mask */ +pds_core_intr_clean (&pdsc->intr_ctrl[index]); +pds_core_intr_mask_assert (&pdsc->intr_ctrl[index],1 ); +pds_core_intr_mask (&pdsc->intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); + +/* Register the isr with a name */ + err =request_irq (intr_info->vector, handler,0 , intr_info->name, data); +if (err) { +dev_err (pdsc->dev,"failed to get intr irq vector %d: %pe\n" , + intr_info->vector,ERR_PTR (err)); + goto err_out_free_intr; + } + +return index; + +err_out_free_intr: +pdsc_intr_free (pdsc, index); +return err; +} + +static void pdsc_qcq_intr_free (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +if (!(qcq->flags &PDS_CORE_QCQ_F_INTR ) || + qcq->intx ==PDS_CORE_INTR_INDEX_NOT_ASSIGNED ) +return ; + +pdsc_intr_free (pdsc, qcq->intx); + qcq->intx =PDS_CORE_INTR_INDEX_NOT_ASSIGNED ; +} + +static int pdsc_qcq_intr_alloc (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +char name[PDSC_INTR_NAME_MAX_SZ ]; +int index; + +if (!(qcq->flags &PDS_CORE_QCQ_F_INTR )) { + qcq->intx =PDS_CORE_INTR_INDEX_NOT_ASSIGNED ; +return 0 ; + } + +snprintf (name,sizeof (name), +"%s-%d-%s" ,PDS_CORE_DRV_NAME , pdsc->pdev->bus->number, qcq->q.name); + index =pdsc_intr_alloc (pdsc, name, pdsc_adminq_isr, qcq); +if (index <0 ) +return index; + qcq->intx = index; + +return 0 ; +} + +void pdsc_qcq_free (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +struct device *dev = pdsc->dev; + +if (!(qcq && qcq->pdsc)) +return ; + +pdsc_debugfs_del_qcq (qcq); + +pdsc_qcq_intr_free (pdsc, qcq); + +if (qcq->q_base) { +dma_free_coherent (dev, qcq->q_size, + qcq->q_base, qcq->q_base_pa); + qcq->q_base =NULL ; + qcq->q_base_pa =0 ; + } + +if (qcq->cq_base) { +dma_free_coherent (dev, qcq->cq_size, qcq->cq_base, qcq->cq_base_pa); + qcq->cq_base =NULL ; + qcq->cq_base_pa =0 ; + } + +if (qcq->cq.info) { +vfree (qcq->cq.info); + qcq->cq.info =NULL ; + } +if (qcq->q.info) { +vfree (qcq->q.info); + qcq->q.info =NULL ; + } + + qcq->pdsc =NULL ; +memset (&qcq->q,0 ,sizeof (qcq->q)); +memset (&qcq->cq,0 ,sizeof (qcq->cq)); +} + +static void pdsc_q_map (struct pdsc_queue *q,void *base,dma_addr_t base_pa) +{ +struct pdsc_q_info *cur; +unsigned int i; + + q->base = base; + q->base_pa = base_pa; + +for (i =0 , cur = q->info; i < q->num_descs; i++, cur++) + cur->desc = base + (i * q->desc_size); +} + +static void pdsc_cq_map (struct pdsc_cq *cq,void *base,dma_addr_t base_pa) +{ +struct pdsc_cq_info *cur; +unsigned int i; + + cq->base = base; + cq->base_pa = base_pa; + +for (i =0 , cur = cq->info; i < cq->num_descs; i++, cur++) + cur->comp = base + (i * cq->desc_size); +} + +int pdsc_qcq_alloc (struct pdsc *pdsc,unsigned int type,unsigned int index, +const char *name,unsigned int flags,unsigned int num_descs, +unsigned int desc_size,unsigned int cq_desc_size, +unsigned int pid,struct pdsc_qcq *qcq) +{ +struct device *dev = pdsc->dev; +dma_addr_t cq_base_pa =0 ; +dma_addr_t q_base_pa =0 ; +void *q_base, *cq_base; +int err; + + qcq->q.info =vzalloc (num_descs *sizeof (*qcq->q.info)); +if (!qcq->q.info) { +dev_err (dev,"Cannot allocate %s queue info\n" , name); + err = -ENOMEM ; + goto err_out; + } + + qcq->pdsc = pdsc; + qcq->flags = flags; +INIT_WORK (&qcq->work, pdsc_work_thread); + + qcq->q.type = type; + qcq->q.index = index; + qcq->q.num_descs = num_descs; + qcq->q.desc_size = desc_size; + qcq->q.tail_idx =0 ; + qcq->q.head_idx =0 ; + qcq->q.pid = pid; +snprintf (qcq->q.name,sizeof (qcq->q.name),"%s%u" , name, index); + + err =pdsc_qcq_intr_alloc (pdsc, qcq); +if (err) + goto err_out_free_q_info; + + qcq->cq.info =vzalloc (num_descs *sizeof (*qcq->cq.info)); +if (!qcq->cq.info) { +dev_err (dev,"Cannot allocate %s completion queue info\n" , name); + err = -ENOMEM ; + goto err_out_free_irq; + } + + qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx]; + qcq->cq.num_descs = num_descs; + qcq->cq.desc_size = cq_desc_size; + qcq->cq.tail_idx =0 ; + qcq->cq.done_color =1 ; + +if (flags &PDS_CORE_QCQ_F_NOTIFYQ ) { +/* q & cq need to be contiguous in case of notifyq */ + qcq->q_size =PAGE_SIZE +ALIGN (num_descs * desc_size,PAGE_SIZE ) + +ALIGN (num_descs * cq_desc_size,PAGE_SIZE ); + qcq->q_base =dma_alloc_coherent (dev, qcq->q_size + qcq->cq_size, + &qcq->q_base_pa, +GFP_KERNEL ); +if (!qcq->q_base) { +dev_err (dev,"Cannot allocate %s qcq DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_cq_info; + } + q_base =PTR_ALIGN (qcq->q_base,PAGE_SIZE ); + q_base_pa =ALIGN (qcq->q_base_pa,PAGE_SIZE ); +pdsc_q_map (&qcq->q, q_base, q_base_pa); + + cq_base =PTR_ALIGN (q_base + +ALIGN (num_descs * desc_size,PAGE_SIZE ),PAGE_SIZE ); + cq_base_pa =ALIGN (qcq->q_base_pa + +ALIGN (num_descs * desc_size,PAGE_SIZE ),PAGE_SIZE ); + + }else { +/* q DMA descriptors */ + qcq->q_size =PAGE_SIZE + (num_descs * desc_size); + qcq->q_base =dma_alloc_coherent (dev, qcq->q_size, + &qcq->q_base_pa, +GFP_KERNEL ); +if (!qcq->q_base) { +dev_err (dev,"Cannot allocate %s queue DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_cq_info; + } + q_base =PTR_ALIGN (qcq->q_base,PAGE_SIZE ); + q_base_pa =ALIGN (qcq->q_base_pa,PAGE_SIZE ); +pdsc_q_map (&qcq->q, q_base, q_base_pa); + +/* cq DMA descriptors */ + qcq->cq_size =PAGE_SIZE + (num_descs * cq_desc_size); + qcq->cq_base =dma_alloc_coherent (dev, qcq->cq_size, + &qcq->cq_base_pa, +GFP_KERNEL ); +if (!qcq->cq_base) { +dev_err (dev,"Cannot allocate %s cq DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_q; + } + cq_base =PTR_ALIGN (qcq->cq_base,PAGE_SIZE ); + cq_base_pa =ALIGN (qcq->cq_base_pa,PAGE_SIZE ); + } + +pdsc_cq_map (&qcq->cq, cq_base, cq_base_pa); + qcq->cq.bound_q = &qcq->q; + +pdsc_debugfs_add_qcq (pdsc, qcq); + +return 0 ; + +err_out_free_q: +dma_free_coherent (dev, qcq->q_size, qcq->q_base, qcq->q_base_pa); +err_out_free_cq_info: +vfree (qcq->cq.info); +err_out_free_irq: +pdsc_qcq_intr_free (pdsc, qcq); +err_out_free_q_info: +vfree (qcq->q.info); +memset (qcq,0 ,sizeof (*qcq)); +err_out: +dev_err (dev,"qcq alloc of %s%d failed %d\n" , name, index, err); +return err; +} + +static int pdsc_core_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .init.opcode =PDS_CORE_CMD_INIT , + }; +struct pds_core_dev_init_data_out cido; +struct pds_core_dev_init_data_in cidi; +u32 dbid_count; +u32 dbpage_num; +size_t sz; +int err; + + cidi.adminq_q_base =cpu_to_le64 (pdsc->adminqcq.q_base_pa); + cidi.adminq_cq_base =cpu_to_le64 (pdsc->adminqcq.cq_base_pa); + cidi.notifyq_cq_base =cpu_to_le64 (pdsc->notifyqcq.cq.base_pa); + cidi.flags =cpu_to_le32 (PDS_CORE_QINIT_F_IRQ |PDS_CORE_QINIT_F_ENA ); + cidi.intr_index =cpu_to_le16 (pdsc->adminqcq.intx); + cidi.adminq_ring_size =ilog2 (pdsc->adminqcq.q.num_descs); + cidi.notifyq_ring_size =ilog2 (pdsc->notifyqcq.q.num_descs); + +mutex_lock (&pdsc->devcmd_lock); + + sz =min_t (size_t,sizeof (cidi),sizeof (pdsc->cmd_regs->data)); +memcpy_toio (&pdsc->cmd_regs->data, &cidi, sz); + + err =pdsc_devcmd_locked (pdsc, &cmd, &comp, pdsc->devcmd_timeout); + + sz =min_t (size_t,sizeof (cido),sizeof (pdsc->cmd_regs->data)); +memcpy_fromio (&cido, &pdsc->cmd_regs->data, sz); + +mutex_unlock (&pdsc->devcmd_lock); + + pdsc->hw_index =le32_to_cpu (cido.core_hw_index); + + dbid_count =le32_to_cpu (pdsc->dev_ident.ndbpgs_per_lif); + dbpage_num = pdsc->hw_index * dbid_count; + pdsc->kern_dbpage =pdsc_map_dbpage (pdsc, dbpage_num); +if (!pdsc->kern_dbpage) { +dev_err (pdsc->dev,"Cannot map dbpage, aborting\n" ); +return -ENOMEM ; + } + + pdsc->adminqcq.q.hw_type = cido.adminq_hw_type; + pdsc->adminqcq.q.hw_index =le32_to_cpu (cido.adminq_hw_index); + pdsc->adminqcq.q.dbval =PDS_CORE_DBELL_QID (pdsc->adminqcq.q.hw_index); + + pdsc->notifyqcq.q.hw_type = cido.notifyq_hw_type; + pdsc->notifyqcq.q.hw_index =le32_to_cpu (cido.notifyq_hw_index); + pdsc->notifyqcq.q.dbval =PDS_CORE_DBELL_QID (pdsc->notifyqcq.q.hw_index); + + pdsc->last_eid =0 ; + +return err; +} +int pdsc_setup (struct pdsc *pdsc,bool init) { +int numdescs;int err =0 ;if (init)
@@ -14,17 +375,60 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (err)return err; +/* Scale the descriptor ring length based on number of CPUs and VFs */ + numdescs =max_t (int,PDSC_ADMINQ_MIN_LENGTH ,num_online_cpus ()); + numdescs +=2 *pci_sriov_get_totalvfs (pdsc->pdev); + numdescs =roundup_pow_of_two (numdescs); + err =pdsc_qcq_alloc (pdsc,PDS_CORE_QTYPE_ADMINQ ,0 ,"adminq" , +PDS_CORE_QCQ_F_CORE |PDS_CORE_QCQ_F_INTR , + numdescs, +sizeof (union pds_core_adminq_cmd ), +sizeof (union pds_core_adminq_comp ), +0 , &pdsc->adminqcq); +if (err) + goto err_out_teardown; + + err =pdsc_qcq_alloc (pdsc,PDS_CORE_QTYPE_NOTIFYQ ,0 ,"notifyq" , +PDS_CORE_QCQ_F_NOTIFYQ , +PDSC_NOTIFYQ_LENGTH , +sizeof (struct pds_core_notifyq_cmd ), +sizeof (union pds_core_notifyq_comp ), +0 , &pdsc->notifyqcq); +if (err) + goto err_out_teardown; + +/* NotifyQ rides on the AdminQ interrupt */ + pdsc->notifyqcq.intx = pdsc->adminqcq.intx; + +/* Set up the Core with the AdminQ and NotifyQ info */ + err =pdsc_core_init (pdsc); +if (err) + goto err_out_teardown; +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ; + +err_out_teardown: +pdsc_teardown (pdsc, init); +return err; }void pdsc_teardown (struct pdsc *pdsc,bool removing) { +int i; +pdsc_devcmd_reset (pdsc); +pdsc_qcq_free (pdsc, &pdsc->notifyqcq); +pdsc_qcq_free (pdsc, &pdsc->adminqcq); + +if (pdsc->intr_info) { +for (i =0 ; i < pdsc->nintrs; i++) +pdsc_intr_free (pdsc, i); -if (removing) { -kfree (pdsc->intr_info); - pdsc->intr_info =NULL ; +if (removing) { +kfree (pdsc->intr_info); + pdsc->intr_info =NULL ; + } }if (pdsc->kern_dbpage) {
@@ -35,6 +439,36 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } +int pdsc_start (struct pdsc *pdsc) +{ +pds_core_intr_mask (&pdsc->intr_ctrl[pdsc->adminqcq.intx], +PDS_CORE_INTR_MASK_CLEAR ); + +return 0 ; +} + +static void pdsc_mask_interrupts (struct pdsc *pdsc) +{ +int i; + +if (!pdsc->intr_info) +return ; + +/* Mask interrupts that are in use */ +for (i =0 ; i < pdsc->nintrs; i++) +if (pdsc->intr_info[i].vector) +pds_core_intr_mask (&pdsc->intr_ctrl[i], +PDS_CORE_INTR_MASK_SET ); +} + +void pdsc_stop (struct pdsc *pdsc) +{ +if (pdsc->wq) +flush_workqueue (pdsc->wq); + +pdsc_mask_interrupts (pdsc); +} +static void pdsc_fw_down (struct pdsc *pdsc) {mutex_lock (&pdsc->config_lock);
@@ -45,6 +479,7 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );mutex_unlock (&pdsc->config_lock);
@@ -66,6 +501,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; + err =pdsc_start (pdsc); +if (err) + goto err_out; +mutex_unlock (&pdsc->config_lock);return ;
drivers/net/ethernet/amd/pds_core/debugfs.c +106 -0
110 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/devlink.c +53 -0
60 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +15 -1
@@ -155,6 +155,13 @@ static int pdsc_map_bars(struct pdsc *pdsc)return err; } +void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num) +{ +return pci_iomap_range (pdsc->pdev, + pdsc->bars[PDS_CORE_PCI_BAR_DBELL ].res_index, + (u64 )page_num <<PAGE_SHIFT ,PAGE_SIZE ); +} +static int pdsc_init_vf (struct pdsc *pdsc) {return -1 ;
@@ -188,6 +195,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)mutex_init (&pdsc->devcmd_lock);mutex_init (&pdsc->config_lock); +spin_lock_init (&pdsc->adminq_lock);mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_FW_DEAD , &pdsc->state);
@@ -195,10 +203,13 @@ static int pdsc_init_pf(struct pdsc *pdsc) err =pdsc_setup (pdsc,PDSC_SETUP_INIT );if (err) goto err_out_unmap_bars; + err =pdsc_start (pdsc); +if (err) + goto err_out_teardown; err =pdsc_dl_register (pdsc);if (err) - goto err_out_teardown; + goto err_out_stop;mutex_unlock (&pdsc->config_lock);
@@ -207,6 +218,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)return 0 ; +err_out_stop: +pdsc_stop (pdsc); err_out_teardown:pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); err_out_unmap_bars:
@@ -313,6 +326,7 @@ static void pdsc_remove(struct pci_dev *pdev) pdsc->wq =NULL ; } +pdsc_stop (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );mutex_unlock (&pdsc->config_lock);mutex_destroy (&pdsc->config_lock);
include/linux/pds/pds_adminq.h new file +633 -0
633 lines not on this page · raw message
include/linux/pds/pds_core.h +150 -0
208 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v4 5/13 pds_core: Add adminq processing and commands
In reply to Shannon NelsonAdd the service routines for submitting and processing the adminq messages and for handling notifyq events.
--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/adminq.c | 281 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/core.c | 11 - include/linux/pds/pds_adminq.h | 2 + include/linux/pds/pds_core.h | 6 + 5 files changed, 290 insertions(+), 11 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ dev.o \ + adminq.o \ core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/adminq.c new file +281 -0
281 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/core.c +0 -11
@@ -7,17 +7,6 @@#include <linux/pds/pds_adminq.h> -void pdsc_work_thread (struct work_struct *work) -{ -/* stub */ -} - -irqreturn_t pdsc_adminq_isr (int irq,void *data) -{ -/* stub */ -return IRQ_HANDLED ; -} -void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;
include/linux/pds/pds_adminq.h +2 -0
@@ -4,6 +4,8 @@#ifndef _PDS_CORE_ADMINQ_H_#define _PDS_CORE_ADMINQ_H_ +#define PDSC_ADMINQ_MAX_POLL_INTERVAL 256 +enum pds_core_adminq_flags {PDS_AQ_FLAG_FASTPOLL =BIT (1 ),/* poll for completion at 1ms intervals */ };
include/linux/pds/pds_core.h +6 -0
@@ -259,6 +259,12 @@ int pdsc_devcmd_reset(struct pdsc *pdsc);int pdsc_dev_reinit (struct pdsc *pdsc);int pdsc_dev_init (struct pdsc *pdsc); +int pdsc_adminq_post (struct pdsc *pdsc, +struct pdsc_qcq *qcq, +union pds_core_adminq_cmd *cmd, +union pds_core_adminq_comp *comp, +bool fast_poll); +int pdsc_intr_alloc (struct pdsc *pdsc,char *name,irq_handler_t handler,void *data);void pdsc_intr_free (struct pdsc *pdsc,int index);
Signature · 1 line
2.17.1
RFC PATCH v4 6/13 pds_core: add FW update feature to devlink
In reply to Shannon NelsonAdd in the support for doing firmware updates. Of the two main banks available, 1, and 2, this updates the one not in use and then selects it for the next boot.
--- drivers/net/ethernet/amd/pds_core/Makefile | 3 +- drivers/net/ethernet/amd/pds_core/devlink.c | 10 ++ drivers/net/ethernet/amd/pds_core/fw.c | 187 ++++++++++++++++++++ include/linux/pds/pds_core.h | 3 + 4 files changed, 202 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
drivers/net/ethernet/amd/pds_core/Makefile +2 -1
@@ -7,6 +7,7 @@ pds_core-y := main.o \ devlink.o \ dev.o \ adminq.o \ - core.o + core.o \ + fw.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/devlink.c +10 -0
@@ -8,6 +8,15 @@#include <linux/pds/pds_core.h> +static int pdsc_dl_flash_update (struct devlink *dl, +struct devlink_flash_update_params *params, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_priv (dl); + +return pdsc_firmware_update (pdsc, params->fw, extack); +} +static int pdsc_dl_info_get (struct devlink *dl,struct devlink_info_req *req,struct netlink_ext_ack *extack) {
@@ -62,6 +71,7 @@ static int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,static const struct devlink_ops pdsc_dl_ops = { .info_get = pdsc_dl_info_get, + .flash_update = pdsc_dl_flash_update, };static const struct devlink_ops pdsc_dl_vf_ops = {
drivers/net/ethernet/amd/pds_core/fw.c new file +187 -0
187 lines not on this page · raw message
include/linux/pds/pds_core.h +3 -0
@@ -283,4 +283,7 @@ void pdsc_process_adminq(struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data); +int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw, +struct netlink_ext_ack *extack); +#endif /* _PDSC_H_ */
Signature · 1 line
2.17.1
RFC PATCH v4 7/13 pds_core: set up the VIF definitions and defaults
In reply to Shannon NelsonThe Virtual Interfaces (VIFs) supported by the DSC's configuration (VFio Live Migration, vDPA, etc) are reported in the dev_ident struct.
--- drivers/net/ethernet/amd/pds_core/core.c | 47 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/debugfs.c | 23 ++++++++++ include/linux/pds/pds_common.h | 17 ++++++++ include/linux/pds/pds_core.h | 13 ++++++ 4 files changed, 100 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +47 -0
@@ -352,6 +352,42 @@ static int pdsc_core_init(struct pdsc *pdsc)return err; } +static struct pdsc_viftype pdsc_viftype_defaults[] = { + [PDS_DEV_TYPE_VDPA ] = { .name =PDS_DEV_TYPE_VDPA_STR , + .vif_id =PDS_DEV_TYPE_VDPA , + .dl_id =DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET }, + [PDS_DEV_TYPE_MAX ] = {0 } +}; + +static int pdsc_viftypes_init (struct pdsc *pdsc) +{ +enum pds_core_vif_types vt; + + pdsc->viftype_status =kzalloc (sizeof (pdsc_viftype_defaults),GFP_KERNEL ); +if (!pdsc->viftype_status) +return -ENOMEM ; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +bool vt_support; + +if (!pdsc_viftype_defaults[vt].name) +continue ; + +/* Grab the defaults */ + pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt]; + +/* See what the Core device has for support */ + vt_support = !!le16_to_cpu (pdsc->dev_ident.vif_types[vt]); +dev_dbg (pdsc->dev,"VIF %s is %ssupported\n" , + pdsc->viftype_status[vt].name, + vt_support ?"" :"not " ); + + pdsc->viftype_status[vt].supported = vt_support; + } + +return 0 ; +} +int pdsc_setup (struct pdsc *pdsc,bool init) {int numdescs;
@@ -394,6 +430,14 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (err) goto err_out_teardown; +/* Set up the VIFs */ + err =pdsc_viftypes_init (pdsc); +if (err) + goto err_out_teardown; + +if (init) +pdsc_debugfs_add_viftype (pdsc); +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ;
@@ -410,6 +454,9 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)pdsc_qcq_free (pdsc, &pdsc->notifyqcq);pdsc_qcq_free (pdsc, &pdsc->adminqcq); +kfree (pdsc->viftype_status); + pdsc->viftype_status =NULL ; +if (pdsc->intr_info) {for (i =0 ; i < pdsc->nintrs; i++)pdsc_intr_free (pdsc, i);
drivers/net/ethernet/amd/pds_core/debugfs.c +23 -0
@@ -82,6 +82,29 @@ void pdsc_debugfs_add_ident(struct pdsc *pdsc)debugfs_create_file ("identity" ,0400 , pdsc->dentry, pdsc, &identity_fops); } +static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name) +continue ; + +seq_printf (seq,"%s\t%d supported %d enabled\n" , + pdsc->viftype_status[vt].name, + pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype); + +void pdsc_debugfs_add_viftype (struct pdsc *pdsc) +{ +debugfs_create_file ("viftypes" ,0400 , pdsc->dentry, pdsc, &viftype_fops); +} +static int irqs_show (struct seq_file *seq,void *v) {struct pdsc *pdsc = seq->private;
include/linux/pds/pds_common.h +17 -0
@@ -50,6 +50,23 @@ enum pds_core_driver_type { PDS_DRIVER_ESXI = 6, }; +enum pds_core_vif_types { +PDS_DEV_TYPE_VDPA =1 , +PDS_DEV_TYPE_VFIO =2 , +PDS_DEV_TYPE_ETH =3 , +PDS_DEV_TYPE_RDMA =4 , +PDS_DEV_TYPE_LM =5 , + +/* new ones added before this line */ +PDS_DEV_TYPE_MAX =16 /* don't change - used in struct size */ +}; + +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_DEV_TYPE_VFIO_STR "VFio" +#define PDS_DEV_TYPE_ETH_STR "Eth" +#define PDS_DEV_TYPE_RDMA_STR "RDMA" +#define PDS_DEV_TYPE_LM_STR "LM" +#define PDS_CORE_IFNAMSIZ 16/**
include/linux/pds/pds_core.h +13 -0
@@ -123,6 +123,15 @@ struct pdsc_qcq {struct dentry *dentry; }; +struct pdsc_viftype { +char *name; +bool supported; +bool enabled; +int dl_id; +int vif_id; +struct pds_auxiliary_dev *padev; +}; +/* No state flags set means we are in a steady running state */ enum pdsc_state_flags {PDSC_S_FW_DEAD ,/* fw stopped, waiting for startup or recovery */
@@ -172,6 +181,7 @@ struct pdsc {struct pdsc_qcq adminqcq;struct pdsc_qcq notifyqcq;u64 last_eid; +struct pdsc_viftype *viftype_status; };/** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -226,6 +236,7 @@ struct pdsc *pdsc_dl_alloc(struct device *dev, bool is_pf);void pdsc_dl_free (struct pdsc *pdsc);int pdsc_dl_register (struct pdsc *pdsc);void pdsc_dl_unregister (struct pdsc *pdsc); +int pdsc_dl_vif_add (struct pdsc *pdsc,enum pds_core_vif_types vt,const char *name);#ifdef CONFIG_DEBUG_FS void pdsc_debugfs_create (void );
@@ -233,6 +244,7 @@ void pdsc_debugfs_destroy(void);void pdsc_debugfs_add_dev (struct pdsc *pdsc);void pdsc_debugfs_del_dev (struct pdsc *pdsc);void pdsc_debugfs_add_ident (struct pdsc *pdsc); +void pdsc_debugfs_add_viftype (struct pdsc *pdsc);void pdsc_debugfs_add_irqs (struct pdsc *pdsc);void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq);void pdsc_debugfs_del_qcq (struct pdsc_qcq *qcq);
@@ -242,6 +254,7 @@ static inline void pdsc_debugfs_destroy(void) { }static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_ident (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_add_viftype (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_irqs (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq) { }static inline void pdsc_debugfs_del_qcq (struct pdsc_qcq *qcq) { }
Signature · 1 line
2.17.1
RFC PATCH v4 8/13 pds_core: add initial VF device handling
In reply to Shannon NelsonThis is the initial VF PCI driver framework for the new pds_vdpa VF device, which will work in conjunction with an auxiliary_bus client of the pds_core driver. This does the very basics of registering for the new PCI device 1dd8:100b, setting up debugfs entries, and registering with devlink.
--- drivers/net/ethernet/amd/pds_core/main.c | 45 +++++++++++++++++++++++- include/linux/pds/pds_core.h | 10 +++++- include/linux/pds/pds_core_if.h | 1 + 3 files changed, 54 insertions(+), 2 deletions(-)
drivers/net/ethernet/amd/pds_core/main.c +44 -1
@@ -21,6 +21,7 @@ MODULE_LICENSE("GPL");/* Supported devices */ static const struct pci_device_id pdsc_id_table[] = { {PCI_VDEVICE (PENSANDO ,PCI_DEVICE_ID_PENSANDO_CORE_PF ) }, + {PCI_VDEVICE (PENSANDO ,PCI_DEVICE_ID_PENSANDO_VDPA_VF ) }, {0 , }/* end of table */ };MODULE_DEVICE_TABLE (pci, pdsc_id_table);
@@ -162,9 +163,48 @@ void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num) (u64)page_num << PAGE_SHIFT, PAGE_SIZE); } +static int pdsc_sriov_configure (struct pci_dev *pdev,int num_vfs) +{ +struct pdsc *pdsc =pci_get_drvdata (pdev); +struct device *dev = pdsc->dev; +int ret =0 ; + +if (num_vfs >0 ) { + pdsc->vfs =kcalloc (num_vfs,sizeof (struct pdsc_vf ),GFP_KERNEL ); +if (!pdsc->vfs) +return -ENOMEM ; + pdsc->num_vfs = num_vfs; + + ret =pci_enable_sriov (pdev, num_vfs); +if (ret) { +dev_err (dev,"Cannot enable SRIOV: %pe\n" ,ERR_PTR (ret)); + goto no_vfs; + } + +return num_vfs; + } + +no_vfs: +pci_disable_sriov (pdev); + +kfree (pdsc->vfs); + pdsc->vfs =NULL ; + pdsc->num_vfs =0 ; + +return ret; +} +static int pdsc_init_vf (struct pdsc *pdsc) { -return -1 ; +int err; + + pdsc->vf_id =pci_iov_vf_id (pdsc->pdev); + + err =pdsc_dl_register (pdsc); +if (err) +return err; + +return 0 ; }#define PDSC_WQ_NAME_LEN 24
@@ -316,6 +356,8 @@ static void pdsc_remove(struct pci_dev *pdev)pdsc_dl_unregister (pdsc);if (!pdev->is_virtfn) { +pdsc_sriov_configure (pdev,0 ); +mutex_lock (&pdsc->config_lock);set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state);
@@ -350,6 +392,7 @@ static struct pci_driver pdsc_driver = { .id_table = pdsc_id_table, .probe = pdsc_probe, .remove = pdsc_remove, + .sriov_configure = pdsc_sriov_configure, };static int __init pdsc_init_module (void )
include/linux/pds/pds_core.h +9 -1
@@ -30,6 +30,12 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc_vf { +struct pds_auxiliary_dev *padev; +u16 index; +__le16 vif_types[PDS_DEV_TYPE_MAX ]; +}; +struct pdsc_devinfo {u8 asic_type;u8 asic_rev;
@@ -147,6 +153,9 @@ struct pdsc {struct dentry *dentry;struct device *dev;struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX ]; +struct pdsc_vf *vfs; +int num_vfs; +int vf_id;int hw_index;int id;
@@ -298,5 +307,4 @@ irqreturn_t pdsc_adminq_isr(int irq, void *data);int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw,struct netlink_ext_ack *extack); -#endif /* _PDSC_H_ */
include/linux/pds/pds_core_if.h +1 -0
@@ -8,6 +8,7 @@#define PCI_VENDOR_ID_PENSANDO 0x1dd8#define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c +#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b#define PDS_CORE_BARS_MAX 4#define PDS_CORE_PCI_BAR_DBELL 1
Signature · 1 line
2.17.1
RFC PATCH v4 9/13 pds_core: add auxiliary_bus devices
In reply to Shannon NelsonAn auxiliary_bus device is created for each VF at VF probe and destroyed at VF remove. The VFs are always removed on PF remove, so there should be no issues with VFs trying to access missing PF structures.
--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/auxbus.c | 138 +++++++++++++++++++++ drivers/net/ethernet/amd/pds_core/main.c | 35 +++++- include/linux/pds/pds_auxbus.h | 16 +++ include/linux/pds/pds_core.h | 7 ++ 5 files changed, 195 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c create mode 100644 include/linux/pds/pds_auxbus.h
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ + auxbus.o \ dev.o \ adminq.o \ core.o \
drivers/net/ethernet/amd/pds_core/auxbus.c new file +138 -0
138 lines not on this page · raw message
drivers/net/ethernet/amd/pds_core/main.c +33 -2
@@ -196,15 +196,34 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)static int pdsc_init_vf (struct pdsc *pdsc) { +struct pdsc *pdsc_pf;int err; + pdsc_pf =pdsc_get_pf_struct (pdsc->pdev); +if (IS_ERR_OR_NULL (pdsc_pf)) +return PTR_ERR (pdsc_pf) ?:-1 ; + pdsc->vf_id =pci_iov_vf_id (pdsc->pdev); err =pdsc_dl_register (pdsc);if (err)return err; -return 0 ; + err =pdsc_auxbus_dev_add_vf (pdsc, pdsc_pf); +if (err) +pdsc_dl_unregister (pdsc); + +return err; +} + +static void pdsc_del_vf (struct pdsc *pdsc) +{ +struct pdsc *pdsc_pf; + + pdsc_pf =pdsc_get_pf_struct (pdsc->pdev); +if (IS_ERR (pdsc_pf)) +return ; +pdsc_auxbus_dev_del_vf (pdsc, pdsc_pf); }#define PDSC_WQ_NAME_LEN 24
@@ -355,7 +374,13 @@ static void pdsc_remove(struct pci_dev *pdev)*/ pdsc_dl_unregister (pdsc); -if (!pdev->is_virtfn) { +if (pdev->is_virtfn) { +pdsc_del_vf (pdsc); + }else { +/* Remove the VFs and their aux_bus connections before other +* cleanup so that the clients can use the AdminQ to cleanly +* shut themselves down. +*/ pdsc_sriov_configure (pdev,0 );mutex_lock (&pdsc->config_lock);
@@ -395,6 +420,12 @@ static struct pci_driver pdsc_driver = { .sriov_configure = pdsc_sriov_configure, }; +struct pdsc *pdsc_get_pf_struct (struct pci_dev *vf_pdev) +{ +return pci_iov_get_pf_drvdata (vf_pdev, &pdsc_driver); +}; +EXPORT_SYMBOL_GPL (pdsc_get_pf_struct); +static int __init pdsc_init_module (void ) {pdsc_debugfs_create ();
include/linux/pds/pds_auxbus.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev { +struct auxiliary_device aux_dev; +struct pdsc *vf; +struct pdsc *pf; +u16 client_id; +void *priv; +}; +#endif /* _PDSC_AUXBUS_H_ */
include/linux/pds/pds_core.h +7 -0
@@ -30,8 +30,11 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc ; +struct pdsc_vf {struct pds_auxiliary_dev *padev; +struct pdsc *vf;u16 index;__le16 vif_types[PDS_DEV_TYPE_MAX ]; };
@@ -238,6 +241,7 @@ static inline void pds_core_dbell_ring(u64 __iomem *db_page,writeq (val, &db_page[qtype]); } +struct pdsc *pdsc_get_pf_struct (struct pci_dev *vf_pdev);void pdsc_queue_health_check (struct pdsc *pdsc);void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num);
@@ -301,6 +305,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_auxbus_dev_add_vf (struct pdsc *pdsc_vf,struct pdsc *pdsc_pf); +int pdsc_auxbus_dev_del_vf (struct pdsc *pdsc_vf,struct pdsc *pdsc_pf); +void pdsc_process_adminq (struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data);
Signature · 1 line
2.17.1
RFC PATCH v4 10/13 pds_core: devlink params for enabling VIF support
In reply to Shannon NelsonAdd the devlink parameter switches so the user can enable the features supported by the VFs.
--- drivers/net/ethernet/amd/pds_core/devlink.c | 88 +++++++++++++++++++++ 1 file changed, 88 insertions(+)
drivers/net/ethernet/amd/pds_core/devlink.c +88 -0
@@ -7,6 +7,86 @@#include <linux/pci.h> #include <linux/pds/pds_core.h> +#include <linux/pds/pds_auxbus.h> + +static struct pdsc_viftype *pdsc_dl_find_viftype_by_id (struct pdsc *pdsc, +enum devlink_param_type dl_id) +{ +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (pdsc->viftype_status[vt].dl_id == dl_id) +return &pdsc->viftype_status[vt]; + } + +return NULL ; +} + +static int pdsc_dl_enable_get (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry) +return -ENOENT ; + + ctx->val.vbool = vt_entry->enabled; + +return 0 ; +} + +static int pdsc_dl_enable_set (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; +int err =0 ; +int vf_id; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry || !vt_entry->supported) +return -EOPNOTSUPP ; + +if (vt_entry->enabled == ctx->val.vbool) +return 0 ; + + vt_entry->enabled = ctx->val.vbool; +for (vf_id =0 ; vf_id < pdsc->num_vfs; vf_id++) { +struct pdsc *vf = pdsc->vfs[vf_id].vf; + + err = ctx->val.vbool ?pdsc_auxbus_dev_add_vf (vf, pdsc) : +pdsc_auxbus_dev_del_vf (vf, pdsc); + } + +return err; +} + +static int pdsc_dl_enable_validate (struct devlink *dl,u32 id, +union devlink_param_value val, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry || !vt_entry->supported) +return -EOPNOTSUPP ; + +if (!pdsc->viftype_status[vt_entry->vif_id].supported) +return -ENODEV ; + +return 0 ; +} + +static const struct devlink_param pdsc_dl_params[] = { +DEVLINK_PARAM_GENERIC (ENABLE_VNET , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate), +}; static int pdsc_dl_flash_update(struct devlink *dl, struct devlink_flash_update_params *params,
@@ -100,6 +180,14 @@ void pdsc_dl_free(struct pdsc *pdsc)int pdsc_dl_register (struct pdsc *pdsc) {struct devlink *dl =priv_to_devlink (pdsc); +int err; + +if (!pdsc->pdev->is_virtfn) { + err =devlink_params_register (dl, pdsc_dl_params, +ARRAY_SIZE (pdsc_dl_params)); +if (err) +return err; + }devlink_register (dl);
Signature · 1 line
2.17.1
RFC PATCH v4 11/13 pds_core: add the aux client API
In reply to Shannon NelsonAdd the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
--- drivers/net/ethernet/amd/pds_core/auxbus.c | 134 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 42 +++++++ 2 files changed, 174 insertions(+), 2 deletions(-)
drivers/net/ethernet/amd/pds_core/auxbus.c +132 -2
@@ -5,6 +5,132 @@#include <linux/pds/pds_adminq.h> #include <linux/pds/pds_auxbus.h> +/** +* pds_client_register - Register the client with the device +* @padev: ptr to the client device info +* +* Register the client with the core and with the DSC. The core +* will fill in the client padev->client_id for use in calls +* to the DSC AdminQ +*/ +static int pds_client_register (struct pds_auxiliary_dev *padev) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev = padev->pf->dev; +struct pdsc *pf = padev->pf; +int err; +u16 ci; + +if (pf->state) +return -ENXIO ; + + cmd.client_reg.opcode =PDS_AQ_CMD_CLIENT_REG ; +strscpy (cmd.client_reg.devname,dev_name (&padev->aux_dev.dev), +sizeof (cmd.client_reg.devname)); + + err =pdsc_adminq_post (pf, &pf->adminqcq, &cmd, &comp, false); +if (err) { +dev_info (dev,"register dev_name %s with DSC failed, status %d: %pe\n" , +dev_name (&padev->aux_dev.dev), comp.status,ERR_PTR (err)); +return err; + } + + ci =le16_to_cpu (comp.client_reg.client_id); +if (!ci) { +dev_err (dev,"%s: device returned null client_id\n" , __func__); +return -EIO ; + } + + padev->client_id = ci; + +return 0 ; +} + +/** +* pds_client_unregister - Disconnect the client from the device +* @padev: ptr to the client device info +* +* Disconnect the client from the core and with the DSC. +*/ +static int pds_client_unregister (struct pds_auxiliary_dev *padev) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev = padev->pf->dev; +struct pdsc *pf = padev->pf; +int err; + +if (pf->state) +return -ENXIO ; + + cmd.client_unreg.opcode =PDS_AQ_CMD_CLIENT_UNREG ; + cmd.client_unreg.client_id =cpu_to_le16 (padev->client_id); + + err =pdsc_adminq_post (pf, &pf->adminqcq, &cmd, &comp, false); +if (err) +dev_info (dev,"unregister dev_name %s failed, status %d: %pe\n" , +dev_name (&padev->aux_dev.dev), comp.status,ERR_PTR (err)); + + padev->client_id =0 ; + +return err; +} + +/** +* pds_client_adminq_cmd - Process an adminq request for the client +* @padev: ptr to the client device +* @req: ptr to buffer with request +* @req_len: length of actual struct used for request +* @resp: ptr to buffer where answer is to be copied +* @flags: optional flags from pds_core_adminq_flags +* +* Return: 0 on success, or +* negative for error +* +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +static int pds_client_adminq_cmd (struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags) +{ +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev = padev->pf->dev; +struct pdsc *pf = padev->pf; +size_t cp_len; +int err; + +dev_dbg (dev,"%s: %s opcode %d\n" , + __func__,dev_name (&padev->aux_dev.dev), req->opcode); + +if (pf->state) +return -ENXIO ; + +/* Wrap the client's request */ + cmd.client_request.opcode =PDS_AQ_CMD_CLIENT_CMD ; + cmd.client_request.client_id =cpu_to_le16 (padev->client_id); + cp_len =min_t (size_t, req_len,sizeof (cmd.client_request.client_cmd)); +memcpy (cmd.client_request.client_cmd, req, cp_len); + + err =pdsc_adminq_post (pf, &pf->adminqcq, &cmd, resp, !!(flags &PDS_AQ_FLAG_FASTPOLL )); +if (err && err != -EAGAIN ) +dev_info (dev,"client admin cmd failed: %pe\n" ,ERR_PTR (err)); + +return err; +} + +static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +}; +static void pdsc_auxbus_dev_release (struct device *dev) { struct pds_auxiliary_dev *padev =
@@ -15,7 +141,8 @@ static void pdsc_auxbus_dev_release(struct device *dev)static struct pds_auxiliary_dev *pdsc_auxbus_dev_register (struct pdsc *vf,struct pdsc *pf, -char *name) +char *name, +struct pds_core_ops *ops) {struct auxiliary_device *aux_dev;struct pds_auxiliary_dev *padev;
@@ -27,6 +154,7 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf, padev->vf = vf; padev->pf = pf; + padev->ops = ops; aux_dev = &padev->aux_dev; aux_dev->name = name;
@@ -94,7 +222,9 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf) pf->viftype_status[vt].enabled))continue ; - padev =pdsc_auxbus_dev_register (vf, pf, pf->viftype_status[vt].name); + padev =pdsc_auxbus_dev_register (vf, pf, + pf->viftype_status[vt].name, + &pds_core_ops); pf->vfs[vf->vf_id].padev = padev;/* We only support a single type per VF, so jump out here */
include/linux/pds/pds_auxbus.h +42 -0
49 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 11/13] pds_core: add the aux client API
Quoted 10 lines · Shannon Nelson
Add the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 134 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 42 +++++++ 2 files changed, 174 insertions(+), 2 deletions(-)
<...>
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +5
-0+static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +};
<...>
Quoted 30 lines
include/linux/pds/pds_auxbus.h +30
-0+/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .register() - register the client with the device +* padev: ptr to the client device info +* Register the client with the core and with the DSC. The core +* will fill in the client padrv->client_id for use in calls +* to the DSC AdminQ +*/ +int (*register_client)(struct pds_auxiliary_dev *padev); + +/* .unregister() - disconnect the client from the device +* padev: ptr to the client device info +* Disconnect the client from the core and with the DSC. +*/ +int (*unregister_client)(struct pds_auxiliary_dev *padev); + +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd
include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 57
Quoted 10 lines
include/linux/pds/pds_auxbus.h +10
-0+* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};
I don't expect to see any register/unregister AUX client code at all.
All clients are registered and unregistered through auxiliary_driver_register()/auxiliary_driver_unregister() calls and perform as standalone drivers.
Maybe client, register and unregister words means something else in this series..
Thanks
Quoted 4 lines
#endif /* _PDSC_AUXBUS_H_ */
Signature · 1 line
2.17.1
Re: [PATCH RFC v4 net-next 11/13] pds_core: add the aux client API
Quoted 65 lines · Leon Romanovsky
Quoted 10 lines · Shannon Nelson
Add the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 134 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 42 +++++++ 2 files changed, 174 insertions(+), 2 deletions(-)<...>
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +5
-0+static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +};<...>
Quoted 30 lines
include/linux/pds/pds_auxbus.h +30
-0+/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .register() - register the client with the device +* padev: ptr to the client device info +* Register the client with the core and with the DSC. The core +* will fill in the client padrv->client_id for use in calls +* to the DSC AdminQ +*/ +int (*register_client)(struct pds_auxiliary_dev *padev); + +/* .unregister() - disconnect the client from the device +* padev: ptr to the client device info +* Disconnect the client from the core and with the DSC. +*/ +int (*unregister_client)(struct pds_auxiliary_dev *padev); + +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 57
Quoted 10 lines
include/linux/pds/pds_auxbus.h +10
-0+* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};I don't expect to see any register/unregister AUX client code at all.
Quoted 6 lines
All clients are registered and unregistered through auxiliary_driver_register()/auxiliary_driver_unregister() calls and perform as standalone drivers.
Maybe client, register and unregister words means something else in this series..
Yeah, I'm not thrilled with the overlap in nomenclature either. In this case we're talking about the logic in the pds_vdpa module connecting to the services needed in the device FW, and getting a client_id from the FW that is used for tracking client context in the FW. Maybe these names can change to something like "fw_client_reg" and "fw_client_unreg" - would that make it more clear?
sln
Quoted 7 lines
Thanks
Quoted 4 lines
#endif /* _PDSC_AUXBUS_H_ */
--
2.17.1
Re: [PATCH RFC v4 net-next 11/13] pds_core: add the aux client API
Quoted 73 lines · Shannon Nelson
Quoted 65 lines · Leon Romanovsky
Quoted 10 lines · Shannon Nelson
Add the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 134 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 42 +++++++ 2 files changed, 174 insertions(+), 2 deletions(-)<...>
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +5
-0+static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +};<...>
Quoted 30 lines
include/linux/pds/pds_auxbus.h +30
-0+/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .register() - register the client with the device +* padev: ptr to the client device info +* Register the client with the core and with the DSC. The core +* will fill in the client padrv->client_id for use in calls +* to the DSC AdminQ +*/ +int (*register_client)(struct pds_auxiliary_dev *padev); + +/* .unregister() - disconnect the client from the device +* padev: ptr to the client device info +* Disconnect the client from the core and with the DSC. +*/ +int (*unregister_client)(struct pds_auxiliary_dev *padev); + +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 57
Quoted 10 lines
include/linux/pds/pds_auxbus.h +10
-0+* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};I don't expect to see any register/unregister AUX client code at all.
Quoted 6 lines
All clients are registered and unregistered through auxiliary_driver_register()/auxiliary_driver_unregister() calls and perform as standalone drivers.
Maybe client, register and unregister words means something else in this series..
Quoted 6 lines
Yeah, I'm not thrilled with the overlap in nomenclature either. In this case we're talking about the logic in the
pds_vdpamodule connecting to the services needed in the device FW, and getting aclient_idfrom the FW that is used for tracking client context in the FW. Maybe these names can change to something like "fw_client_reg" and "fw_client_unreg" - would that make it more clear?
I feel that such ops are not needed at all. Once you create aux devices (vdpa, eth, e.t.c), you would be able to connect only one driver with one such device. It means context is already known at that point. In addition, user controls if he/she wants aux specific devices by relevant devlink *_enable knob.
Thanks
Quoted 11 lines
sln
Quoted 7 lines
Thanks
Quoted 4 lines
#endif /* _PDSC_AUXBUS_H_ */
--
2.17.1
Re: [PATCH RFC v4 net-next 11/13] pds_core: add the aux client API
Quoted 81 lines · Leon Romanovsky
Quoted 73 lines · Shannon Nelson
Quoted 65 lines · Leon Romanovsky
Quoted 10 lines · Shannon Nelson
Add the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
Signed-off-by Shannon Nelson <[email protected]>--- drivers/net/ethernet/amd/pds_core/auxbus.c | 134 ++++++++++++++++++++- include/linux/pds/pds_auxbus.h | 42 +++++++ 2 files changed, 174 insertions(+), 2 deletions(-)<...>
Quoted 5 lines
drivers/net/ethernet/amd/pds_core/auxbus.c +5
-0+static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +};<...>
Quoted 30 lines
include/linux/pds/pds_auxbus.h +30
-0+/* +* ptrs to functions to be used by the client for core services +*/ +struct pds_core_ops { +/* .register() - register the client with the device +* padev: ptr to the client device info +* Register the client with the core and with the DSC. The core +* will fill in the client padrv->client_id for use in calls +* to the DSC AdminQ +*/ +int (*register_client)(struct pds_auxiliary_dev *padev); + +/* .unregister() - disconnect the client from the device +* padev: ptr to the client device info +* Disconnect the client from the core and with the DSC. +*/ +int (*unregister_client)(struct pds_auxiliary_dev *padev); + +/* .adminq_cmd() - process an adminq request for the client +* padev: ptr to the client device +* req: ptr to buffer with request +* req_len: length of actual struct used for request +* resp: ptr to buffer where answer is to be copied +* flags: optional flags defined by enum pds_core_adminq_flags +* and used for more flexible adminq behvior +* +* returns 0 on success, or +* negative for error +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd include/linux/pds/pds_auxbus.h struct pds_auxiliary_dev { · line 57
Quoted 10 lines
include/linux/pds/pds_auxbus.h +10
-0+* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +int (*adminq_cmd)(struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags); +};I don't expect to see any register/unregister AUX client code at all.
Quoted 6 lines
All clients are registered and unregistered through auxiliary_driver_register()/auxiliary_driver_unregister() calls and perform as standalone drivers.
Maybe client, register and unregister words means something else in this series..
Quoted 6 lines
Yeah, I'm not thrilled with the overlap in nomenclature either. In this case we're talking about the logic in the
pds_vdpamodule connecting to the services needed in the device FW, and getting aclient_idfrom the FW that is used for tracking client context in the FW. Maybe these names can change to something like "fw_client_reg" and "fw_client_unreg" - would that make it more clear?
Quoted 7 lines
I feel that such ops are not needed at all. Once you create aux devices (vdpa, eth,
e.t.c), you would be able to connect only one driver with one such device. It means context is already known at that point. In addition, user controls if he/she wants aux specific devices by relevant devlink*_enableknob.Thanks
As this code matures here, as well as the internal gyrations happening, this makes more and more sense. Some of the original work was there for greater flexibility in what we might need "soon", but much has gone away. This is one of those things that we can whittle down. The registration is still needed to tell the FW about the client, but can be moved out of the clients' responsibility and into the VF startup.
Thanks,
sln
RFC PATCH v4 12/13 pds_core: publish events to the clients
In reply to Shannon NelsonWhen the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to the clients.
--- drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++ drivers/net/ethernet/amd/pds_core/core.c | 34 ++++++++++++++++++++++ include/linux/pds/pds_core.h | 3 ++ 3 files changed, 39 insertions(+)
drivers/net/ethernet/amd/pds_core/adminq.c +2 -0
@@ -28,11 +28,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (pdsc,PDS_EVENT_LINK_CHANGE , comp);break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_notify (pdsc,PDS_EVENT_RESET , comp);break ;case PDS_EVENT_XCVR :
drivers/net/ethernet/amd/pds_core/core.c +34 -0
@@ -7,6 +7,25 @@#include <linux/pds/pds_adminq.h> +static BLOCKING_NOTIFIER_HEAD (pds_notify_chain); + +int pdsc_register_notify (struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_register_notify); + +void pdsc_unregister_notify (struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL (pdsc_unregister_notify); + +void pdsc_notify (struct pdsc *pdsc,unsigned long event,void *data) +{ +blocking_notifier_call_chain (&pds_notify_chain, event, data); +} +void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;
@@ -507,6 +526,11 @@ void pdsc_stop(struct pdsc *pdsc)static void pdsc_fw_down (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =0 , + }; +mutex_lock (&pdsc->config_lock);if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) {
@@ -515,6 +539,9 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +/* Notify clients of fw_down */ +pdsc_notify (pdsc,PDS_EVENT_RESET , &reset_event); +pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );
@@ -523,6 +550,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)static void pdsc_fw_up (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =1 , + };int err;mutex_lock (&pdsc->config_lock);
@@ -543,6 +574,9 @@ static void pdsc_fw_up(struct pdsc *pdsc)mutex_unlock (&pdsc->config_lock); +/* Notify clients of fw_up */ +pdsc_notify (pdsc,PDS_EVENT_RESET , &reset_event); +return ; err_out:
include/linux/pds/pds_core.h +3 -0
@@ -305,6 +305,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_register_notify (struct notifier_block *nb); +void pdsc_unregister_notify (struct notifier_block *nb); +void pdsc_notify (struct pdsc *pdsc,unsigned long event,void *data);int pdsc_auxbus_dev_add_vf (struct pdsc *pdsc_vf,struct pdsc *pdsc_pf);int pdsc_auxbus_dev_del_vf (struct pdsc *pdsc_vf,struct pdsc *pdsc_pf);
Signature · 1 line
2.17.1
RFC PATCH v4 13/13 pds_core: Kconfig and pds_core.rst
In reply to Shannon NelsonDocumentation and Kconfig hook for building the driver.
--- .../device_drivers/ethernet/amd/pds_core.rst | 133 ++++++++++++++++++ .../device_drivers/ethernet/index.rst | 1 + drivers/net/ethernet/amd/Kconfig | 12 ++ drivers/net/ethernet/amd/Makefile | 1 + 4 files changed, 147 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst new file +133 -0
@@ -0,0 +1,133 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +======================================================== +Linux Driver for the AMD/Pensando(R) DSC adapter family +======================================================== + +AMD/Pensando Linux Core driver +Copyright(c) 2023 Advanced Micro Devices, Inc + +Identifying the Adapter +======================= + +To find if one or more AMD/Pensando PCI Core devices are installed on the +host, check for the PCI devices:: + + # lspci -d 1dd8:100c + b5:00.0 Processing accelerators: Pensando Systems Device 100c + b6:00.0 Processing accelerators: Pensando Systems Device 100c + +If such devices are listed as above, then the pds_core.ko driver should find +and configure them for use. There should be log entries in the kernel +messages such as these:: + + $ dmesg | grep pds_core + pds_core 0000:b5:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b5:00.0: FW: 1.60.0-73 + pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b6:00.0: FW: 1.60.0-73 + +Driver and firmware version information can be gathered with devlink:: + + $ devlink dev info pci/0000:b5:00.0 + pci/0000:b5:00.0: + driver pds_core + serial_number FLM18420073 + versions: + fixed: + asic.id 0x0 + asic.rev 0x0 + running: + fw 1.51.0-73 + stored: + fw.goldfw 1.15.9-C-22 + fw.mainfwa 1.60.0-73 + fw.mainfwb 1.60.0-57 + + +Info versions +============= + +The ``pds_core`` driver reports the following versions + +.. list-table:: devlink info versions implemented + :widths: 5 5 90 + + * - Name + - Type + - Description + * - ``fw`` + - running + - Version of firmware running on the device + * - ``fw.goldfw`` + - stored + - Version of firmware stored in the goldfw slot + * - ``fw.mainfwa`` + - stored + - Version of firmware stored in the mainfwa slot + * - ``fw.mainfwb`` + - stored + - Version of firmware stored in the mainfwb slot + * - ``asic.id`` + - fixed + - The ASIC type for this device + * - ``asic.rev`` + - fixed + - The revision of the ASIC for this device + + +Parameters +========== + +The ``pds_core`` driver implements the following generic +parameters for controlling the functionality to be made available +as auxiliary_bus devices. + +.. list-table:: Generic parameters implemented + :widths: 5 5 8 82 + + * - Name + - Mode + - Type + - Description + * - ``enable_vnet`` + - runtime + - Boolean + - Enables vDPA functionality through an auxiliary_bus device + + +Firmware Management +=================== + +The ``flash`` command can update a the DSC firmware. The downloaded firmware +will be saved into either of firmware bank 1 or bank 2, whichever is not +currrently in use, and that bank will be then selected for the next boot. + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> AMD devices + -> AMD/Pensando Ethernet PDS_CORE Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by AMD/Pensando personnel:: + + [email protected] + +For more specific support needs, please use the AMD/Pensando driver support +email:: + + [email protected]
Documentation/networking/device_drivers/ethernet/index.rst +1 -0
@@ -13,6 +13,7 @@ Contents: 3com/3c509 3com/vortex amazon/ena + amd/pds_core altera/altera_tse aquantia/atlantic chelsio/cxgb
drivers/net/ethernet/amd/Kconfig +12 -0
@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC bool default n +config PDS_CORE + tristate "AMD/Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the AMD/Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + endif # NET_VENDOR_AMD
drivers/net/ethernet/amd/Makefile +1 -0
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o obj-$(CONFIG_SUN3LANCE) += sun3lance.o obj-$(CONFIG_SUNLANCE) += sunlance.o obj-$(CONFIG_AMD_XGBE) += xgbe/ +obj-$(CONFIG_PDS_CORE) += pds_core/
Signature · 1 line
2.17.1