Skip to contentSkip to conversation

pds_core driver

PATCH Superseded Shannon Nelson 4 participants 27 messages Feb 17, 2023 – Feb 25, 2023

Subsystems (5)

v3 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v3 cover pds_core driver

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

To keep this patchset to a manageable size, the pds_vdpa and pds_vfio drivers have been split out into their own patchsets to be reviewed separately.

Detail:
-------
AMD/Pensando is making available a new set of devices for supporting vDPA, VFio, and potentially other features in the Distributed Services Card (DSC). These features are implemented through a PF that serves as a Core device for controlling and configuring its VF devices. These VF devices have separate drivers that use the auxiliary_bus to work through the Core device as the control path.

Currently, the DSC supports standard ethernet operations using the ionic driver. This is not replaced by the Core-based devices - these new devices are in addition to the existing Ethernet device. Typical DSC configurations will include both PDS devices and Ionic Eth devices.

The Core device is a new PCI PF device managed by a new driver 'pds_core'. It sets up auxiliary_bus devices for each VF for communicating with the drivers for the VF devices. The VFs may be for VFio or vDPA, and other services in the future; these VF types are selected as part of the DSC internal FW configurations, which is out of the scope of this patchset. The Core device sets up devlink parameters for enabling available feature sets.

Once a feature set is enabled in the core device, auxiliary_bus devices are created for each VF that supports the feature. These auxiliary_bus devices are named by their feature plus VF PCI bdf so that the auxiliary device driver can find its related VF PCI driver instance. The VF's driver then connects to and uses this auxiliary_device to do control path configuration of the feature through the PF device.

A cheap ASCII diagram of a vDPA instance looks something like this and can then be used with the vdpa kernel module to provide devices for virtio_vdpa kernel module for host interfaces, or vhost_vdpa kernel module for interfaces exported into your favorite VM.

                                  ,----------.
                                  |   vdpa   |
                                  '----------'
                                       |
                                     vdpa_dev
                                    ctl   data
                                     |     ||
           pds_core.vDPA.2305 <---+  |     ||
                   |              |  |     ||
       netdev      |              |  |     ||
          |        |              |  |     ||
         .------------.         .------------.
         |  pds_core  |         |  pds_vdpa  |
         '------------'         '------------'
               ||                     ||
             09:00.0                09:00.1
== PCI =========================================================
               ||                     ||
          .----------.           .----------.
    ,-----|    PF    |-----------|    VF    |-------,
    |     '----------'           -----------'       |
    |                     DSC                       |
    |                                               |
    -------------------------------------------------

Changes:
v3:
- changed names from "pensando" to "amd" and updated copyright strings
- dropped the DEVLINK_PARAM_GENERIC_ID_FW_BANK for future development
- changed the auxiliary device creation to be triggered by the PCI bus event BOUND_DRIVER, and torn down at UNBIND_DRIVER in order to properly handle users using the sysfs bind/unbind functions
- dropped some noisy log messages
- rebased to current net-next

RFC to v2:
https://lore.kernel.org/netdev/[email protected]/
- added separate devlink param patches for DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION and DEVLINK_PARAM_GENERIC_ID_FW_BANK, and dropped the driver specific implementations
- updated descriptions for the new devlink parameters
- dropped netdev support
- dropped vDPA patches, will followup later
- separated fw update and fw bank select into their own patches

RFC:
https://lore.kernel.org/netdev/[email protected]/

Shannon Nelson (14):
devlink: add enable_migration parameter
pds_core: initial framework for pds_core driver
pds_core: add devcmd device interfaces
pds_core: health timer and workqueue
pds_core: set up device and adminq
pds_core: Add adminq processing and commands
pds_core: add FW update feature to devlink
pds_core: set up the VIF definitions and defaults
pds_core: initial VF configuration
pds_core: add auxiliary_bus devices
pds_core: devlink params for enabling VIF support
pds_core: add the aux client API
pds_core: publish events to the clients
pds_core: Kconfig and pds_core.rst

 .../device_drivers/ethernet/amd/pds_core.rst  | 150 ++++
 .../device_drivers/ethernet/index.rst         |   1 +
 .../networking/devlink/devlink-params.rst     |   3 +
 drivers/net/ethernet/amd/Kconfig              |  12 +
 drivers/net/ethernet/amd/Makefile             |   1 +
 drivers/net/ethernet/amd/pds_core/Makefile    |  14 +
 drivers/net/ethernet/amd/pds_core/adminq.c    | 299 ++++++++
 drivers/net/ethernet/amd/pds_core/auxbus.c    | 303 +++++++++
 drivers/net/ethernet/amd/pds_core/core.c      | 599 ++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.h      | 318 +++++++++
 drivers/net/ethernet/amd/pds_core/debugfs.c   | 262 +++++++
 drivers/net/ethernet/amd/pds_core/dev.c       | 358 ++++++++++
 drivers/net/ethernet/amd/pds_core/devlink.c   | 199 ++++++
 drivers/net/ethernet/amd/pds_core/fw.c        | 192 ++++++
 drivers/net/ethernet/amd/pds_core/main.c      | 435 ++++++++++++
 include/linux/pds/pds_adminq.h                | 643 ++++++++++++++++++
 include/linux/pds/pds_auxbus.h                |  85 +++
 include/linux/pds/pds_common.h                |  93 +++
 include/linux/pds/pds_core_if.h               | 580 ++++++++++++++++
 include/linux/pds/pds_intr.h                  | 161 +++++
 include/net/devlink.h                         |   4 +
 net/devlink/leftover.c                        |   5 +
 22 files changed, 4717 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
 create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
 create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
 create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
 create mode 100644 include/linux/pds/pds_adminq.h
 create mode 100644 include/linux/pds/pds_auxbus.h
 create mode 100644 include/linux/pds/pds_common.h
 create mode 100644 include/linux/pds/pds_core_if.h
 create mode 100644 include/linux/pds/pds_intr.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v3 1/14 devlink: add enable_migration parameter

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

  $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
  $ devlink dev param show pci/0000:07:00.0 name enable_migration
  pci/0000:07:00.0:
    name enable_migration type generic
      values:
        cmode runtime value true
---
 Documentation/networking/devlink/devlink-params.rst | 3 +++
 include/net/devlink.h                               | 4 ++++
 net/devlink/leftover.c                              | 5 +++++
 3 files changed, 12 insertions(+)
Documentation/networking/devlink/devlink-params.rst +3 -0
index 4e01dc32bc08..900216fd56b9 100644
@@ -137,3 +137,6 @@ own name.
    * - ``event_eq_size``
      - u32
      - Control the size of asynchronous control events EQ.
+   * - ``enable_migration``
+     - Boolean
+     - Enable/disable support for live migration in the devlink device.
include/net/devlink.h +4 -0
index 6a942e70e451..bd85a3ff3774 100644
@@ -513,6 +513,7 @@ enum devlink_param_generic_id {
 	DEVLINK_PARAM_GENERIC_ID_ENABLE_IWARP,
 	DEVLINK_PARAM_GENERIC_ID_IO_EQ_SIZE,
 	DEVLINK_PARAM_GENERIC_ID_EVENT_EQ_SIZE,
+	DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION,
 
 	/* add new param generic ids above here*/
 	__DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -571,6 +572,9 @@ enum devlink_param_generic_id {
 #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME "event_eq_size"
 #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE DEVLINK_PARAM_TYPE_U32
 
+#define DEVLINK_PARAM_GENERIC_ENABLE_MIGRATION_NAME "enable_migration"
+#define DEVLINK_PARAM_GENERIC_ENABLE_MIGRATION_TYPE DEVLINK_PARAM_TYPE_BOOL
+
 #define DEVLINK_PARAM_GENERIC(_id, _cmodes, _get, _set, _validate)	\
 {									\
 	.id = DEVLINK_PARAM_GENERIC_ID_##_id,				\
net/devlink/leftover.c +5 -0
index 3569706c49e1..2e56dfadd37b 100644
@@ -3923,6 +3923,11 @@ static const struct devlink_param devlink_param_generic[] = {
 		.name = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME,
 		.type = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE,
 	},
+	{
+		.id = DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION,
+		.name = DEVLINK_PARAM_GENERIC_ENABLE_MIGRATION_NAME,
+		.type = DEVLINK_PARAM_GENERIC_ENABLE_MIGRATION_TYPE,
+	},
 };
 
 static int devlink_param_generic_verify(const struct devlink_param *param)
Signature · 1 line
2.17.1
maintainer
#2 lore raw

Re: [PATCH v3 net-next 01/14] devlink: add enable_migration parameter

Quoted 7 lines · Fri, Feb 17, 2023 at 11:55:45PM CET, [email protected] wrote:

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

Quoted 8 lines
 $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
 $ devlink dev param show pci/0000:07:00.0 name enable_migration
 pci/0000:07:00.0:
   name enable_migration type generic
     values:
       cmode runtime value true

Could you please elaborate why exactly is this needed?

From my perspective, the migration capability is something that is related to the actual function (VF/SF).

When instantiating/configuring SF/VF, the admin ask for the particular function to support live migration. We have port function caps now, which is exactly where this makes sense.

See DEVLINK_PORT_FN_CAP_MIGRATABLE.

#3 lore raw

Re: [PATCH v3 net-next 01/14] devlink: add enable_migration parameter

Quoted 19 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 17, 2023 at 11:55:45PM CET, [email protected] wrote:

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

Quoted 8 lines
  $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
  $ devlink dev param show pci/0000:07:00.0 name enable_migration
  pci/0000:07:00.0:
    name enable_migration type generic
      values:
        cmode runtime value true

Could you please elaborate why exactly is this needed?

Quoted 8 lines

From my perspective, the migration capability is something that
is related to the actual function (VF/SF).

When instantiating/configuring SF/VF, the admin ask for the particular function to support live migration. We have port function caps now, which is exactly where this makes sense.

See DEVLINK_PORT_FN_CAP_MIGRATABLE.

Hi Jiri,

Thanks for your questions. My apologies for not getting your name into the To: list – a late Friday afternoon miss.

This enable_migration flag is intended to be similar to the enable_vnet, enable_rdma, and similar existing parameters that are used by other core devices.

Our pds_core device can be used to support several features (currently VFio and vDPA), and this gives the user a way to control how many of the features are made available in any particular configuration. This is to be enabled to turn on support for our pds_vfio client devices as a whole, not individually port-by-port. I understand FN_CAP_MIGRATABLE to be applied to an individual devlink port, which could be used in conjunction with this once the general feature is enable in pds_core.

Thanks,
sln

maintainer
#4 lore raw

Re: [PATCH v3 net-next 01/14] devlink: add enable_migration parameter

Quoted 38 lines · Tue, Feb 21, 2023 at 12:54:25AM CET, [email protected] wrote:
Quoted 19 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 17, 2023 at 11:55:45PM CET, [email protected] wrote:

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

Quoted 8 lines
  $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
  $ devlink dev param show pci/0000:07:00.0 name enable_migration
  pci/0000:07:00.0:
    name enable_migration type generic
      values:
        cmode runtime value true

Could you please elaborate why exactly is this needed?

Quoted 8 lines

From my perspective, the migration capability is something that
is related to the actual function (VF/SF).

When instantiating/configuring SF/VF, the admin ask for the particular function to support live migration. We have port function caps now, which is exactly where this makes sense.

See DEVLINK_PORT_FN_CAP_MIGRATABLE.

Hi Jiri,

Thanks for your questions. My apologies for not getting your name into the
To: list – a late Friday afternoon miss.

This enable_migration flag is intended to be similar to the enable_vnet, enable_rdma, and similar existing parameters that are used by other core devices.

Quoted 7 lines

Our pds_core device can be used to support several features (currently VFio and vDPA), and this gives the user a way to control how many of the features are made available in any particular configuration. This is to be enabled to turn on support for our pds_vfio client devices as a whole, not individually port-by-port. I understand FN_CAP_MIGRATABLE to be applied to an individual devlink port, which could be used in conjunction with this once the general feature is enable in pds_core.

Okay, that sounds legit. Could yout please extend the patch description with this? Thanks!

Quoted 3 lines

Thanks,
sln

#5 lore raw

Re: [PATCH v3 net-next 01/14] devlink: add enable_migration parameter

Quoted 47 lines · Jiri Pirko
Quoted 38 lines · Tue, Feb 21, 2023 at 12:54:25AM CET, [email protected] wrote:
Quoted 19 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 17, 2023 at 11:55:45PM CET, [email protected] wrote:

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

Quoted 8 lines
   $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
   $ devlink dev param show pci/0000:07:00.0 name enable_migration
   pci/0000:07:00.0:
     name enable_migration type generic
       values:
         cmode runtime value true

Could you please elaborate why exactly is this needed?

Quoted 8 lines

From my perspective, the migration capability is something that
is related to the actual function (VF/SF).

When instantiating/configuring SF/VF, the admin ask for the particular function to support live migration. We have port function caps now, which is exactly where this makes sense.

See DEVLINK_PORT_FN_CAP_MIGRATABLE.

Hi Jiri,

Thanks for your questions. My apologies for not getting your name into the
To: list – a late Friday afternoon miss.

This enable_migration flag is intended to be similar to the enable_vnet, enable_rdma, and similar existing parameters that are used by other core devices.

Quoted 7 lines

Our pds_core device can be used to support several features (currently VFio and vDPA), and this gives the user a way to control how many of the features are made available in any particular configuration. This is to be enabled to turn on support for our pds_vfio client devices as a whole, not individually port-by-port. I understand FN_CAP_MIGRATABLE to be applied to an individual devlink port, which could be used in conjunction with this once the general feature is enable in pds_core.

Quoted 2 lines

Okay, that sounds legit. Could yout please extend the patch description with this? Thanks!

Will do - thanks.
sln

maintainer
#6 lore raw

Re: [PATCH v3 net-next 01/14] devlink: add enable_migration parameter

In reply to Shannon Nelson
Quoted 14 lines · Fri, Feb 17, 2023 at 11:55:45PM CET, [email protected] wrote:

Add a new device generic parameter to enable/disable support for live migration in the devlink device. This is intended primarily for a core device that supports other ports/VFs/SFs. Those dependent ports may need their own migratable parameter for individual enable/disable control.

Examples:

 $ devlink dev param set pci/0000:07:00.0 name enable_migration value true cmode runtime
 $ devlink dev param show pci/0000:07:00.0 name enable_migration
 pci/0000:07:00.0:
   name enable_migration type generic
     values:
       cmode runtime value true
Quoted 5 lines

---

Documentation/networking/devlink/devlink-params.rst | 3 +++
include/net/devlink.h                               | 4 ++++
net/devlink/leftover.c                              | 5 +++++

Why you didn't use get_maintainers.pl script for the cc list?

#7 lore raw

PATCH v3 2/14 pds_core: initial framework for pds_core driver

In reply to Shannon Nelson

This is the initial PCI driver framework for the new pds_core device driver and its family of client drivers. This does the very basics of registering for the new PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.

---
 drivers/net/ethernet/amd/pds_core/Makefile  |   9 +
 drivers/net/ethernet/amd/pds_core/core.h    |  68 +++
 drivers/net/ethernet/amd/pds_core/debugfs.c |  47 ++
 drivers/net/ethernet/amd/pds_core/devlink.c |  46 ++
 drivers/net/ethernet/amd/pds_core/main.c    | 263 +++++++++
 include/linux/pds/pds_common.h              |  13 +
 include/linux/pds/pds_core_if.h             | 580 ++++++++++++++++++++
 7 files changed, 1026 insertions(+)
 create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
 create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
 create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
 create mode 100644 include/linux/pds/pds_common.h
 create mode 100644 include/linux/pds/pds_core_if.h
drivers/net/ethernet/amd/pds_core/Makefile new file +9 -0
new file mode 100644
index 000000000000..b4cc4b242e44
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Advanced Micro Devices, Inc.
+
+obj-$(CONFIG_PDS_CORE) := pds_core.o
+
+pds_core-y := main.o \
+	      devlink.o
+
+pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.h new file +68 -0

68 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/debugfs.c new file +47 -0

47 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/devlink.c new file +46 -0

46 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/main.c new file +263 -0

263 lines not on this page · raw message

include/linux/pds/pds_common.h new file +13 -0
new file mode 100644
index 000000000000..023da482e320
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
+/* Copyright (c) 2023 Advanced Micro Devices, Inc. */
+
+#ifndef _PDS_COMMON_H_
+#define _PDS_COMMON_H_
+
+#define PDS_CORE_DRV_NAME			"pds_core"
+
+/* the device's internal addressing uses up to 52 bits */
+#define PDS_CORE_ADDR_LEN	52
+#define PDS_CORE_ADDR_MASK	(BIT_ULL(PDS_ADDR_LEN) - 1)
+
+#endif /* _PDS_COMMON_H_ */
include/linux/pds/pds_core_if.h new file +580 -0

580 lines not on this page · raw message

Signature · 1 line
2.17.1
#8 lore raw

PATCH v3 3/14 pds_core: add devcmd device interfaces

In reply to Shannon Nelson

The devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.

---
 drivers/net/ethernet/amd/pds_core/Makefile  |   4 +-
 drivers/net/ethernet/amd/pds_core/core.c    |  43 +++
 drivers/net/ethernet/amd/pds_core/core.h    |  55 +++
 drivers/net/ethernet/amd/pds_core/debugfs.c |  67 ++++
 drivers/net/ethernet/amd/pds_core/dev.c     | 355 ++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/main.c    |  30 ++
 include/linux/pds/pds_common.h              |  63 ++++
 include/linux/pds/pds_intr.h                | 161 +++++++++
 8 files changed, 777 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
 create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
 create mode 100644 include/linux/pds/pds_intr.h
drivers/net/ethernet/amd/pds_core/Makefile +3 -1
index b4cc4b242e44..eaca8557ba66 100644
@@ -4,6 +4,8 @@
 obj-$(CONFIG_PDS_CORE) := pds_core.o
 
 pds_core-y := main.o \
-	      devlink.o
+	      devlink.o \
+	      dev.o \
+	      core.o
 
 pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.c new file +43 -0

43 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/core.h +55 -0

97 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/debugfs.c +67 -0

71 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/dev.c new file +355 -0

355 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/main.c +30 -0
index b9f29b13d6ba..8573e119087a 100644
@@ -189,6 +189,15 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto err_out_pci_disable_device;
 
+	/* PDS device setup */
+	mutex_init(&pdsc->devcmd_lock);
+	mutex_init(&pdsc->config_lock);
+
+	mutex_lock(&pdsc->config_lock);
+	err = pdsc_setup(pdsc, PDSC_SETUP_INIT);
+	if (err)
+		goto err_out_unmap_bars;
+
 	/* publish devlink device */
 	err = pdsc_dl_register(pdsc);
 	if (err) {
@@ -196,10 +205,21 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_out;
 	}
 
+	mutex_unlock(&pdsc->config_lock);
+
+	pdsc->fw_generation = PDS_CORE_FW_STS_F_GENERATION &
+			      ioread8(&pdsc->info_regs->fw_status);
+
 	clear_bit(PDSC_S_INITING_DRIVER, &pdsc->state);
 	return 0;
 
 err_out:
+	pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
+err_out_unmap_bars:
+	mutex_unlock(&pdsc->config_lock);
+	mutex_destroy(&pdsc->config_lock);
+	mutex_destroy(&pdsc->devcmd_lock);
+	pci_free_irq_vectors(pdev);
 	pci_clear_master(pdev);
 	pdsc_unmap_bars(pdsc);
 	pci_release_regions(pdev);
@@ -224,10 +244,20 @@ static void pdsc_remove(struct pci_dev *pdev)
 	 */
 	pdsc_dl_unregister(pdsc);
 
+	/* Now we can lock it up and tear it down */
+	mutex_lock(&pdsc->config_lock);
+	set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
+
 	/* Device teardown */
+	pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
+	pdsc_debugfs_del_dev(pdsc);
+	mutex_unlock(&pdsc->config_lock);
+	mutex_destroy(&pdsc->config_lock);
+	mutex_destroy(&pdsc->devcmd_lock);
 	ida_free(&pdsc_pf_ida, pdsc->id);
 
 	/* PCI teardown */
+	pci_free_irq_vectors(pdev);
 	pci_clear_master(pdev);
 	pdsc_unmap_bars(pdsc);
 	pci_release_regions(pdev);
include/linux/pds/pds_common.h +63 -0

67 lines not on this page · raw message

include/linux/pds/pds_intr.h new file +161 -0

161 lines not on this page · raw message

Signature · 1 line
2.17.1
#9 lore raw

PATCH v3 4/14 pds_core: health timer and workqueue

In reply to Shannon Nelson

Add in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.

---
 drivers/net/ethernet/amd/pds_core/core.c | 60 ++++++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.h |  9 ++++
 drivers/net/ethernet/amd/pds_core/dev.c  |  3 ++
 drivers/net/ethernet/amd/pds_core/main.c | 50 ++++++++++++++++++++
 4 files changed, 122 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +60 -0
index 0efc03be9abc..5dfa72001e27 100644
@@ -41,3 +41,63 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
 
 	set_bit(PDSC_S_FW_DEAD, &pdsc->state);
 }
+
+static void pdsc_fw_down(struct pdsc *pdsc)
+{
+	mutex_lock(&pdsc->config_lock);
+
+	if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+		dev_err(pdsc->dev, "%s: already happening\n", __func__);
+		mutex_unlock(&pdsc->config_lock);
+		return;
+	}
+
+	pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
+
+	mutex_unlock(&pdsc->config_lock);
+}
+
+static void pdsc_fw_up(struct pdsc *pdsc)
+{
+	int err;
+
+	mutex_lock(&pdsc->config_lock);
+
+	if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+		dev_err(pdsc->dev, "%s: fw not dead\n", __func__);
+		mutex_unlock(&pdsc->config_lock);
+		return;
+	}
+
+	err = pdsc_setup(pdsc, PDSC_SETUP_RECOVERY);
+	if (err)
+		goto err_out;
+
+	mutex_unlock(&pdsc->config_lock);
+
+	return;
+
+err_out:
+	pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
+	mutex_unlock(&pdsc->config_lock);
+}
+
+void pdsc_health_thread(struct work_struct *work)
+{
+	struct pdsc *pdsc = container_of(work, struct pdsc, health_work);
+	bool healthy;
+
+	healthy = pdsc_is_fw_good(pdsc);
+	dev_dbg(pdsc->dev, "%s: health %d fw_status %#02x fw_heartbeat %d\n",
+		__func__, healthy, pdsc->fw_status, pdsc->last_hb);
+
+	if (test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+		if (healthy)
+			pdsc_fw_up(pdsc);
+	} else {
+		if (!healthy)
+			pdsc_fw_down(pdsc);
+	}
+
+	pdsc->fw_generation = pdsc->fw_status & PDS_CORE_FW_STS_F_GENERATION;
+}
drivers/net/ethernet/amd/pds_core/core.h +9 -0
index 48618bdf0921..153889f98c25 100644
@@ -12,6 +12,8 @@
 #include <linux/pds/pds_intr.h>
 
 #define PDSC_DRV_DESCRIPTION	"AMD/Pensando Core PF Driver"
+
+#define PDSC_WATCHDOG_SECS	5
 #define PDSC_TEARDOWN_RECOVERY	false
 #define PDSC_TEARDOWN_REMOVING	true
 #define PDSC_SETUP_RECOVERY	false
@@ -64,12 +66,17 @@ struct pdsc {
 	u8 fw_generation;
 	unsigned long last_fw_time;
 	u32 last_hb;
+	struct timer_list wdtimer;
+	unsigned int wdtimer_period;
+	struct work_struct health_work;
 
 	struct pdsc_devinfo dev_info;
 	struct pds_core_dev_identity dev_ident;
 	unsigned int nintrs;
 	struct pdsc_intr_info *intr_info;	/* array of nintrs elements */
 
+	struct workqueue_struct *wq;
+
 	unsigned int devcmd_timeout;
 	struct mutex devcmd_lock;	/* lock for dev_cmd operations */
 	struct mutex config_lock;	/* lock for configuration operations */
@@ -82,6 +89,7 @@ struct pdsc {
 	u64 __iomem *kern_dbpage;
 };
 
+void pdsc_queue_health_check(struct pdsc *pdsc);
 void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num);
 
 struct pdsc *pdsc_dl_alloc(struct device *dev);
@@ -119,5 +127,6 @@ int pdsc_dev_init(struct pdsc *pdsc);
 
 int pdsc_setup(struct pdsc *pdsc, bool init);
 void pdsc_teardown(struct pdsc *pdsc, bool removing);
+void pdsc_health_thread(struct work_struct *work);
 
 #endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/dev.c +3 -0
index 1aef124a38cb..0f5a65e0269b 100644
@@ -181,6 +181,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
 	err = pdsc_devcmd_wait(pdsc, max_seconds);
 	memcpy_fromio(comp, &pdsc->cmd_regs->comp, sizeof(*comp));
 
+	if (err == -ENXIO || err == -ETIMEDOUT)
+		pdsc_queue_health_check(pdsc);
+
 	return err;
 }
 
drivers/net/ethernet/amd/pds_core/main.c +50 -0

89 lines not on this page · raw message

Signature · 1 line
2.17.1
#10 lore raw

PATCH v3 5/14 pds_core: set up device and adminq

In reply to Shannon Nelson

Set up the basic adminq and notifyq services. These are used mostly by the client drivers for feature configuration.

---
 drivers/net/ethernet/amd/pds_core/core.c    | 444 +++++++++++++-
 drivers/net/ethernet/amd/pds_core/core.h    | 149 +++++
 drivers/net/ethernet/amd/pds_core/debugfs.c | 125 ++++
 drivers/net/ethernet/amd/pds_core/devlink.c |  55 ++
 drivers/net/ethernet/amd/pds_core/main.c    |  14 +
 include/linux/pds/pds_adminq.h              | 641 ++++++++++++++++++++
 6 files changed, 1425 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/pds/pds_adminq.h
drivers/net/ethernet/amd/pds_core/core.c +441 -3
index 5dfa72001e27..f0bccc6a0f3a 100644
@@ -10,8 +10,368 @@
 
 #include "core.h"
 
+#include <linux/pds/pds_adminq.h>
+
+void pdsc_work_thread(struct work_struct *work)
+{
+	/* stub */
+}
+
+irqreturn_t pdsc_adminq_isr(int irq, void *data)
+{
+	/* stub */
+	return IRQ_HANDLED;
+}
+
+void pdsc_intr_free(struct pdsc *pdsc, int index)
+{
+	struct pdsc_intr_info *intr_info;
+
+	if (index >= pdsc->nintrs || index < 0) {
+		WARN(true, "bad intr index %d\n", index);
+		return;
+	}
+
+	intr_info = &pdsc->intr_info[index];
+	if (!intr_info->vector)
+		return;
+	dev_dbg(pdsc->dev, "%s: idx %d vec %d name %s\n",
+		__func__, index, intr_info->vector, intr_info->name);
+
+	pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET);
+	pds_core_intr_clean(&pdsc->intr_ctrl[index]);
+
+	devm_free_irq(pdsc->dev, intr_info->vector, intr_info->data);
+
+	memset(intr_info, 0, sizeof(*intr_info));
+}
+
+int pdsc_intr_alloc(struct pdsc *pdsc, char *name,
+		    irq_handler_t handler, void *data)
+{
+	struct pdsc_intr_info *intr_info;
+	unsigned int index;
+	int err;
+
+	/* Find the first available interrupt */
+	for (index = 0; index < pdsc->nintrs; index++)
+		if (!pdsc->intr_info[index].vector)
+			break;
+	if (index >= pdsc->nintrs) {
+		dev_warn(pdsc->dev, "%s: no intr, index=%d nintrs=%d\n",
+			 __func__, index, pdsc->nintrs);
+		return -ENOSPC;
+	}
+
+	pds_core_intr_clean_flags(&pdsc->intr_ctrl[index],
+				  PDS_CORE_INTR_CRED_RESET_COALESCE);
+
+	intr_info = &pdsc->intr_info[index];
+
+	intr_info->index = index;
+	intr_info->data = data;
+	strscpy(intr_info->name, name, sizeof(intr_info->name));
+
+	/* Get the OS vector number for the interrupt */
+	err = pci_irq_vector(pdsc->pdev, index);
+	if (err < 0) {
+		dev_err(pdsc->dev, "failed to get intr vector index %d: %pe\n",
+			index, ERR_PTR(err));
+		goto err_out_free_intr;
+	}
+	intr_info->vector = err;
+
+	/* Init the device's intr mask */
+	pds_core_intr_clean(&pdsc->intr_ctrl[index]);
+	pds_core_intr_mask_assert(&pdsc->intr_ctrl[index], 1);
+	pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET);
+
+	/* Register the isr with a name */
+	err = devm_request_irq(pdsc->dev, intr_info->vector,
+			       handler, 0, intr_info->name, data);
+	if (err) {
+		dev_err(pdsc->dev, "failed to get intr irq vector %d: %pe\n",
+			intr_info->vector, ERR_PTR(err));
+		goto err_out_free_intr;
+	}
+
+	return index;
+
+err_out_free_intr:
+	pdsc_intr_free(pdsc, index);
+	return err;
+}
+
+static void pdsc_qcq_intr_free(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+	if (!(qcq->flags & PDS_CORE_QCQ_F_INTR) ||
+	    qcq->intx == PDS_CORE_INTR_INDEX_NOT_ASSIGNED)
+		return;
+
+	pdsc_intr_free(pdsc, qcq->intx);
+	qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED;
+}
+
+static int pdsc_qcq_intr_alloc(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+	char name[PDSC_INTR_NAME_MAX_SZ];
+	int index;
+
+	if (!(qcq->flags & PDS_CORE_QCQ_F_INTR)) {
+		qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED;
+		return 0;
+	}
+
+	snprintf(name, sizeof(name),
+		 "%s-%d-%s", PDS_CORE_DRV_NAME, pdsc->pdev->bus->number, qcq->q.name);
+	index = pdsc_intr_alloc(pdsc, name, pdsc_adminq_isr, qcq);
+	if (index < 0)
+		return index;
+	qcq->intx = index;
+
+	return 0;
+}
+
+void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+	struct device *dev = pdsc->dev;
+
+	if (!(qcq && qcq->pdsc))
+		return;
+
+	pdsc_debugfs_del_qcq(qcq);
+
+	pdsc_qcq_intr_free(pdsc, qcq);
+
+	if (qcq->q_base) {
+		dmam_free_coherent(dev, qcq->q_size,
+				   qcq->q_base, qcq->q_base_pa);
+		qcq->q_base = NULL;
+		qcq->q_base_pa = 0;
+	}
+
+	if (qcq->cq_base) {
+		dmam_free_coherent(dev, qcq->cq_size, qcq->cq_base, qcq->cq_base_pa);
+		qcq->cq_base = NULL;
+		qcq->cq_base_pa = 0;
+	}
+
+	if (qcq->cq.info) {
+		vfree(qcq->cq.info);
+		qcq->cq.info = NULL;
+	}
+	if (qcq->q.info) {
+		vfree(qcq->q.info);
+		qcq->q.info = NULL;
+	}
+
+	qcq->pdsc = NULL;
+	memset(&qcq->q, 0, sizeof(qcq->q));
+	memset(&qcq->cq, 0, sizeof(qcq->cq));
+}
+
+static void pdsc_q_map(struct pdsc_queue *q, void *base, dma_addr_t base_pa)
+{
+	struct pdsc_q_info *cur;
+	unsigned int i;
+
+	q->base = base;
+	q->base_pa = base_pa;
+
+	for (i = 0, cur = q->info; i < q->num_descs; i++, cur++)
+		cur->desc = base + (i * q->desc_size);
+}
+
+static void pdsc_cq_map(struct pdsc_cq *cq, void *base, dma_addr_t base_pa)
+{
+	struct pdsc_cq_info *cur;
+	unsigned int i;
+
+	cq->base = base;
+	cq->base_pa = base_pa;
+
+	for (i = 0, cur = cq->info; i < cq->num_descs; i++, cur++)
+		cur->comp = base + (i * cq->desc_size);
+}
+
+int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index,
+		   const char *name, unsigned int flags, unsigned int num_descs,
+		   unsigned int desc_size, unsigned int cq_desc_size,
+		   unsigned int pid, struct pdsc_qcq *qcq)
+{
+	struct device *dev = pdsc->dev;
+	dma_addr_t cq_base_pa = 0;
+	dma_addr_t q_base_pa = 0;
+	void *q_base, *cq_base;
+	int err;
+
+	qcq->q.info = vzalloc(num_descs * sizeof(*qcq->q.info));
+	if (!qcq->q.info) {
+		dev_err(dev, "Cannot allocate %s queue info\n", name);
+		err = -ENOMEM;
+		goto err_out;
+	}
+
+	qcq->pdsc = pdsc;
+	qcq->flags = flags;
+	INIT_WORK(&qcq->work, pdsc_work_thread);
+
+	qcq->q.type = type;
+	qcq->q.index = index;
+	qcq->q.num_descs = num_descs;
+	qcq->q.desc_size = desc_size;
+	qcq->q.tail_idx = 0;
+	qcq->q.head_idx = 0;
+	qcq->q.pid = pid;
+	snprintf(qcq->q.name, sizeof(qcq->q.name), "%s%u", name, index);
+
+	err = pdsc_qcq_intr_alloc(pdsc, qcq);
+	if (err)
+		goto err_out_free_q_info;
+
+	qcq->cq.info = vzalloc(num_descs * sizeof(*qcq->cq.info));
+	if (!qcq->cq.info) {
+		dev_err(dev, "Cannot allocate %s completion queue info\n", name);
+		err = -ENOMEM;
+		goto err_out_free_irq;
+	}
+
+	qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx];
+	qcq->cq.num_descs = num_descs;
+	qcq->cq.desc_size = cq_desc_size;
+	qcq->cq.tail_idx = 0;
+	qcq->cq.done_color = 1;
+
+	if (flags & PDS_CORE_QCQ_F_NOTIFYQ) {
+		/* q & cq need to be contiguous in case of notifyq */
+		qcq->q_size = PAGE_SIZE + ALIGN(num_descs * desc_size, PAGE_SIZE) +
+						ALIGN(num_descs * cq_desc_size, PAGE_SIZE);
+		qcq->q_base = dmam_alloc_coherent(dev, qcq->q_size + qcq->cq_size,
+						  &qcq->q_base_pa,
+						  GFP_KERNEL);
+		if (!qcq->q_base) {
+			dev_err(dev, "Cannot allocate %s qcq DMA memory\n", name);
+			err = -ENOMEM;
+			goto err_out_free_cq_info;
+		}
+		q_base = PTR_ALIGN(qcq->q_base, PAGE_SIZE);
+		q_base_pa = ALIGN(qcq->q_base_pa, PAGE_SIZE);
+		pdsc_q_map(&qcq->q, q_base, q_base_pa);
+
+		cq_base = PTR_ALIGN(q_base +
+			ALIGN(num_descs * desc_size, PAGE_SIZE), PAGE_SIZE);
+		cq_base_pa = ALIGN(qcq->q_base_pa +
+			ALIGN(num_descs * desc_size, PAGE_SIZE), PAGE_SIZE);
+
+	} else {
+		/* q DMA descriptors */
+		qcq->q_size = PAGE_SIZE + (num_descs * desc_size);
+		qcq->q_base = dmam_alloc_coherent(dev, qcq->q_size,
+						  &qcq->q_base_pa,
+						  GFP_KERNEL);
+		if (!qcq->q_base) {
+			dev_err(dev, "Cannot allocate %s queue DMA memory\n", name);
+			err = -ENOMEM;
+			goto err_out_free_cq_info;
+		}
+		q_base = PTR_ALIGN(qcq->q_base, PAGE_SIZE);
+		q_base_pa = ALIGN(qcq->q_base_pa, PAGE_SIZE);
+		pdsc_q_map(&qcq->q, q_base, q_base_pa);
+
+		/* cq DMA descriptors */
+		qcq->cq_size = PAGE_SIZE + (num_descs * cq_desc_size);
+		qcq->cq_base = dmam_alloc_coherent(dev, qcq->cq_size,
+						   &qcq->cq_base_pa,
+						   GFP_KERNEL);
+		if (!qcq->cq_base) {
+			dev_err(dev, "Cannot allocate %s cq DMA memory\n", name);
+			err = -ENOMEM;
+			goto err_out_free_q;
+		}
+		cq_base = PTR_ALIGN(qcq->cq_base, PAGE_SIZE);
+		cq_base_pa = ALIGN(qcq->cq_base_pa, PAGE_SIZE);
+	}
+
+	pdsc_cq_map(&qcq->cq, cq_base, cq_base_pa);
+	qcq->cq.bound_q = &qcq->q;
+
+	pdsc_debugfs_add_qcq(pdsc, qcq);
+
+	return 0;
+
+err_out_free_q:
+	dmam_free_coherent(dev, qcq->q_size, qcq->q_base, qcq->q_base_pa);
+err_out_free_cq_info:
+	vfree(qcq->cq.info);
+err_out_free_irq:
+	pdsc_qcq_intr_free(pdsc, qcq);
+err_out_free_q_info:
+	vfree(qcq->q.info);
+	memset(qcq, 0, sizeof(*qcq));
+err_out:
+	dev_err(dev, "qcq alloc of %s%d failed %d\n", name, index, err);
+	return err;
+}
+
+static int pdsc_core_init(struct pdsc *pdsc)
+{
+	union pds_core_dev_comp comp = { 0 };
+	union pds_core_dev_cmd cmd = {
+		.init.opcode = PDS_CORE_CMD_INIT,
+	};
+	struct pds_core_dev_init_data_out cido;
+	struct pds_core_dev_init_data_in cidi;
+	u32 dbid_count;
+	u32 dbpage_num;
+	size_t sz;
+	int err;
+
+	cidi.adminq_q_base = cpu_to_le64(pdsc->adminqcq.q_base_pa);
+	cidi.adminq_cq_base = cpu_to_le64(pdsc->adminqcq.cq_base_pa);
+	cidi.notifyq_cq_base = cpu_to_le64(pdsc->notifyqcq.cq.base_pa);
+	cidi.flags = cpu_to_le32(PDS_CORE_QINIT_F_IRQ | PDS_CORE_QINIT_F_ENA);
+	cidi.intr_index = cpu_to_le16(pdsc->adminqcq.intx);
+	cidi.adminq_ring_size = ilog2(pdsc->adminqcq.q.num_descs);
+	cidi.notifyq_ring_size = ilog2(pdsc->notifyqcq.q.num_descs);
+
+	mutex_lock(&pdsc->devcmd_lock);
+
+	sz = min_t(size_t, sizeof(cidi), sizeof(pdsc->cmd_regs->data));
+	memcpy_toio(&pdsc->cmd_regs->data, &cidi, sz);
+
+	err = pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+
+	sz = min_t(size_t, sizeof(cido), sizeof(pdsc->cmd_regs->data));
+	memcpy_fromio(&cido, &pdsc->cmd_regs->data, sz);
+
+	mutex_unlock(&pdsc->devcmd_lock);
+
+	pdsc->hw_index = le32_to_cpu(cido.core_hw_index);
+
+	dbid_count = le32_to_cpu(pdsc->dev_ident.ndbpgs_per_lif);
+	dbpage_num = pdsc->hw_index * dbid_count;
+	pdsc->kern_dbpage = pdsc_map_dbpage(pdsc, dbpage_num);
+	if (!pdsc->kern_dbpage) {
+		dev_err(pdsc->dev, "Cannot map dbpage, aborting\n");
+		return -ENOMEM;
+	}
+
+	pdsc->adminqcq.q.hw_type = cido.adminq_hw_type;
+	pdsc->adminqcq.q.hw_index = le32_to_cpu(cido.adminq_hw_index);
+	pdsc->adminqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->adminqcq.q.hw_index);
+
+	pdsc->notifyqcq.q.hw_type = cido.notifyq_hw_type;
+	pdsc->notifyqcq.q.hw_index = le32_to_cpu(cido.notifyq_hw_index);
+	pdsc->notifyqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->notifyqcq.q.hw_index);
+
+	pdsc->last_eid = 0;
+
+	return err;
+}
+
 int pdsc_setup(struct pdsc *pdsc, bool init)
 {
+	int numdescs;
 	int err = 0;
 
 	if (init)
@@ -21,17 +381,60 @@ int pdsc_setup(struct pdsc *pdsc, bool init)
 	if (err)
 		return err;
 
+	/* Scale the descriptor ring length based on number of CPUs and VFs */
+	numdescs = max_t(int, PDSC_ADMINQ_MIN_LENGTH, num_online_cpus());
+	numdescs += 2 * pci_sriov_get_totalvfs(pdsc->pdev);
+	numdescs = roundup_pow_of_two(numdescs);
+	err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_ADMINQ, 0, "adminq",
+			     PDS_CORE_QCQ_F_CORE | PDS_CORE_QCQ_F_INTR,
+			     numdescs,
+			     sizeof(union pds_core_adminq_cmd),
+			     sizeof(union pds_core_adminq_comp),
+			     0, &pdsc->adminqcq);
+	if (err)
+		goto err_out_teardown;
+
+	err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_NOTIFYQ, 0, "notifyq",
+			     PDS_CORE_QCQ_F_NOTIFYQ,
+			     PDSC_NOTIFYQ_LENGTH,
+			     sizeof(struct pds_core_notifyq_cmd),
+			     sizeof(union pds_core_notifyq_comp),
+			     0, &pdsc->notifyqcq);
+	if (err)
+		goto err_out_teardown;
+
+	/* NotifyQ rides on the AdminQ interrupt */
+	pdsc->notifyqcq.intx = pdsc->adminqcq.intx;
+
+	/* Set up the Core with the AdminQ and NotifyQ info */
+	err = pdsc_core_init(pdsc);
+	if (err)
+		goto err_out_teardown;
+
 	clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
 	return 0;
+
+err_out_teardown:
+	pdsc_teardown(pdsc, init);
+	return err;
 }
 
 void pdsc_teardown(struct pdsc *pdsc, bool removing)
 {
+	int i;
+
 	pdsc_devcmd_reset(pdsc);
+	pdsc_qcq_free(pdsc, &pdsc->notifyqcq);
+	pdsc_qcq_free(pdsc, &pdsc->adminqcq);
 
-	if (removing && pdsc->intr_info) {
-		devm_kfree(pdsc->dev, pdsc->intr_info);
-		pdsc->intr_info = NULL;
+	if (pdsc->intr_info) {
+		for (i = 0; i < pdsc->nintrs; i++)
+			pdsc_intr_free(pdsc, i);
+
+		if (removing) {
+			devm_kfree(pdsc->dev, pdsc->intr_info);
+			pdsc->intr_info = NULL;
+		}
 	}
 
 	if (pdsc->kern_dbpage) {
@@ -42,6 +445,36 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
 	set_bit(PDSC_S_FW_DEAD, &pdsc->state);
 }
 
+int pdsc_start(struct pdsc *pdsc)
+{
+	pds_core_intr_mask(&pdsc->intr_ctrl[pdsc->adminqcq.intx],
+			   PDS_CORE_INTR_MASK_CLEAR);
+
+	return 0;
+}
+
+static void pdsc_mask_interrupts(struct pdsc *pdsc)
+{
+	int i;
+
+	if (!pdsc->intr_info)
+		return;
+
+	/* Mask interrupts that are in use */
+	for (i = 0; i < pdsc->nintrs; i++)
+		if (pdsc->intr_info[i].vector)
+			pds_core_intr_mask(&pdsc->intr_ctrl[i],
+					   PDS_CORE_INTR_MASK_SET);
+}
+
+void pdsc_stop(struct pdsc *pdsc)
+{
+	if (pdsc->wq)
+		flush_workqueue(pdsc->wq);
+
+	pdsc_mask_interrupts(pdsc);
+}
+
 static void pdsc_fw_down(struct pdsc *pdsc)
 {
 	mutex_lock(&pdsc->config_lock);
@@ -52,6 +485,7 @@ static void pdsc_fw_down(struct pdsc *pdsc)
 		return;
 	}
 
+	pdsc_mask_interrupts(pdsc);
 	pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
 
 	mutex_unlock(&pdsc->config_lock);
@@ -73,6 +507,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)
 	if (err)
 		goto err_out;
 
+	err = pdsc_start(pdsc);
+	if (err)
+		goto err_out;
+
 	mutex_unlock(&pdsc->config_lock);
 
 	return;
drivers/net/ethernet/amd/pds_core/core.h +149 -0

206 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/debugfs.c +125 -0

129 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/devlink.c +55 -0

62 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/main.c +14 -0
index 3150a2a665ec..23e9bf48a05e 100644
@@ -158,6 +158,13 @@ static int pdsc_map_bars(struct pdsc *pdsc)
 	return err;
 }
 
+void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num)
+{
+	return pci_iomap_range(pdsc->pdev,
+			       pdsc->bars[PDS_CORE_PCI_BAR_DBELL].res_index,
+			       (u64)page_num << PAGE_SHIFT, PAGE_SIZE);
+}
+
 static DEFINE_IDA(pdsc_pf_ida);
 
 #define PDSC_WQ_NAME_LEN 24
@@ -227,11 +234,15 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	/* PDS device setup */
 	mutex_init(&pdsc->devcmd_lock);
 	mutex_init(&pdsc->config_lock);
+	spin_lock_init(&pdsc->adminq_lock);
 
 	mutex_lock(&pdsc->config_lock);
 	err = pdsc_setup(pdsc, PDSC_SETUP_INIT);
 	if (err)
 		goto err_out_unmap_bars;
+	err = pdsc_start(pdsc);
+	if (err)
+		goto err_out_teardown;
 
 	/* publish devlink device */
 	err = pdsc_dl_register(pdsc);
@@ -251,6 +262,8 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	return 0;
 
 err_out:
+	pdsc_stop(pdsc);
+err_out_teardown:
 	pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
 err_out_unmap_bars:
 	del_timer_sync(&pdsc->wdtimer);
@@ -299,6 +312,7 @@ static void pdsc_remove(struct pci_dev *pdev)
 	}
 
 	/* Device teardown */
+	pdsc_stop(pdsc);
 	pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
 	pdsc_debugfs_del_dev(pdsc);
 	mutex_unlock(&pdsc->config_lock);
include/linux/pds/pds_adminq.h new file +641 -0

641 lines not on this page · raw message

Signature · 1 line
2.17.1
#11 lore raw

PATCH v3 6/14 pds_core: Add adminq processing and commands

In reply to Shannon Nelson

Add the service routines for submitting and processing the adminq messages and for handling notifyq events.

---
 drivers/net/ethernet/amd/pds_core/Makefile |   1 +
 drivers/net/ethernet/amd/pds_core/adminq.c | 282 +++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.c   |  11 -
 drivers/net/ethernet/amd/pds_core/core.h   |   6 +
 include/linux/pds/pds_adminq.h             |   2 +
 5 files changed, 291 insertions(+), 11 deletions(-)
 create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
index eaca8557ba66..ef76dcd7fccd 100644
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
 pds_core-y := main.o \
 	      devlink.o \
 	      dev.o \
+	      adminq.o \
 	      core.o
 
 pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/adminq.c new file +282 -0

282 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/core.c +0 -11
index f0bccc6a0f3a..0b785aae9189 100644
@@ -12,17 +12,6 @@
 
 #include <linux/pds/pds_adminq.h>
 
-void pdsc_work_thread(struct work_struct *work)
-{
-	/* stub */
-}
-
-irqreturn_t pdsc_adminq_isr(int irq, void *data)
-{
-	/* stub */
-	return IRQ_HANDLED;
-}
-
 void pdsc_intr_free(struct pdsc *pdsc, int index)
 {
 	struct pdsc_intr_info *intr_info;
drivers/net/ethernet/amd/pds_core/core.h +6 -0
index fdf9dd1898c7..1610b7b485d2 100644
@@ -260,6 +260,12 @@ int pdsc_devcmd_reset(struct pdsc *pdsc);
 int pdsc_dev_reinit(struct pdsc *pdsc);
 int pdsc_dev_init(struct pdsc *pdsc);
 
+int pdsc_adminq_post(struct pdsc *pdsc,
+		     struct pdsc_qcq *qcq,
+		     union pds_core_adminq_cmd *cmd,
+		     union pds_core_adminq_comp *comp,
+		     bool fast_poll);
+
 int pdsc_intr_alloc(struct pdsc *pdsc, char *name,
 		    irq_handler_t handler, void *data);
 void pdsc_intr_free(struct pdsc *pdsc, int index);
include/linux/pds/pds_adminq.h +2 -0
index b39131eaa134..ac620eb7b121 100644
@@ -4,6 +4,8 @@
 #ifndef _PDS_CORE_ADMINQ_H_
 #define _PDS_CORE_ADMINQ_H_
 
+#define PDSC_ADMINQ_MAX_POLL_INTERVAL	256
+
 enum pds_core_adminq_flags {
 	PDS_AQ_FLAG_FASTPOLL	= BIT(1),	/* poll for completion at 1ms intervals */
 };
Signature · 1 line
2.17.1
#12 lore raw

PATCH v3 7/14 pds_core: add FW update feature to devlink

In reply to Shannon Nelson

Add in the support for doing firmware updates. Of the two main banks available, 1, and 2, this updates the one not in use and then selects it for the next boot.

---
 drivers/net/ethernet/amd/pds_core/Makefile  |   3 +-
 drivers/net/ethernet/amd/pds_core/core.h    |   3 +
 drivers/net/ethernet/amd/pds_core/devlink.c |  10 +
 drivers/net/ethernet/amd/pds_core/fw.c      | 192 ++++++++++++++++++++
 4 files changed, 207 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
drivers/net/ethernet/amd/pds_core/Makefile +2 -1
index ef76dcd7fccd..6d1d6c58a1fa 100644
@@ -7,6 +7,7 @@ pds_core-y := main.o \
 	      devlink.o \
 	      dev.o \
 	      adminq.o \
-	      core.o
+	      core.o \
+	      fw.o
 
 pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.h +3 -0
index 1610b7b485d2..d87d6d9e625e 100644
@@ -284,4 +284,7 @@ void pdsc_process_adminq(struct pdsc_qcq *qcq);
 void pdsc_work_thread(struct work_struct *work);
 irqreturn_t pdsc_adminq_isr(int irq, void *data);
 
+int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
+			 struct netlink_ext_ack *extack);
+
 #endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/devlink.c +10 -0
index f3706f9109f2..df125ed0aa8c 100644
@@ -8,6 +8,15 @@
 
 #include "core.h"
 
+static int pdsc_dl_flash_update(struct devlink *dl,
+				struct devlink_flash_update_params *params,
+				struct netlink_ext_ack *extack)
+{
+	struct pdsc *pdsc = devlink_priv(dl);
+
+	return pdsc_firmware_update(pdsc, params->fw, extack);
+}
+
 static int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
 			    struct netlink_ext_ack *extack)
 {
@@ -64,6 +73,7 @@ static int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
 
 static const struct devlink_ops pdsc_dl_ops = {
 	.info_get	= pdsc_dl_info_get,
+	.flash_update	= pdsc_dl_flash_update,
 };
 
 struct pdsc *pdsc_dl_alloc(struct device *dev)
drivers/net/ethernet/amd/pds_core/fw.c new file +192 -0

192 lines not on this page · raw message

Signature · 1 line
2.17.1
#13 lore raw

PATCH v3 8/14 pds_core: set up the VIF definitions and defaults

In reply to Shannon Nelson

The Virtual Interfaces (VIFs) supported by the DSC's configuration (VFio Live Migration, vDPA, etc) are reported in the dev_ident struct.

---
 drivers/net/ethernet/amd/pds_core/core.c    | 54 +++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.h    | 13 +++++
 drivers/net/ethernet/amd/pds_core/debugfs.c | 23 +++++++++
 include/linux/pds/pds_common.h              | 17 +++++++
 4 files changed, 107 insertions(+)
drivers/net/ethernet/amd/pds_core/core.c +54 -0
index 0b785aae9189..393ec10ad745 100644
@@ -358,6 +358,47 @@ static int pdsc_core_init(struct pdsc *pdsc)
 	return err;
 }
 
+static struct pdsc_viftype pdsc_viftype_defaults[] = {
+	[PDS_DEV_TYPE_VDPA] = { .name = PDS_DEV_TYPE_VDPA_STR,
+				.vif_id = PDS_DEV_TYPE_VDPA,
+				.dl_id = DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET },
+	[PDS_DEV_TYPE_LM]   = { .name = PDS_DEV_TYPE_LM_STR,
+				.vif_id = PDS_DEV_TYPE_LM,
+				.dl_id = DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION},
+	[PDS_DEV_TYPE_MAX] = { 0 }
+};
+
+static int pdsc_viftypes_init(struct pdsc *pdsc)
+{
+	enum pds_core_vif_types vt;
+
+	pdsc->viftype_status = devm_kzalloc(pdsc->dev,
+					    sizeof(pdsc_viftype_defaults),
+					    GFP_KERNEL);
+	if (!pdsc->viftype_status)
+		return -ENOMEM;
+
+	for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+		bool vt_support;
+
+		if (!pdsc_viftype_defaults[vt].name)
+			continue;
+
+		/* Grab the defaults */
+		pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt];
+
+		/* See what the Core device has for support */
+		vt_support = !!le16_to_cpu(pdsc->dev_ident.vif_types[vt]);
+		dev_dbg(pdsc->dev, "VIF %s is %ssupported\n",
+			pdsc->viftype_status[vt].name,
+			vt_support ? "" : "not ");
+
+		pdsc->viftype_status[vt].supported = vt_support;
+	}
+
+	return 0;
+}
+
 int pdsc_setup(struct pdsc *pdsc, bool init)
 {
 	int numdescs;
@@ -400,6 +441,14 @@ int pdsc_setup(struct pdsc *pdsc, bool init)
 	if (err)
 		goto err_out_teardown;
 
+	/* Set up the VIFs */
+	err = pdsc_viftypes_init(pdsc);
+	if (err)
+		goto err_out_teardown;
+
+	if (init)
+		pdsc_debugfs_add_viftype(pdsc);
+
 	clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
 	return 0;
 
@@ -416,6 +465,11 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
 	pdsc_qcq_free(pdsc, &pdsc->notifyqcq);
 	pdsc_qcq_free(pdsc, &pdsc->adminqcq);
 
+	if (pdsc->viftype_status) {
+		devm_kfree(pdsc->dev, pdsc->viftype_status);
+		pdsc->viftype_status = NULL;
+	}
+
 	if (pdsc->intr_info) {
 		for (i = 0; i < pdsc->nintrs; i++)
 			pdsc_intr_free(pdsc, i);
drivers/net/ethernet/amd/pds_core/core.h +13 -0
index d87d6d9e625e..0355a0b850eb 100644
@@ -123,6 +123,15 @@ struct pdsc_qcq {
 	struct dentry *dentry;
 };
 
+struct pdsc_viftype {
+	char *name;
+	bool supported;
+	bool enabled;
+	int dl_id;
+	int vif_id;
+	struct pds_auxiliary_dev *padev;
+};
+
 /* No state flags set means we are in a steady running state */
 enum pdsc_state_flags {
 	PDSC_S_FW_DEAD,		    /* fw stopped, waiting for startup or recovery */
@@ -173,6 +182,7 @@ struct pdsc {
 	struct pdsc_qcq adminqcq;
 	struct pdsc_qcq notifyqcq;
 	u64 last_eid;
+	struct pdsc_viftype *viftype_status;
 };
 
 /** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -227,6 +237,7 @@ struct pdsc *pdsc_dl_alloc(struct device *dev);
 void pdsc_dl_free(struct pdsc *pdsc);
 int pdsc_dl_register(struct pdsc *pdsc);
 void pdsc_dl_unregister(struct pdsc *pdsc);
+int pdsc_dl_vif_add(struct pdsc *pdsc, enum pds_core_vif_types vt, const char *name);
 
 #ifdef CONFIG_DEBUG_FS
 void pdsc_debugfs_create(void);
@@ -234,6 +245,7 @@ void pdsc_debugfs_destroy(void);
 void pdsc_debugfs_add_dev(struct pdsc *pdsc);
 void pdsc_debugfs_del_dev(struct pdsc *pdsc);
 void pdsc_debugfs_add_ident(struct pdsc *pdsc);
+void pdsc_debugfs_add_viftype(struct pdsc *pdsc);
 void pdsc_debugfs_add_irqs(struct pdsc *pdsc);
 void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq);
 void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq);
@@ -243,6 +255,7 @@ static inline void pdsc_debugfs_destroy(void) { }
 static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
 static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
 static inline void pdsc_debugfs_add_ident(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_add_viftype(struct pdsc *pdsc) { }
 static inline void pdsc_debugfs_add_irqs(struct pdsc *pdsc) { }
 static inline void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq) { }
 static inline void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq) { }
drivers/net/ethernet/amd/pds_core/debugfs.c +23 -0
index dd7291e4d205..5766b2d4204f 100644
@@ -82,6 +82,29 @@ void pdsc_debugfs_add_ident(struct pdsc *pdsc)
 	debugfs_create_file("identity", 0400, pdsc->dentry, pdsc, &identity_fops);
 }
 
+static int viftype_show(struct seq_file *seq, void *v)
+{
+	struct pdsc *pdsc = seq->private;
+	int vt;
+
+	for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+		if (!pdsc->viftype_status[vt].name)
+			continue;
+
+		seq_printf(seq, "%s\t%d supported %d enabled\n",
+			   pdsc->viftype_status[vt].name,
+			   pdsc->viftype_status[vt].supported,
+			   pdsc->viftype_status[vt].enabled);
+	}
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(viftype);
+
+void pdsc_debugfs_add_viftype(struct pdsc *pdsc)
+{
+	debugfs_create_file("viftypes", 0400, pdsc->dentry, pdsc, &viftype_fops);
+}
+
 static int irqs_show(struct seq_file *seq, void *v)
 {
 	struct pdsc *pdsc = seq->private;
include/linux/pds/pds_common.h +17 -0
index fd889c01f24e..93ce1256e8ff 100644
@@ -50,6 +50,23 @@ enum pds_core_driver_type {
 	PDS_DRIVER_ESXI    = 6,
 };
 
+enum pds_core_vif_types {
+	PDS_DEV_TYPE_VDPA	= 1,
+	PDS_DEV_TYPE_VFIO	= 2,
+	PDS_DEV_TYPE_ETH	= 3,
+	PDS_DEV_TYPE_RDMA	= 4,
+	PDS_DEV_TYPE_LM		= 5,
+
+	/* new ones added before this line */
+	PDS_DEV_TYPE_MAX	= 16   /* don't change - used in struct size */
+};
+
+#define PDS_DEV_TYPE_VDPA_STR	"vDPA"
+#define PDS_DEV_TYPE_VFIO_STR	"VFio"
+#define PDS_DEV_TYPE_ETH_STR	"Eth"
+#define PDS_DEV_TYPE_RDMA_STR	"RDMA"
+#define PDS_DEV_TYPE_LM_STR	"LM"
+
 #define PDS_CORE_IFNAMSIZ		16
 
 /**
Signature · 1 line
2.17.1
#14 lore raw

PATCH v3 9/14 pds_core: initial VF configuration

In reply to Shannon Nelson

Set up the VF control and management

---
 drivers/net/ethernet/amd/pds_core/core.h | 10 ++++++-
 drivers/net/ethernet/amd/pds_core/main.c | 35 ++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 1 deletion(-)
drivers/net/ethernet/amd/pds_core/core.h +9 -1
index 0355a0b850eb..1885976c6486 100644
@@ -30,6 +30,13 @@ struct pdsc_dev_bar {
 	int res_index;
 };
 
+struct pdsc_vf {
+	u16     index;
+	__le16  vif_types[PDS_DEV_TYPE_MAX];
+
+	struct pds_auxiliary_dev *padev;
+};
+
 struct pdsc_devinfo {
 	u8 asic_type;
 	u8 asic_rev;
@@ -147,6 +154,7 @@ struct pdsc {
 	struct dentry *dentry;
 	struct device *dev;
 	struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX];
+	struct pdsc_vf *vfs;
 	int num_vfs;
 	int hw_index;
 	int id;
@@ -166,6 +174,7 @@ struct pdsc {
 	struct pdsc_intr_info *intr_info;	/* array of nintrs elements */
 
 	struct workqueue_struct *wq;
+	struct rw_semaphore vf_op_lock;	/* lock for VF operations */
 
 	unsigned int devcmd_timeout;
 	struct mutex devcmd_lock;	/* lock for dev_cmd operations */
@@ -299,5 +308,4 @@ irqreturn_t pdsc_adminq_isr(int irq, void *data);
 
 int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
 			 struct netlink_ext_ack *extack);
-
 #endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/main.c +35 -0
index 23e9bf48a05e..1376dec84756 100644
@@ -165,6 +165,37 @@ void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num)
 			       (u64)page_num << PAGE_SHIFT, PAGE_SIZE);
 }
 
+static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)
+{
+	struct pdsc *pdsc = pci_get_drvdata(pdev);
+	struct device *dev = pdsc->dev;
+	int ret = 0;
+
+	if (num_vfs > 0) {
+		pdsc->vfs = kcalloc(num_vfs, sizeof(struct pdsc_vf), GFP_KERNEL);
+		if (!pdsc->vfs)
+			return -ENOMEM;
+		pdsc->num_vfs = num_vfs;
+
+		ret = pci_enable_sriov(pdev, num_vfs);
+		if (ret) {
+			dev_err(dev, "Cannot enable SRIOV: %pe\n", ERR_PTR(ret));
+			goto no_vfs;
+		}
+
+		return num_vfs;
+	}
+
+no_vfs:
+	pci_disable_sriov(pdev);
+
+	kfree(pdsc->vfs);
+	pdsc->vfs = NULL;
+	pdsc->num_vfs = 0;
+
+	return ret;
+}
+
 static DEFINE_IDA(pdsc_pf_ida);
 
 #define PDSC_WQ_NAME_LEN 24
@@ -237,6 +268,7 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	spin_lock_init(&pdsc->adminq_lock);
 
 	mutex_lock(&pdsc->config_lock);
+	init_rwsem(&pdsc->vf_op_lock);
 	err = pdsc_setup(pdsc, PDSC_SETUP_INIT);
 	if (err)
 		goto err_out_unmap_bars;
@@ -300,6 +332,8 @@ static void pdsc_remove(struct pci_dev *pdev)
 	 */
 	pdsc_dl_unregister(pdsc);
 
+	pdsc_sriov_configure(pdev, 0);
+
 	/* Now we can lock it up and tear it down */
 	mutex_lock(&pdsc->config_lock);
 	set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
@@ -337,6 +371,7 @@ static struct pci_driver pdsc_driver = {
 	.id_table = pdsc_id_table,
 	.probe = pdsc_probe,
 	.remove = pdsc_remove,
+	.sriov_configure = pdsc_sriov_configure,
 };
 
 static int __init pdsc_init_module(void)
Signature · 1 line
2.17.1
#15 lore raw

PATCH v3 10/14 pds_core: add auxiliary_bus devices

In reply to Shannon Nelson

An auxiliary_bus device is created for each VF, and the device name is made up of the PF driver name, VIF name, and PCI BDF of the VF.

---
 drivers/net/ethernet/amd/pds_core/Makefile |   1 +
 drivers/net/ethernet/amd/pds_core/auxbus.c | 123 +++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.h   |   4 +
 drivers/net/ethernet/amd/pds_core/main.c   |  43 +++++++
 include/linux/pds/pds_auxbus.h             |  35 ++++++
 5 files changed, 206 insertions(+)
 create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
 create mode 100644 include/linux/pds/pds_auxbus.h
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
index 6d1d6c58a1fa..0abc33ce826c 100644
@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
 
 pds_core-y := main.o \
 	      devlink.o \
+	      auxbus.o \
 	      dev.o \
 	      adminq.o \
 	      core.o \
drivers/net/ethernet/amd/pds_core/auxbus.c new file +123 -0

123 lines not on this page · raw message

drivers/net/ethernet/amd/pds_core/core.h +4 -0
index 1885976c6486..7abd2cc4efc7 100644
@@ -188,6 +188,7 @@ struct pdsc {
 	dma_addr_t phy_db_pages;
 	u64 __iomem *kern_dbpage;
 
+	struct notifier_block nb;
 	struct pdsc_qcq adminqcq;
 	struct pdsc_qcq notifyqcq;
 	u64 last_eid;
@@ -302,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);
 void pdsc_stop(struct pdsc *pdsc);
 void pdsc_health_thread(struct work_struct *work);
 
+int pdsc_auxbus_dev_add_vf(struct pdsc *pdsc, int vf_id);
+int pdsc_auxbus_dev_del_vf(struct pdsc *pdsc, int vf_id);
+
 void pdsc_process_adminq(struct pdsc_qcq *qcq);
 void pdsc_work_thread(struct work_struct *work);
 irqreturn_t pdsc_adminq_isr(int irq, void *data);
drivers/net/ethernet/amd/pds_core/main.c +43 -0

75 lines not on this page · raw message

include/linux/pds/pds_auxbus.h new file +35 -0
new file mode 100644
index 000000000000..737fd4dbbf5a
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _PDSC_AUXBUS_H_
+#define _PDSC_AUXBUS_H_
+
+#include <linux/auxiliary_bus.h>
+
+struct pds_auxiliary_dev;
+
+struct pds_auxiliary_drv {
+	/* .event_handler() - callback for receiving events
+	 * padev:  ptr to the client device info
+	 * event:  ptr to event data
+	 * The client can provide an event handler callback that can
+	 * receive DSC events.  The Core driver may generate its
+	 * own events which can notify the client of changes in the
+	 * DSC status, such as a RESET event generated when the Core
+	 * has lost contact with the FW - in this case the event.eid
+	 * field will be 0.
+	 */
+	void (*event_handler)(struct pds_auxiliary_dev *padev,
+			      union pds_core_notifyq_comp *event);
+};
+
+struct pds_auxiliary_dev {
+	struct auxiliary_device aux_dev;
+	struct pci_dev *pcidev;
+	u32 id;
+	u16 client_id;
+	void (*event_handler)(struct pds_auxiliary_dev *padev,
+			      union pds_core_notifyq_comp *event);
+	void *priv;
+};
+#endif /* _PDSC_AUXBUS_H_ */
Signature · 1 line
2.17.1
#16 lore raw

PATCH v3 11/14 pds_core: devlink params for enabling VIF support

In reply to Shannon Nelson

Add the devlink parameter switches so the user can enable the features supported by the VFs.

---
 drivers/net/ethernet/amd/pds_core/devlink.c | 88 +++++++++++++++++++++
 1 file changed, 88 insertions(+)
drivers/net/ethernet/amd/pds_core/devlink.c +88 -0
index df125ed0aa8c..bd673cbf07c4 100644
@@ -8,6 +8,88 @@
 
 #include "core.h"
 
+static struct pdsc_viftype *pdsc_dl_find_viftype_by_id(struct pdsc *pdsc,
+						       enum devlink_param_type dl_id)
+{
+	int vt;
+
+	for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+		if (pdsc->viftype_status[vt].dl_id == dl_id)
+			return &pdsc->viftype_status[vt];
+	}
+
+	return NULL;
+}
+
+static int pdsc_dl_enable_get(struct devlink *dl, u32 id,
+			      struct devlink_param_gset_ctx *ctx)
+{
+	struct pdsc *pdsc = devlink_priv(dl);
+	struct pdsc_viftype *vt_entry;
+
+	vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+	if (!vt_entry)
+		return -ENOENT;
+
+	ctx->val.vbool = vt_entry->enabled;
+
+	return 0;
+}
+
+static int pdsc_dl_enable_set(struct devlink *dl, u32 id,
+			      struct devlink_param_gset_ctx *ctx)
+{
+	struct pdsc *pdsc = devlink_priv(dl);
+	struct pdsc_viftype *vt_entry;
+	int err = 0;
+	int vf;
+
+	vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+	if (!vt_entry || !vt_entry->supported)
+		return -EOPNOTSUPP;
+
+	if (vt_entry->enabled == ctx->val.vbool)
+		return 0;
+
+	vt_entry->enabled = ctx->val.vbool;
+	for (vf = 0; vf < pdsc->num_vfs; vf++) {
+		err = ctx->val.vbool ? pdsc_auxbus_dev_add_vf(pdsc, vf) :
+				       pdsc_auxbus_dev_del_vf(pdsc, vf);
+	}
+
+	return err;
+}
+
+static int pdsc_dl_enable_validate(struct devlink *dl, u32 id,
+				   union devlink_param_value val,
+				   struct netlink_ext_ack *extack)
+{
+	struct pdsc *pdsc = devlink_priv(dl);
+	struct pdsc_viftype *vt_entry;
+
+	vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+	if (!vt_entry || !vt_entry->supported)
+		return -EOPNOTSUPP;
+
+	if (!pdsc->viftype_status[vt_entry->vif_id].supported)
+		return -ENODEV;
+
+	return 0;
+}
+
+static const struct devlink_param pdsc_dl_params[] = {
+	DEVLINK_PARAM_GENERIC(ENABLE_VNET,
+			      BIT(DEVLINK_PARAM_CMODE_RUNTIME),
+			      pdsc_dl_enable_get,
+			      pdsc_dl_enable_set,
+			      pdsc_dl_enable_validate),
+	DEVLINK_PARAM_GENERIC(ENABLE_MIGRATION,
+			      BIT(DEVLINK_PARAM_CMODE_RUNTIME),
+			      pdsc_dl_enable_get,
+			      pdsc_dl_enable_set,
+			      pdsc_dl_enable_validate),
+};
+
 static int pdsc_dl_flash_update(struct devlink *dl,
 				struct devlink_flash_update_params *params,
 				struct netlink_ext_ack *extack)
@@ -97,6 +179,12 @@ void pdsc_dl_free(struct pdsc *pdsc)
 int pdsc_dl_register(struct pdsc *pdsc)
 {
 	struct devlink *dl = priv_to_devlink(pdsc);
+	int err;
+
+	err = devlink_params_register(dl, pdsc_dl_params,
+				      ARRAY_SIZE(pdsc_dl_params));
+	if (err)
+		return err;
 
 	devlink_register(dl);
 
Signature · 1 line
2.17.1
#17 lore raw

PATCH v3 12/14 pds_core: add the aux client API

In reply to Shannon Nelson

Add the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,

---
 drivers/net/ethernet/amd/pds_core/auxbus.c | 144 ++++++++++++++++++++-
 include/linux/pds/pds_auxbus.h             |  50 +++++++
 2 files changed, 192 insertions(+), 2 deletions(-)
drivers/net/ethernet/amd/pds_core/auxbus.c +142 -2
index dc36fc98de52..6d5e7e1d176c 100644
@@ -11,6 +11,144 @@
 #include <linux/pds/pds_adminq.h>
 #include <linux/pds/pds_auxbus.h>
 
+/**
+ * pds_client_register - Register the client with the device
+ * @padev:  ptr to the client device info
+ * @padrv:  ptr to the client driver info
+ *
+ * Register the client with the core and with the DSC.  The core
+ * will fill in the client padev->client_id for use in calls
+ * to the DSC AdminQ
+ */
+static int pds_client_register(struct pds_auxiliary_dev *padev,
+			       struct pds_auxiliary_drv *padrv)
+{
+	union pds_core_adminq_comp comp = { 0 };
+	union pds_core_adminq_cmd cmd = { 0 };
+	struct device *dev;
+	struct pdsc *pdsc;
+	int err;
+	u16 ci;
+
+	pdsc = (struct pdsc *)dev_get_drvdata(padev->aux_dev.dev.parent);
+	dev = pdsc->dev;
+
+	if (pdsc->state)
+		return -ENXIO;
+
+	cmd.client_reg.opcode = PDS_AQ_CMD_CLIENT_REG;
+	strscpy(cmd.client_reg.devname, dev_name(&padev->aux_dev.dev),
+		sizeof(cmd.client_reg.devname));
+
+	err = pdsc_adminq_post(pdsc, &pdsc->adminqcq, &cmd, &comp, false);
+	if (err) {
+		dev_info(dev, "register dev_name %s with DSC failed, status %d: %pe\n",
+			 dev_name(&padev->aux_dev.dev), comp.status, ERR_PTR(err));
+		return err;
+	}
+
+	ci = le16_to_cpu(comp.client_reg.client_id);
+	if (!ci) {
+		dev_err(dev, "%s: device returned null client_id\n", __func__);
+		return -EIO;
+	}
+
+	padev->client_id = ci;
+	padev->event_handler = padrv->event_handler;
+
+	return 0;
+}
+
+/**
+ * pds_client_unregister - Disconnect the client from the device
+ * @padev:  ptr to the client device info
+ *
+ * Disconnect the client from the core and with the DSC.
+ */
+static int pds_client_unregister(struct pds_auxiliary_dev *padev)
+{
+	union pds_core_adminq_comp comp = { 0 };
+	union pds_core_adminq_cmd cmd = { 0 };
+	struct device *dev;
+	struct pdsc *pdsc;
+	int err;
+
+	pdsc = (struct pdsc *)dev_get_drvdata(padev->aux_dev.dev.parent);
+	dev = pdsc->dev;
+
+	if (pdsc->state)
+		return -ENXIO;
+
+	cmd.client_unreg.opcode = PDS_AQ_CMD_CLIENT_UNREG;
+	cmd.client_unreg.client_id = cpu_to_le16(padev->client_id);
+
+	err = pdsc_adminq_post(pdsc, &pdsc->adminqcq, &cmd, &comp, false);
+	if (err)
+		dev_info(dev, "unregister dev_name %s failed, status %d: %pe\n",
+			 dev_name(&padev->aux_dev.dev), comp.status, ERR_PTR(err));
+
+	padev->client_id = 0;
+
+	return err;
+}
+
+/**
+ * pds_client_adminq_cmd - Process an adminq request for the client
+ * @padev:   ptr to the client device
+ * @req:     ptr to buffer with request
+ * @req_len: length of actual struct used for request
+ * @resp:    ptr to buffer where answer is to be copied
+ * @flags:   optional flags from pds_core_adminq_flags
+ *
+ * Return: 0 on success, or
+ *         negative for error
+ *
+ * Client sends pointers to request and response buffers
+ * Core copies request data into pds_core_client_request_cmd
+ * Core sets other fields as needed
+ * Core posts to AdminQ
+ * Core copies completion data into response buffer
+ */
+static int pds_client_adminq_cmd(struct pds_auxiliary_dev *padev,
+				 union pds_core_adminq_cmd *req,
+				 size_t req_len,
+				 union pds_core_adminq_comp *resp,
+				 u64 flags)
+{
+	union pds_core_adminq_cmd cmd = { 0 };
+	struct device *dev;
+	struct pdsc *pdsc;
+	size_t cp_len;
+	int err;
+
+	pdsc = (struct pdsc *)dev_get_drvdata(padev->aux_dev.dev.parent);
+	dev = pdsc->dev;
+
+	dev_dbg(dev, "%s: %s opcode %d\n",
+		__func__, dev_name(&padev->aux_dev.dev), req->opcode);
+
+	if (pdsc->state)
+		return -ENXIO;
+
+	/* Wrap the client's request */
+	cmd.client_request.opcode = PDS_AQ_CMD_CLIENT_CMD;
+	cmd.client_request.client_id = cpu_to_le16(padev->client_id);
+	cp_len = min_t(size_t, req_len, sizeof(cmd.client_request.client_cmd));
+	memcpy(cmd.client_request.client_cmd, req, cp_len);
+
+	err = pdsc_adminq_post(pdsc, &pdsc->adminqcq, &cmd, resp, !!(flags & PDS_AQ_FLAG_FASTPOLL));
+	if (err && err != -EAGAIN)
+		dev_info(dev, "client admin cmd failed: %pe\n", ERR_PTR(err));
+
+	return err;
+}
+
+static struct pds_core_ops pds_core_ops = {
+	.register_client = pds_client_register,
+	.unregister_client = pds_client_unregister,
+	.adminq_cmd = pds_client_adminq_cmd,
+};
+
 static void pdsc_auxbus_dev_release(struct device *dev)
 {
 	struct pds_auxiliary_dev *padev =
@@ -21,7 +159,8 @@ static void pdsc_auxbus_dev_release(struct device *dev)
 
 static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *pdsc,
 							  char *name, u32 id,
-							  struct pci_dev *client_dev)
+							  struct pci_dev *client_dev,
+							  struct pds_core_ops *ops)
 {
 	struct auxiliary_device *aux_dev;
 	struct pds_auxiliary_dev *padev;
@@ -31,6 +170,7 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *pdsc,
 	if (!padev)
 		return NULL;
 
+	padev->ops = ops;
 	padev->pcidev = client_dev;
 
 	aux_dev = &padev->aux_dev;
@@ -96,7 +236,7 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *pdsc, int vf_id)
 		id = PCI_DEVID(pdsc->pdev->bus->number,
 			       pci_iov_virtfn_devfn(pdsc->pdev, vf_id));
 		padev = pdsc_auxbus_dev_register(pdsc, pdsc->viftype_status[vt].name, id,
-						 pdsc->pdev);
+						 pdsc->pdev, &pds_core_ops);
 		pdsc->vfs[vf_id].padev = padev;
 
 		/* We only support a single type per VF, so jump out here */
include/linux/pds/pds_auxbus.h +50 -0

60 lines not on this page · raw message

Signature · 1 line
2.17.1
#18 lore raw

PATCH v3 13/14 pds_core: publish events to the clients

In reply to Shannon Nelson

When the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler. Add the code to pass along the events to either the specific client or to all the clients.

---
 drivers/net/ethernet/amd/pds_core/adminq.c | 17 +++++++++
 drivers/net/ethernet/amd/pds_core/auxbus.c | 40 ++++++++++++++++++++++
 drivers/net/ethernet/amd/pds_core/core.c   | 15 ++++++++
 drivers/net/ethernet/amd/pds_core/core.h   |  3 ++
 4 files changed, 75 insertions(+)
drivers/net/ethernet/amd/pds_core/adminq.c +17 -0
index 59a58305a08a..64ced7851549 100644
@@ -34,11 +34,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
 		case PDS_EVENT_LINK_CHANGE:
 			dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
 				 ecode, eid);
+			pdsc_auxbus_publish(pdsc, PDSC_ALL_CLIENT_IDS, comp);
 			break;
 
 		case PDS_EVENT_RESET:
 			dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
 				 ecode, eid);
+			pdsc_auxbus_publish(pdsc, PDSC_ALL_CLIENT_IDS, comp);
 			break;
 
 		case PDS_EVENT_XCVR:
@@ -46,6 +48,21 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
 				 ecode, eid);
 			break;
 
+		case PDS_EVENT_CLIENT:
+		{
+			struct pds_core_client_event *ce;
+			union pds_core_notifyq_comp *cc;
+			u16 client_id;
+
+			ce = (struct pds_core_client_event *)comp;
+			cc = (union pds_core_notifyq_comp *)&ce->client_event;
+			client_id = le16_to_cpu(ce->client_id);
+			dev_info(pdsc->dev, "NotifyQ CLIENT %d ecode %d eid %lld cc->ecode %d\n",
+				 client_id, ecode, eid, le16_to_cpu(cc->ecode));
+			pdsc_auxbus_publish(pdsc, client_id, cc);
+			break;
+		}
+
 		default:
 			dev_info(pdsc->dev, "NotifyQ ecode %d eid %lld\n",
 				 ecode, eid);
drivers/net/ethernet/amd/pds_core/auxbus.c +40 -0
index 6d5e7e1d176c..be2b2ab35233 100644
@@ -205,6 +205,46 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *pdsc,
 	return NULL;
 }
 
+static int pdsc_core_match(struct device *dev, const void *data)
+{
+	struct pds_auxiliary_dev *curr_padev;
+	struct pdsc *curr_pdsc;
+	const struct pdsc *pdsc;
+
+	/* Match the core device searching for its clients */
+	curr_padev = container_of(dev, struct pds_auxiliary_dev, aux_dev.dev);
+	curr_pdsc = (struct pdsc *)dev_get_drvdata(curr_padev->aux_dev.dev.parent);
+	pdsc = data;
+
+	if (curr_pdsc == pdsc)
+		return 1;
+
+	return 0;
+}
+
+int pdsc_auxbus_publish(struct pdsc *pdsc, u16 client_id,
+			union pds_core_notifyq_comp *event)
+{
+	struct pds_auxiliary_dev *padev;
+	struct auxiliary_device *aux_dev;
+
+	/* Search aux bus for this core's devices */
+	aux_dev = auxiliary_find_device(NULL, pdsc, pdsc_core_match);
+	while (aux_dev) {
+		padev = container_of(aux_dev, struct pds_auxiliary_dev, aux_dev);
+		if ((padev->client_id == client_id ||
+		     client_id == PDSC_ALL_CLIENT_IDS) &&
+		    padev->event_handler)
+			padev->event_handler(padev, event);
+		put_device(&aux_dev->dev);
+
+		aux_dev = auxiliary_find_device(&aux_dev->dev,
+						pdsc, pdsc_core_match);
+	}
+
+	return 0;
+}
+
 int pdsc_auxbus_dev_add_vf(struct pdsc *pdsc, int vf_id)
 {
 	struct pds_auxiliary_dev *padev;
drivers/net/ethernet/amd/pds_core/core.c +15 -0
index 393ec10ad745..bbbffe358406 100644
@@ -520,6 +520,11 @@ void pdsc_stop(struct pdsc *pdsc)
 
 static void pdsc_fw_down(struct pdsc *pdsc)
 {
+	union pds_core_notifyq_comp reset_event = {
+		.reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
+		.reset.state = 0,
+	};
+
 	mutex_lock(&pdsc->config_lock);
 
 	if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
@@ -528,6 +533,9 @@ static void pdsc_fw_down(struct pdsc *pdsc)
 		return;
 	}
 
+	/* Notify clients of fw_down */
+	pdsc_auxbus_publish(pdsc, PDSC_ALL_CLIENT_IDS, &reset_event);
+
 	pdsc_mask_interrupts(pdsc);
 	pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
 
@@ -536,6 +544,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)
 
 static void pdsc_fw_up(struct pdsc *pdsc)
 {
+	union pds_core_notifyq_comp reset_event = {
+		.reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
+		.reset.state = 1,
+	};
 	int err;
 
 	mutex_lock(&pdsc->config_lock);
@@ -556,6 +568,9 @@ static void pdsc_fw_up(struct pdsc *pdsc)
 
 	mutex_unlock(&pdsc->config_lock);
 
+	/* Notify clients of fw_up */
+	pdsc_auxbus_publish(pdsc, PDSC_ALL_CLIENT_IDS, &reset_event);
+
 	return;
 
 err_out:
drivers/net/ethernet/amd/pds_core/core.h +3 -0
index 7abd2cc4efc7..6f2421187ad5 100644
@@ -303,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);
 void pdsc_stop(struct pdsc *pdsc);
 void pdsc_health_thread(struct work_struct *work);
 
+#define PDSC_ALL_CLIENT_IDS   0xffff
+int pdsc_auxbus_publish(struct pdsc *pdsc, u16 client_id,
+			union pds_core_notifyq_comp *event);
 int pdsc_auxbus_dev_add_vf(struct pdsc *pdsc, int vf_id);
 int pdsc_auxbus_dev_del_vf(struct pdsc *pdsc, int vf_id);
 
Signature · 1 line
2.17.1
#19 lore raw

PATCH v3 14/14 pds_core: Kconfig and pds_core.rst

In reply to Shannon Nelson

Documentation and Kconfig hook for building the driver.

---
 .../device_drivers/ethernet/amd/pds_core.rst  | 150 ++++++++++++++++++
 .../device_drivers/ethernet/index.rst         |   1 +
 drivers/net/ethernet/amd/Kconfig              |  12 ++
 drivers/net/ethernet/amd/Makefile             |   1 +
 4 files changed, 164 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
Documentation/networking/device_drivers/ethernet/amd/pds_core.rst new file +150 -0
new file mode 100644
index 000000000000..defa2f9463f5
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+========================================================
+Linux Driver for the AMD/Pensando(R) DSC adapter family
+========================================================
+
+AMD/Pensando Linux Core driver
+Copyright(c) 2023 Advanced Micro Devices, Inc
+
+Identifying the Adapter
+=======================
+
+To find if one or more AMD/Pensando PCI Core devices are installed on the
+host, check for the PCI devices::
+
+  # lspci -d 1dd8:100c
+  39:00.0 Processing accelerators: Pensando Systems Device 100c
+  3a:00.0 Processing accelerators: Pensando Systems Device 100c
+
+If such devices are listed as above, then the pds_core.ko driver should find
+and configure them for use.  There should be log entries in the kernel
+messages such as these::
+
+  $ dmesg | grep pds_core
+  pds_core 0000:b5:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link)
+  pds_core 0000:b5:00.0: FW: 1.51.0-73
+  pds_core 0000:b6:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link)
+  pds_core 0000:b5:00.0: FW: 1.51.0-73
+
+Driver and firmware version information can be gathered with devlink::
+
+  $ devlink dev info pci/0000:b5:00.0
+  pci/0000:b5:00.0:
+    driver pds_core
+    serial_number FLM18420073
+    versions:
+        fixed:
+          asic.id 0x0
+          asic.rev 0x0
+        running:
+          fw 1.51.0-73
+        stored:
+          fw.goldfw 1.15.9-C-22
+          fw.mainfwa 1.51.0-73
+          fw.mainfwb 1.51.0-57
+
+
+Info versions
+=============
+
+The ``pds_core`` driver reports the following versions
+
+.. list-table:: devlink info versions implemented
+   :widths: 5 5 90
+
+   * - Name
+     - Type
+     - Description
+   * - ``fw``
+     - running
+     - Version of firmware running on the device
+   * - ``fw.goldfw``
+     - stored
+     - Version of firmware stored in the goldfw slot
+   * - ``fw.mainfwa``
+     - stored
+     - Version of firmware stored in the mainfwa slot
+   * - ``fw.mainfwb``
+     - stored
+     - Version of firmware stored in the mainfwb slot
+   * - ``asic.id``
+     - fixed
+     - The ASIC type for this device
+   * - ``asic.rev``
+     - fixed
+     - The revision of the ASIC for this device
+
+
+Parameters
+==========
+
+The ``pds_core`` driver implements the following generic
+parameters for controlling the functionality to be made available
+as auxiliary_bus devices.
+
+.. list-table:: Generic parameters implemented
+   :widths: 5 5 8 82
+
+   * - Name
+     - Mode
+     - Type
+     - Description
+   * - ``enable_vnet``
+     - runtime
+     - Boolean
+     - Enables vDPA functionality through an auxiliary_bus device
+
+
+The ``pds_core`` driver also implements the following driver-specific
+parameters for similar uses, as well as for selecting the next boot firmware:
+
+.. list-table:: Driver-specific parameters implemented
+   :widths: 5 5 8 82
+
+   * - Name
+     - Mode
+     - Type
+     - Description
+   * - ``enable_migration``
+     - runtime
+     - Boolean
+     - Enables Live Migration functionality through an auxiliary_bus device
+
+
+Firmware Management
+===================
+
+The ``flash`` command can update a the DSC firmware.  The downloaded firmware
+will be saved into either of firmware bank 1 or bank 2, whichever is not
+currrently in use, and that bank will be then selected for the next boot.
+The ``fw_bank`` parameter will be updated to reflect this.
+
+Enabling the driver
+===================
+
+The driver is enabled via the standard kernel configuration system,
+using the make command::
+
+  make oldconfig/menuconfig/etc.
+
+The driver is located in the menu structure at:
+
+  -> Device Drivers
+    -> Network device support (NETDEVICES [=y])
+      -> Ethernet driver support
+        -> AMD devices
+          -> AMD/Pensando Ethernet PDS_CORE Support
+
+Support
+=======
+
+For general Linux networking support, please use the netdev mailing
+list, which is monitored by AMD/Pensando personnel::
+
+  [email protected]
+
+For more specific support needs, please use the AMD/Pensando driver support
+email::
+
+  [email protected]
Documentation/networking/device_drivers/ethernet/index.rst +1 -0
index 392969ac88ad..aae0955eb26b 100644
@@ -13,6 +13,7 @@ Contents:
    3com/3c509
    3com/vortex
    amazon/ena
+   amd/pds_core
    altera/altera_tse
    aquantia/atlantic
    chelsio/cxgb
drivers/net/ethernet/amd/Kconfig +12 -0
index ab42f75b9413..235fcacef5c5 100644
@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC
 	bool
 	default n
 
+config PDS_CORE
+	tristate "AMD/Pensando Data Systems Core Device Support"
+	depends on 64BIT && PCI
+	help
+	  This enables the support for the AMD/Pensando Core device family of
+	  adapters.  More specific information on this driver can be
+	  found in
+	  <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>.
+
+	  To compile this driver as a module, choose M here. The module
+	  will be called pds_core.
+
 endif # NET_VENDOR_AMD
drivers/net/ethernet/amd/Makefile +1 -0
index 42742afe9115..2dcfb84731e1 100644
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o
 obj-$(CONFIG_SUN3LANCE) += sun3lance.o
 obj-$(CONFIG_SUNLANCE) += sunlance.o
 obj-$(CONFIG_AMD_XGBE) += xgbe/
+obj-$(CONFIG_PDS_CORE) += pds_core/
Signature · 1 line
2.17.1
#20 lore raw

Re: [PATCH v3 net-next 14/14] pds_core: Kconfig and pds_core.rst

I love your patch! Perhaps something to improve:

Hi Shannon,

I love your patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Shannon-Nelson/devlink-add-enable_migration-parameter/20230218-065953
patch link:    https://lore.kernel.org/r/20230217225558.19837-15-shannon.nelson%40amd.com
patch subject: [PATCH v3 net-next 14/14] pds_core: Kconfig and pds_core.rst
config: sparc-allyesconfig (https://download.01.org/0day-ci/archive/20230218/[email protected]/config)
compiler: sparc64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/18ec7f7bcf0005650ea77647a39a2271f70cf1c2
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Shannon-Nelson/devlink-add-enable_migration-parameter/20230218-065953 git checkout 18ec7f7bcf0005650ea77647a39a2271f70cf1c2
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc SHELL=/bin/bash drivers/net/

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

drivers/net/ethernet/amd/pds_core/adminq.c: In function 'pdsc_process_adminq':

>> drivers/net/ethernet/amd/pds_core/adminq.c:97:13: warning: variable 'index' set but not used [-Wunused-but-set-variable]
      97 |         u32 index;
         |             ^~~~~

vim +/index +97 drivers/net/ethernet/amd/pds_core/adminq.c

b708113c48cc6f Shannon Nelson 2023-02-17   85  
b708113c48cc6f Shannon Nelson 2023-02-17   86  void pdsc_process_adminq(struct pdsc_qcq *qcq)
b708113c48cc6f Shannon Nelson 2023-02-17   87  {
b708113c48cc6f Shannon Nelson 2023-02-17   88  	union pds_core_adminq_comp *comp;
b708113c48cc6f Shannon Nelson 2023-02-17   89  	struct pdsc_queue *q = &qcq->q;
b708113c48cc6f Shannon Nelson 2023-02-17   90  	struct pdsc *pdsc = qcq->pdsc;
b708113c48cc6f Shannon Nelson 2023-02-17   91  	struct pdsc_cq *cq = &qcq->cq;
b708113c48cc6f Shannon Nelson 2023-02-17   92  	struct pdsc_q_info *q_info;
b708113c48cc6f Shannon Nelson 2023-02-17   93  	unsigned long irqflags;
b708113c48cc6f Shannon Nelson 2023-02-17   94  	int nq_work = 0;
b708113c48cc6f Shannon Nelson 2023-02-17   95  	int aq_work = 0;
b708113c48cc6f Shannon Nelson 2023-02-17   96  	int credits;
b708113c48cc6f Shannon Nelson 2023-02-17  @97  	u32 index;
b708113c48cc6f Shannon Nelson 2023-02-17   98  
b708113c48cc6f Shannon Nelson 2023-02-17   99  	/* Check for NotifyQ event */
b708113c48cc6f Shannon Nelson 2023-02-17  100  	nq_work = pdsc_process_notifyq(&pdsc->notifyqcq);
b708113c48cc6f Shannon Nelson 2023-02-17  101  
b708113c48cc6f Shannon Nelson 2023-02-17  102  	/* Check for empty queue, which can happen if the interrupt was
b708113c48cc6f Shannon Nelson 2023-02-17  103  	 * for a NotifyQ event and there are no new AdminQ completions.
b708113c48cc6f Shannon Nelson 2023-02-17  104  	 */
b708113c48cc6f Shannon Nelson 2023-02-17  105  	if (q->tail_idx == q->head_idx)
b708113c48cc6f Shannon Nelson 2023-02-17  106  		goto credits;
b708113c48cc6f Shannon Nelson 2023-02-17  107  
b708113c48cc6f Shannon Nelson 2023-02-17  108  	/* Find the first completion to clean,
b708113c48cc6f Shannon Nelson 2023-02-17  109  	 * run the callback in the related q_info,
b708113c48cc6f Shannon Nelson 2023-02-17  110  	 * and continue while we still match done color
b708113c48cc6f Shannon Nelson 2023-02-17  111  	 */
b708113c48cc6f Shannon Nelson 2023-02-17  112  	spin_lock_irqsave(&pdsc->adminq_lock, irqflags);
b708113c48cc6f Shannon Nelson 2023-02-17  113  	comp = cq->info[cq->tail_idx].comp;
b708113c48cc6f Shannon Nelson 2023-02-17  114  	while (pdsc_color_match(comp->color, cq->done_color)) {
b708113c48cc6f Shannon Nelson 2023-02-17  115  		q_info = &q->info[q->tail_idx];
b708113c48cc6f Shannon Nelson 2023-02-17  116  		index = q->tail_idx;
b708113c48cc6f Shannon Nelson 2023-02-17  117  		q->tail_idx = (q->tail_idx + 1) & (q->num_descs - 1);
b708113c48cc6f Shannon Nelson 2023-02-17  118  
b708113c48cc6f Shannon Nelson 2023-02-17  119  		/* Copy out the completion data */
b708113c48cc6f Shannon Nelson 2023-02-17  120  		memcpy(q_info->dest, comp, sizeof(*comp));
b708113c48cc6f Shannon Nelson 2023-02-17  121  
b708113c48cc6f Shannon Nelson 2023-02-17  122  		complete_all(&q_info->wc->wait_completion);
b708113c48cc6f Shannon Nelson 2023-02-17  123  
b708113c48cc6f Shannon Nelson 2023-02-17  124  		if (cq->tail_idx == cq->num_descs - 1)
b708113c48cc6f Shannon Nelson 2023-02-17  125  			cq->done_color = !cq->done_color;
b708113c48cc6f Shannon Nelson 2023-02-17  126  		cq->tail_idx = (cq->tail_idx + 1) & (cq->num_descs - 1);
b708113c48cc6f Shannon Nelson 2023-02-17  127  		comp = cq->info[cq->tail_idx].comp;
b708113c48cc6f Shannon Nelson 2023-02-17  128  
b708113c48cc6f Shannon Nelson 2023-02-17  129  		aq_work++;
b708113c48cc6f Shannon Nelson 2023-02-17  130  	}
b708113c48cc6f Shannon Nelson 2023-02-17  131  	spin_unlock_irqrestore(&pdsc->adminq_lock, irqflags);
b708113c48cc6f Shannon Nelson 2023-02-17  132  
b708113c48cc6f Shannon Nelson 2023-02-17  133  	qcq->accum_work += aq_work;
b708113c48cc6f Shannon Nelson 2023-02-17  134  
b708113c48cc6f Shannon Nelson 2023-02-17  135  credits:
b708113c48cc6f Shannon Nelson 2023-02-17  136  	/* Return the interrupt credits, one for each completion */
b708113c48cc6f Shannon Nelson 2023-02-17  137  	credits = nq_work + aq_work;
b708113c48cc6f Shannon Nelson 2023-02-17  138  	if (credits)
b708113c48cc6f Shannon Nelson 2023-02-17  139  		pds_core_intr_credits(&pdsc->intr_ctrl[qcq->intx],
b708113c48cc6f Shannon Nelson 2023-02-17  140  				      credits,
b708113c48cc6f Shannon Nelson 2023-02-17  141  				      PDS_CORE_INTR_CRED_REARM);
b708113c48cc6f Shannon Nelson 2023-02-17  142  }
b708113c48cc6f Shannon Nelson 2023-02-17  143  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#21 lore raw
In reply to Shannon Nelson
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Hi,

I didn't look very deeply to this series, but three things caught my attention and IMHO they need to be changed/redesinged before someone can consider to merge it.

1. Use of bus_register_notifier to communicate between auxiliary devices. This whole concept makes aux logic in this driver looks horrid. The idea of auxiliary bus is separate existing device to sub-devices, while every such sub-device is controlled through relevant subsystem. Current implementation challenges this assumption by inventing own logic.
2. devm_* interfaces. It is bad. Please don't use them in such a complex driver.
3. Listen to PCI BOUND_DRIVER event can't be correct either.

Thanks

#22 lore raw
Quoted 23 lines · Leon Romanovsky
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Hi,

I didn't look very deeply to this series, but three things caught my attention and IMHO they need to be changed/redesinged before someone can consider to merge it.

1. Use of bus_register_notifier to communicate between auxiliary devices. This whole concept makes aux logic in this driver looks horrid. The idea of auxiliary bus is separate existing device to sub-devices, while every such sub-device is controlled through relevant subsystem. Current implementation challenges this assumption by inventing own logic.
2. devm_* interfaces. It is bad. Please don't use them in such a complex driver.
3. Listen to PCI BOUND_DRIVER event can't be correct either.

Quoted 1 line

Thanks

Hi Leon,

Thanks for your comments. I’ll respond to 1 and 3 together.

Quoted 2 lines

1. Use of bus_register_notifier to communicate between auxiliary devices.
3. Listen to PCI BOUND_DRIVER event can't be correct either

We’re only using the notifier for the core driver to know when to create and destroy auxiliary devices, not for communicate between auxiliary devices or drivers – I agree, that would be ugly.

We want to create the auxiliary device after a VF PCI device is bound to a driver (BUS_NOTIFY_BOUND_DRIVER), and remove that auxiliary device just before a VF is unbound from its PCI driver (BUS_NOTIFY_UNBIND_DRIVER); bus_notify_register gives us access to these messages. I believe this is not too far off from other examples that can be seen in vfio/pci/vfio_pci_core, net/ethernet/ibm/emac, and net/mctp.

Early on we were creating and deleting the auxiliary devices at the same time as calling sriov_enable() and sriov_disable() but found that when the user was doing unbind/bind operations we could end up with in-operative clients and occasional kernel panics because the devices and drivers were out-of-sync. Listening for the bind/unbind events allows the pds_core driver to make sure the auxiliary devices serving each of the VF drivers are kept in sync with the state of the VF PCI drivers.

Quoted 1 line

2. devm_* interfaces

Can you elaborate a bit on why you see using the devm interfaces as bad?
I don’t see the code complexity being any different than using the
non-devm interfaces, and these give the added protection of making sure to not leak resources on driver removals, whether clean or on error. We are using these allocations for longer term driver structures, not for ephemeral short-term use buffers or fast-path operations, so the overhead should remain minimal. It seems to me this is the advertised use as described in the devres notes under Documentation.

Thanks,
sln

#23 lore raw
Quoted 44 lines · Shannon Nelson
Quoted 23 lines · Leon Romanovsky
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Hi,

I didn't look very deeply to this series, but three things caught my attention and IMHO they need to be changed/redesinged before someone can consider to merge it.

1. Use of bus_register_notifier to communicate between auxiliary devices. This whole concept makes aux logic in this driver looks horrid. The idea of auxiliary bus is separate existing device to sub-devices, while every such sub-device is controlled through relevant subsystem. Current implementation challenges this assumption by inventing own logic.
2. devm_* interfaces. It is bad. Please don't use them in such a complex driver.
3. Listen to PCI BOUND_DRIVER event can't be correct either.

Quoted 1 line

Thanks

Hi Leon,

Thanks for your comments. I’ll respond to 1 and 3 together.

Quoted 2 lines

1. Use of bus_register_notifier to communicate between auxiliary devices.
3. Listen to PCI BOUND_DRIVER event can't be correct either

We’re only using the notifier for the core driver to know when to create and destroy auxiliary devices, not for communicate between auxiliary devices or drivers – I agree, that would be ugly.

We want to create the auxiliary device after a VF PCI device is bound to a driver (BUS_NOTIFY_BOUND_DRIVER), and remove that auxiliary device just before a VF is unbound from its PCI driver (BUS_NOTIFY_UNBIND_DRIVER); bus_notify_register gives us access to these messages. I believe this is not too far off from other examples that can be seen in vfio/pci/vfio_pci_core, net/ethernet/ibm/emac, and net/mctp.

Quoted 7 lines

Early on we were creating and deleting the auxiliary devices at the same time as calling sriov_enable() and sriov_disable() but found that when the user was doing unbind/bind operations we could end up with in-operative clients and occasional kernel panics because the devices and drivers were out-of-sync. Listening for the bind/unbind events allows the pds_core driver to make sure the auxiliary devices serving each of the VF drivers are kept in sync with the state of the VF PCI drivers.

Auxiliary devices are intended to statically re-partition existing physical devices. You are not supposed to create such devices on the fly. So once you create VF physical device, it should already have aux device created too.

This is traditional driver device model which you should follow and not reinvent your own schema, just to overcome bugs.

Additionally, it is unclear how much we should invest in this review given the fact that pds VFIO series was NACKed.

Quoted 4 lines
Quoted 3 lines

2. devm_* interfaces

Quoted 8 lines

Can you elaborate a bit on why you see using the devm interfaces as bad? I don’t see the code complexity being any different than using the non-devm interfaces, and these give the added protection of making sure to not leak resources on driver removals, whether clean or on error. We are using these allocations for longer term driver structures, not for ephemeral short-term use buffers or fast-path operations, so the overhead should remain minimal. It seems to me this is the advertised use as described in the devres notes under Documentation.

We had a very interesting talk in last LPC and overall agreement across various subsystem maintainers was to stay away from devm_* interfaces.
https://lpc.events/event/16/contributions/1227/

We prefer explicit nature of kzalloc/kfree instead of implicit approach of devm_kzalloc.

Thanks

Quoted 5 lines

Thanks,
sln

#24 lore raw
Quoted 53 lines · Leon Romanovsky
Quoted 44 lines · Shannon Nelson
Quoted 23 lines · Leon Romanovsky
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Hi,

I didn't look very deeply to this series, but three things caught my attention and IMHO they need to be changed/redesinged before someone can consider to merge it.

1. Use of bus_register_notifier to communicate between auxiliary devices. This whole concept makes aux logic in this driver looks horrid. The idea of auxiliary bus is separate existing device to sub-devices, while every such sub-device is controlled through relevant subsystem. Current implementation challenges this assumption by inventing own logic.
2. devm_* interfaces. It is bad. Please don't use them in such a complex driver.
3. Listen to PCI BOUND_DRIVER event can't be correct either.

Quoted 1 line

Thanks

Hi Leon,

Thanks for your comments. I’ll respond to 1 and 3 together.

Quoted 2 lines

1. Use of bus_register_notifier to communicate between auxiliary devices.
3. Listen to PCI BOUND_DRIVER event can't be correct either

We’re only using the notifier for the core driver to know when to create and destroy auxiliary devices, not for communicate between auxiliary devices or drivers – I agree, that would be ugly.

We want to create the auxiliary device after a VF PCI device is bound to a driver (BUS_NOTIFY_BOUND_DRIVER), and remove that auxiliary device just before a VF is unbound from its PCI driver (BUS_NOTIFY_UNBIND_DRIVER); bus_notify_register gives us access to these messages. I believe this is not too far off from other examples that can be seen in vfio/pci/vfio_pci_core, net/ethernet/ibm/emac, and net/mctp.

Quoted 7 lines

Early on we were creating and deleting the auxiliary devices at the same time as calling sriov_enable() and sriov_disable() but found that when the user was doing unbind/bind operations we could end up with in-operative clients and occasional kernel panics because the devices and drivers were out-of-sync. Listening for the bind/unbind events allows the pds_core driver to make sure the auxiliary devices serving each of the VF drivers are kept in sync with the state of the VF PCI drivers.

Quoted 6 lines

Auxiliary devices are intended to statically re-partition existing physical devices. You are not supposed to create such devices on the fly. So once you create VF physical device, it should already have aux device created too.

This is traditional driver device model which you should follow and not reinvent your own schema, just to overcome bugs.

Not so much a “bug”, I think, as an incomplete model which we addressed by using a hotplug model with the aux-devices. We felt we were following a part of the device model, but perhaps from the wrong angle: hot-plugging the aux-devices, which worked nicely to simplify the aux-driver/pci-driver relationship. However, I think the hole we fell into was expecting the client to be tied to a pci device on the other end, and this shouldn’t be a factor in the core’s management of the aux device.

So, the aux-device built by the core needs to remain the constant, which is what we originally had: the aux devices created at pci_enable_sriov() time and torn down at pci_disable_sriov(). It’s easy enough to go back to that model, and that makes sense.

Meanwhile, for those clients that do rely on the existence of a pci device, would you see that as a proper use of a bus listener to have the aux-driver subscribe and listen for its bind/unbind events?

Quoted 3 lines

Additionally, it is unclear how much we should invest in this review given the fact that pds VFIO series was NACKed.

We haven’t given up on that one yet, and we have a vDPA client in mind as well possibly one or two others.

Quoted 14 lines
Quoted 5 lines
Quoted 4 lines

2. devm_* interfaces

Quoted 8 lines

Can you elaborate a bit on why you see using the devm interfaces as bad? I don’t see the code complexity being any different than using the non-devm interfaces, and these give the added protection of making sure to not leak resources on driver removals, whether clean or on error. We are using these allocations for longer term driver structures, not for ephemeral short-term use buffers or fast-path operations, so the overhead should remain minimal. It seems to me this is the advertised use as described in the devres notes under Documentation.

Quoted 6 lines

We had a very interesting talk in last LPC and overall agreement across various subsystem maintainers was to stay away from devm_* interfaces.
https://lpc.events/event/16/contributions/1227/

We prefer explicit nature of kzalloc/kfree instead of implicit approach of devm_kzalloc.

Interesting – thanks, I missed that presentation. Sure, we can pull back on this.

Thanks for your comments.

sln

#25 lore raw
Quoted 58 lines · Shannon Nelson
Quoted 57 lines · Leon Romanovsky
Quoted 44 lines · Shannon Nelson
Quoted 23 lines · Leon Romanovsky
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Hi,

I didn't look very deeply to this series, but three things caught my attention and IMHO they need to be changed/redesinged before someone can consider to merge it.

1. Use of bus_register_notifier to communicate between auxiliary devices. This whole concept makes aux logic in this driver looks horrid. The idea of auxiliary bus is separate existing device to sub-devices, while every such sub-device is controlled through relevant subsystem. Current implementation challenges this assumption by inventing own logic.
2. devm_* interfaces. It is bad. Please don't use them in such a complex driver.
3. Listen to PCI BOUND_DRIVER event can't be correct either.

Quoted 1 line

Thanks

Hi Leon,

Thanks for your comments. I’ll respond to 1 and 3 together.

Quoted 2 lines

1. Use of bus_register_notifier to communicate between auxiliary devices.
3. Listen to PCI BOUND_DRIVER event can't be correct either

We’re only using the notifier for the core driver to know when to create and destroy auxiliary devices, not for communicate between auxiliary devices or drivers – I agree, that would be ugly.

We want to create the auxiliary device after a VF PCI device is bound to a driver (BUS_NOTIFY_BOUND_DRIVER), and remove that auxiliary device just before a VF is unbound from its PCI driver (BUS_NOTIFY_UNBIND_DRIVER); bus_notify_register gives us access to these messages. I believe this is not too far off from other examples that can be seen in vfio/pci/vfio_pci_core, net/ethernet/ibm/emac, and net/mctp.

Quoted 7 lines

Early on we were creating and deleting the auxiliary devices at the same time as calling sriov_enable() and sriov_disable() but found that when the user was doing unbind/bind operations we could end up with in-operative clients and occasional kernel panics because the devices and drivers were out-of-sync. Listening for the bind/unbind events allows the pds_core driver to make sure the auxiliary devices serving each of the VF drivers are kept in sync with the state of the VF PCI drivers.

Auxiliary devices are intended to statically re-partition existing physical devices. You are not supposed to create such devices on the fly. So once you create VF physical device, it should already have aux device created too.

Quoted 8 lines
Quoted 2 lines

This is traditional driver device model which you should follow and not reinvent your own schema, just to overcome bugs.

Not so much a “bug”, I think, as an incomplete model which we addressed by using a hotplug model with the aux-devices. We felt we were following a part of the device model, but perhaps from the wrong angle: hot-plugging the aux-devices, which worked nicely to simplify the aux-driver/pci-driver relationship.

By looking (very briefly) on your pds_core and pds_vfio code, your notification scheme worked by chance as it lacks proper PCI and driver locks. You probably didn't stress enough PCI PF teardown.

Quoted 8 lines

However, I think the hole we fell into was expecting the
client to be tied to a pci device on the other end, and this shouldn’t be a factor in the core’s management of the aux device.

So, the aux-device built by the core needs to remain the constant, which is what we originally had: the aux devices created at pci_enable_sriov() time and torn down at pci_disable_sriov(). It’s easy enough to go back to that model, and that makes sense.

Not really, aux device creation is supposed to be in .probe() call of your PCI function, whenever it is PF or VF. After call to pci_enable_sriov(), the HW will add VFs, which will cause to PCI core to attempt and load pds_core .probe() callback.

Quoted 4 lines

Meanwhile, for those clients that do rely on the existence of a pci device, would you see that as a proper use of a bus listener to have the aux-driver subscribe and listen for its bind/unbind events?

No, these events must go.

Quoted 8 lines
Quoted 5 lines

Additionally, it is unclear how much we should invest in this review given the fact that pds VFIO series was NACKed.

We haven’t given up on that one yet, and we have a vDPA client in mind as well possibly one or two others.

I don't know about your plans and judge by what I saw in ML, VFIO was NACKed for second time already on the same ground.

Thanks

Quoted 30 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

2. devm_* interfaces

Quoted 8 lines

Can you elaborate a bit on why you see using the devm interfaces as bad? I don’t see the code complexity being any different than using the non-devm interfaces, and these give the added protection of making sure to not leak resources on driver removals, whether clean or on error. We are using these allocations for longer term driver structures, not for ephemeral short-term use buffers or fast-path operations, so the overhead should remain minimal. It seems to me this is the advertised use as described in the devres notes under Documentation.

Quoted 6 lines

We had a very interesting talk in last LPC and overall agreement across various subsystem maintainers was to stay away from devm_* interfaces.
https://lpc.events/event/16/contributions/1227/

We prefer explicit nature of kzalloc/kfree instead of implicit approach of devm_kzalloc.

Interesting – thanks, I missed that presentation. Sure, we can pull back on this.

Thanks for your comments.

sln

#26 lore raw
In reply to Shannon Nelson
Quoted 6 lines · Shannon Nelson

Summary:
--------
This patchset implements new driver for use with the AMD/Pensando Distributed Services Card (DSC), intended to provide core configuration services through the auxiliary_bus for VFio and vDPA feature specific drivers.

Thank you all for your responses and commentary on this initial review of our PDS drivers, we appreciate the constructive input.

Obviously, the organization of our driver parts, PCI vs auxiliary and PF vs VF, doesn’t match what most folks were expecting to see. The primary issue seems to be that the modules responsible for the VF’s functions (e.g. the VFio or vDPA modules) are set up to be both PCI and auxiliary drivers, which doesn’t match other designs, and presents potential coordination/locking concerns.

The expectation seems to be that the one Core module should handle both the VF and PF PCI driver work and from there set up the needed auxiliary devices, and things like vDPA modules should just be auxiliary drivers on one end and supply their respective features on the other end. Meanwhile, because the nature of VFio is to be a PCI driver, pds_vfio needs to have its aux stuff removed and needs to access the pds_core adminq directly by getting a pointer to the PF’s data, as done in other similar drivers.

We think this reorganization of parts will resolve the complexity, bind/unbind, and most other points of discussion.

I expect to get a first pass at this re-org done over the next few days and have an RFC for viewing sometime in the next week.

Cheers,
sln