pds core and vdpa drivers
Subsystems (7)
cover pds core and vdpa drivers
Summary:
--------
This is a first draft patchset of a pair of new drivers for use with the AMD/Pensando Distributed Services Card (DSC), intended to work along side the existing ionic Ethernet driver to provide support of devices for better virtualization support. These drivers work together using the auxiliary_bus for client drivers (pds_vdpa) to use the core configuration services (pds_core).
This large patchset is both drivers combined in order to give a full RFC view and can be split into separate pds_core and pds_vdpa patchsets in the future.
Detail:
-------
AMD/Pensando is making available a new set of devices for supporting vDPA, VFio, and potentially other features in the Distributed Services Card (DSC). These features are implemented through a PF that serves as a Core device for controlling and configuring its VF devices. These VF devices have separate drivers that use the auxiliary_bus to work through the Core device as the control path.
Currently, the DSC supports standard ethernet operations using the ionic driver. This is not replaced by the Core-based devices - these new devices are in addition to the existing Ethernet device. Typical DSC configurations will include both PDS devices and Ionic Eth devices.
The Core device is a new PCI PF device managed by a new driver 'pds_core'. It sets up a small representer netdev for managing the associated VFs, and sets up auxiliary_bus devices for each VF for communicating with the drivers for the VF devices. The VFs may be for VFio/NVMe or vDPA, and other services in the future; these VF types are selected as part of the DSC internal FW configurations, which is out of the scope of this patchset. The Core device sets up devlink parameters for enabling available feature sets.
Once a feature set is enabled, auxiliary_bus devices are created for each VF that supports the feature. These auxiliary_bus devices are named by their feature plus VF PCI bdf so the auxiliary device driver can find its related VF PCI driver instance. The VF's driver then connects to and uses this auxiliary_device to do control path configuration of the feature through the PF device.
A cheap ASCII diagram of vDPA instance looks something like this and can then be used with the vdpa kernel module to provide devices for virtio_vdpa kernel module for host interfaces, vhost_vdpa kernel module for interfaces exported into your favorite VM.
,----------.
| vdpa |
'----------'
|
vdpa_dev
ctl data
| ||
pds_core.vDPA.2305 <---+ | ||
| | | ||
netdev | | | ||
| | | | ||
.------------. .------------.
| pds_core | | pds_vdpa |
'------------' '------------'
|| ||
09:00.0 09:00.1
== PCI =========================================================
|| ||
.----------. .----------.
,-----| PF |-----------| VF |-------,
| '----------' -----------' |
| DSC |
| |
-------------------------------------------------The pds_core driver is targeted to reside in drivers/net/ethernet/pensando/pds_core and the pds_vdpa driver lands in drivers/vdpa/pds. There are some shared include files placed in include/linux/pds, which seemed reasonable at the time, but I've recently seen suggestions of putting files like this under include/net instead, so that may be up for some discussion.
I appreciate any and all time folks can spend reviewing and commenting.
Thanks,
sln
Shannon Nelson (19):
pds_core: initial framework for pds_core driver
pds_core: add devcmd device interfaces
pds_core: health timer and workqueue
pds_core: set up device and adminq
pds_core: Add adminq processing and commands
pds_core: add FW update feature to devlink
pds_core: set up the VIF definitions and defaults
pds_core: initial VF configuration
pds_core: add auxiliary_bus devices
pds_core: devlink params for enabling VIF support
pds_core: add the aux client API
pds_core: publish events to the clients
pds_core: Kconfig and pds_core.rst
pds_vdpa: Add new PCI VF device for PDS vDPA services
pds_vdpa: virtio bar setup for vdpa
pds_vdpa: add auxiliary driver
pds_vdpa: add vdpa config client commands
pds_vdpa: add support for vdpa and vdpamgmt interfaces
pds_vdpa: add Kconfig entry and pds_vdpa.rst
.../ethernet/pensando/pds_core.rst | 162 ++++ .../ethernet/pensando/pds_vdpa.rst | 85 ++ MAINTAINERS | 4 +- drivers/net/ethernet/pensando/Kconfig | 12 + .../net/ethernet/pensando/pds_core/Makefile | 15 + .../net/ethernet/pensando/pds_core/adminq.c | 299 +++++++ .../net/ethernet/pensando/pds_core/auxbus.c | 306 +++++++ drivers/net/ethernet/pensando/pds_core/core.c | 616 ++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.h | 342 ++++++++ .../net/ethernet/pensando/pds_core/debugfs.c | 262 ++++++ drivers/net/ethernet/pensando/pds_core/dev.c | 403 +++++++++ .../net/ethernet/pensando/pds_core/devlink.c | 310 +++++++ drivers/net/ethernet/pensando/pds_core/fw.c | 192 +++++ drivers/net/ethernet/pensando/pds_core/main.c | 440 ++++++++++ .../net/ethernet/pensando/pds_core/netdev.c | 504 +++++++++++ drivers/vdpa/Kconfig | 7 + drivers/vdpa/pds/Makefile | 11 + drivers/vdpa/pds/aux_drv.c | 156 ++++ drivers/vdpa/pds/aux_drv.h | 28 + drivers/vdpa/pds/cmds.c | 266 ++++++ drivers/vdpa/pds/cmds.h | 17 + drivers/vdpa/pds/debugfs.c | 234 +++++ drivers/vdpa/pds/debugfs.h | 28 + drivers/vdpa/pds/pci_drv.c | 172 ++++ drivers/vdpa/pds/pci_drv.h | 49 ++ drivers/vdpa/pds/vdpa_dev.c | 796 ++++++++++++++++++ drivers/vdpa/pds/vdpa_dev.h | 60 ++ drivers/vdpa/pds/virtio_pci.c | 283 +++++++ include/linux/pds/pds_adminq.h | 643 ++++++++++++++ include/linux/pds/pds_auxbus.h | 88 ++ include/linux/pds/pds_common.h | 99 +++ include/linux/pds/pds_core_if.h | 582 +++++++++++++ include/linux/pds/pds_intr.h | 160 ++++ include/linux/pds/pds_vdpa.h | 219 +++++ 34 files changed, 7849 insertions(+), 1 deletion(-) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_core.rst create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst create mode 100644 drivers/net/ethernet/pensando/pds_core/Makefile create mode 100644 drivers/net/ethernet/pensando/pds_core/adminq.c create mode 100644 drivers/net/ethernet/pensando/pds_core/auxbus.c create mode 100644 drivers/net/ethernet/pensando/pds_core/core.c create mode 100644 drivers/net/ethernet/pensando/pds_core/core.h create mode 100644 drivers/net/ethernet/pensando/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/pensando/pds_core/dev.c create mode 100644 drivers/net/ethernet/pensando/pds_core/devlink.c create mode 100644 drivers/net/ethernet/pensando/pds_core/fw.c create mode 100644 drivers/net/ethernet/pensando/pds_core/main.c create mode 100644 drivers/net/ethernet/pensando/pds_core/netdev.c create mode 100644 drivers/vdpa/pds/Makefile create mode 100644 drivers/vdpa/pds/aux_drv.c create mode 100644 drivers/vdpa/pds/aux_drv.h create mode 100644 drivers/vdpa/pds/cmds.c create mode 100644 drivers/vdpa/pds/cmds.h create mode 100644 drivers/vdpa/pds/debugfs.c create mode 100644 drivers/vdpa/pds/debugfs.h create mode 100644 drivers/vdpa/pds/pci_drv.c create mode 100644 drivers/vdpa/pds/pci_drv.h create mode 100644 drivers/vdpa/pds/vdpa_dev.c create mode 100644 drivers/vdpa/pds/vdpa_dev.h create mode 100644 drivers/vdpa/pds/virtio_pci.c create mode 100644 include/linux/pds/pds_adminq.h create mode 100644 include/linux/pds/pds_auxbus.h create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core_if.h create mode 100644 include/linux/pds/pds_intr.h create mode 100644 include/linux/pds/pds_vdpa.h
Signature · 1 line
2.17.1
RFC PATCH 1/19 pds_core: initial framework for pds_core driver
This is the initial PCI driver framework for the new pds_core device driver and its family of client drivers. This does the very basics of registering for the new PCI device 1dd8:100c, setting up debugfs entries, and registering with devlink.
The new PCI device id has not made it to the official PCI ID Repository yet, but will soon be registered there.
--- .../net/ethernet/pensando/pds_core/Makefile | 9 + drivers/net/ethernet/pensando/pds_core/core.h | 68 ++ .../net/ethernet/pensando/pds_core/debugfs.c | 47 ++ .../net/ethernet/pensando/pds_core/devlink.c | 55 ++ drivers/net/ethernet/pensando/pds_core/main.c | 263 ++++++++ include/linux/pds/pds_common.h | 13 + include/linux/pds/pds_core_if.h | 581 ++++++++++++++++++ 7 files changed, 1036 insertions(+) create mode 100644 drivers/net/ethernet/pensando/pds_core/Makefile create mode 100644 drivers/net/ethernet/pensando/pds_core/core.h create mode 100644 drivers/net/ethernet/pensando/pds_core/debugfs.c create mode 100644 drivers/net/ethernet/pensando/pds_core/devlink.c create mode 100644 drivers/net/ethernet/pensando/pds_core/main.c create mode 100644 include/linux/pds/pds_common.h create mode 100644 include/linux/pds/pds_core_if.h
drivers/net/ethernet/pensando/pds_core/Makefile new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2022 Pensando Systems, Inc + +obj-$(CONFIG_PDS_CORE) := pds_core.o + +pds_core-y := main.o \ + devlink.o + +pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/pensando/pds_core/core.h new file +68 -0
68 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/debugfs.c new file +47 -0
47 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/devlink.c new file +55 -0
55 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/main.c new file +263 -0
263 lines not on this page · raw message
include/linux/pds/pds_common.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright (c) 2022 Pensando Systems, Inc. All rights reserved. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core" + +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) + +#endif /* _PDS_COMMON_H_ */
include/linux/pds/pds_core_if.h new file +581 -0
581 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 2/19 pds_core: add devcmd device interfaces
In reply to Shannon NelsonThe devcmd interface is the basic connection to the device through the PCI BAR for low level identification and command services. This does the early device initialization and finds the identity data, and adds devcmd routines to be used by later driver bits.
--- .../net/ethernet/pensando/pds_core/Makefile | 4 +- drivers/net/ethernet/pensando/pds_core/core.c | 43 ++ drivers/net/ethernet/pensando/pds_core/core.h | 58 +++ .../net/ethernet/pensando/pds_core/debugfs.c | 67 +++ drivers/net/ethernet/pensando/pds_core/dev.c | 400 ++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/main.c | 30 ++ include/linux/pds/pds_common.h | 67 +++ include/linux/pds/pds_intr.h | 160 +++++++ 8 files changed, 828 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/pensando/pds_core/core.c create mode 100644 drivers/net/ethernet/pensando/pds_core/dev.c create mode 100644 include/linux/pds/pds_intr.h
drivers/net/ethernet/pensando/pds_core/Makefile +3 -1
@@ -4,6 +4,8 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ - devlink.o + devlink.o \ + dev.o \ + core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/pensando/pds_core/core.c new file +43 -0
43 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/core.h +58 -0
100 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/debugfs.c +67 -0
71 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/dev.c new file +400 -0
400 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/main.c +30 -0
@@ -189,6 +189,15 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)if (err) goto err_out_pci_disable_device; +/* PDS device setup */ +mutex_init (&pdsc->devcmd_lock); +mutex_init (&pdsc->config_lock); + +mutex_lock (&pdsc->config_lock); + err =pdsc_setup (pdsc,PDSC_SETUP_INIT ); +if (err) + goto err_out_unmap_bars; +/* publish devlink device */ err =pdsc_dl_register (pdsc);if (err) {
@@ -196,10 +205,21 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_out; } +mutex_unlock (&pdsc->config_lock); + + pdsc->fw_generation =PDS_CORE_FW_STS_F_GENERATION & +ioread8 (&pdsc->info_regs->fw_status); +clear_bit (PDSC_S_INITING_DRIVER , &pdsc->state);return 0 ; err_out: +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); +err_out_unmap_bars: +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock); +pci_free_irq_vectors (pdev);pci_clear_master (pdev);pdsc_unmap_bars (pdsc);pci_release_regions (pdev);
@@ -224,10 +244,20 @@ static void pdsc_remove(struct pci_dev *pdev)*/ pdsc_dl_unregister (pdsc); +/* Now we can lock it up and tear it down */ +mutex_lock (&pdsc->config_lock); +set_bit (PDSC_S_STOPPING_DRIVER , &pdsc->state); +/* Device teardown */ +pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); +pdsc_debugfs_del_dev (pdsc); +mutex_unlock (&pdsc->config_lock); +mutex_destroy (&pdsc->config_lock); +mutex_destroy (&pdsc->devcmd_lock);ida_free (&pdsc_pf_ida, pdsc->id);/* PCI teardown */ +pci_free_irq_vectors (pdev);pci_clear_master (pdev);pdsc_unmap_bars (pdsc);pci_release_regions (pdev);
include/linux/pds/pds_common.h +67 -0
71 lines not on this page · raw message
include/linux/pds/pds_intr.h new file +160 -0
160 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 3/19 pds_core: health timer and workqueue
In reply to Shannon NelsonAdd in the periodic health check and the related workqueue, as well as the handlers for when a FW reset is seen.
--- drivers/net/ethernet/pensando/pds_core/core.c | 60 +++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.h | 9 +++ drivers/net/ethernet/pensando/pds_core/dev.c | 3 + drivers/net/ethernet/pensando/pds_core/main.c | 50 ++++++++++++++++ 4 files changed, 122 insertions(+)
drivers/net/ethernet/pensando/pds_core/core.c +60 -0
@@ -41,3 +41,63 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } + +static void pdsc_fw_down (struct pdsc *pdsc) +{ +mutex_lock (&pdsc->config_lock); + +if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: already happening\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); + +mutex_unlock (&pdsc->config_lock); +} + +static void pdsc_fw_up (struct pdsc *pdsc) +{ +int err; + +mutex_lock (&pdsc->config_lock); + +if (!test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +dev_err (pdsc->dev,"%s: fw not dead\n" , __func__); +mutex_unlock (&pdsc->config_lock); +return ; + } + + err =pdsc_setup (pdsc,PDSC_SETUP_RECOVERY ); +if (err) + goto err_out; + +mutex_unlock (&pdsc->config_lock); + +return ; + +err_out: +pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY ); +mutex_unlock (&pdsc->config_lock); +} + +void pdsc_health_thread (struct work_struct *work) +{ + struct pdsc *pdsc = container_of(work, struct pdsc, health_work); +bool healthy; + + healthy =pdsc_is_fw_good (pdsc); +dev_dbg (pdsc->dev,"%s: health %d fw_status %#02x fw_heartbeat %d\n" , + __func__, healthy, pdsc->fw_status, pdsc->last_hb); + +if (test_bit (PDSC_S_FW_DEAD , &pdsc->state)) { +if (healthy) +pdsc_fw_up (pdsc); + }else { +if (!healthy) +pdsc_fw_down (pdsc); + } + + pdsc->fw_generation = pdsc->fw_status &PDS_CORE_FW_STS_F_GENERATION ; +}
drivers/net/ethernet/pensando/pds_core/core.h +9 -0
@@ -12,6 +12,8 @@#include <linux/pds/pds_intr.h> #define PDSC_DRV_DESCRIPTION "Pensando Core PF Driver" + +#define PDSC_WATCHDOG_SECS 5#define PDSC_TEARDOWN_RECOVERY false#define PDSC_TEARDOWN_REMOVING true#define PDSC_SETUP_RECOVERY false
@@ -64,12 +66,17 @@ struct pdsc {u8 fw_generation;unsigned long last_fw_time;u32 last_hb; +struct timer_list wdtimer; +unsigned int wdtimer_period; +struct work_struct health_work;struct pdsc_devinfo dev_info;struct pds_core_dev_identity dev_ident;unsigned int nintrs;struct pdsc_intr_info *intr_info;/* array of nintrs elements */ +struct workqueue_struct *wq; +unsigned int devcmd_timeout;struct mutex devcmd_lock;/* lock for dev_cmd operations */ struct mutex config_lock;/* lock for configuration operations */
@@ -82,6 +89,7 @@ struct pdsc {u64 __iomem *kern_dbpage; }; +void pdsc_queue_health_check (struct pdsc *pdsc);void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num);struct pdsc *pdsc_dl_alloc (struct device *dev);
@@ -122,5 +130,6 @@ int pdsc_dev_init(struct pdsc *pdsc);int pdsc_setup (struct pdsc *pdsc,bool init);void pdsc_teardown (struct pdsc *pdsc,bool removing); +void pdsc_health_thread (struct work_struct *work);#endif /* _PDSC_H_ */
drivers/net/ethernet/pensando/pds_core/dev.c +3 -0
@@ -179,6 +179,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, err =pdsc_devcmd_wait (pdsc, max_seconds);memcpy_fromio (comp, &pdsc->cmd_regs->comp,sizeof (*comp)); +if (err == -ENXIO || err == -ETIMEDOUT ) +pdsc_queue_health_check (pdsc); +return err; }
drivers/net/ethernet/pensando/pds_core/main.c +50 -0
89 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 4/19 pds_core: set up device and adminq
In reply to Shannon NelsonSet up the basic adminq and notifyq services. These are used mostly by the client drivers for feature configuration.
--- drivers/net/ethernet/pensando/pds_core/core.c | 444 +++++++++++- drivers/net/ethernet/pensando/pds_core/core.h | 149 ++++ .../net/ethernet/pensando/pds_core/debugfs.c | 125 ++++ .../net/ethernet/pensando/pds_core/devlink.c | 55 +- drivers/net/ethernet/pensando/pds_core/main.c | 14 + include/linux/pds/pds_adminq.h | 641 ++++++++++++++++++ 6 files changed, 1424 insertions(+), 4 deletions(-) create mode 100644 include/linux/pds/pds_adminq.h
drivers/net/ethernet/pensando/pds_core/core.c +441 -3
@@ -10,8 +10,368 @@#include "core.h" +#include <linux/pds/pds_adminq.h> + +void pdsc_work_thread (struct work_struct *work) +{ +/* stub */ +} + +irqreturn_t pdsc_adminq_isr (int irq,void *data) +{ +/* stub */ +return IRQ_HANDLED ; +} + +void pdsc_intr_free (struct pdsc *pdsc,int index) +{ +struct pdsc_intr_info *intr_info; + +if (index >= pdsc->nintrs || index <0 ) { +WARN (true,"bad intr index %d\n" , index); +return ; + } + + intr_info = &pdsc->intr_info[index]; +if (!intr_info->vector) +return ; +dev_dbg (pdsc->dev,"%s: idx %d vec %d name %s\n" , + __func__, index, intr_info->vector, intr_info->name); + +pds_core_intr_mask (&pdsc->intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); +pds_core_intr_clean (&pdsc->intr_ctrl[index]); + +devm_free_irq (pdsc->dev, intr_info->vector, intr_info->data); + +memset (intr_info,0 ,sizeof (*intr_info)); +} + +int pdsc_intr_alloc (struct pdsc *pdsc,char *name, +irq_handler_t handler,void *data) +{ +struct pdsc_intr_info *intr_info; +unsigned int index; +int err; + +/* Find the first available interrupt */ +for (index =0 ; index < pdsc->nintrs; index++) +if (!pdsc->intr_info[index].vector) +break ; +if (index >= pdsc->nintrs) { +dev_warn (pdsc->dev,"%s: no intr, index=%d nintrs=%d\n" , + __func__, index, pdsc->nintrs); +return -ENOSPC ; + } + +pds_core_intr_clean_flags (&pdsc->intr_ctrl[index], +PDS_CORE_INTR_CRED_RESET_COALESCE ); + + intr_info = &pdsc->intr_info[index]; + + intr_info->index = index; + intr_info->data = data; +strscpy (intr_info->name, name,sizeof (intr_info->name)); + +/* Get the OS vector number for the interrupt */ + err =pci_irq_vector (pdsc->pdev, index); +if (err <0 ) { +dev_err (pdsc->dev,"failed to get intr vector index %d: %pe\n" , + index,ERR_PTR (err)); + goto err_out_free_intr; + } + intr_info->vector = err; + +/* Init the device's intr mask */ +pds_core_intr_clean (&pdsc->intr_ctrl[index]); +pds_core_intr_mask_assert (&pdsc->intr_ctrl[index],1 ); +pds_core_intr_mask (&pdsc->intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); + +/* Register the isr with a name */ + err =devm_request_irq (pdsc->dev, intr_info->vector, + handler,0 , intr_info->name, data); +if (err) { +dev_err (pdsc->dev,"failed to get intr irq vector %d: %pe\n" , + intr_info->vector,ERR_PTR (err)); + goto err_out_free_intr; + } + +return index; + +err_out_free_intr: +pdsc_intr_free (pdsc, index); +return err; +} + +static void pdsc_qcq_intr_free (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +if (!(qcq->flags &PDS_CORE_QCQ_F_INTR ) || + qcq->intx ==PDS_CORE_INTR_INDEX_NOT_ASSIGNED ) +return ; + +pdsc_intr_free (pdsc, qcq->intx); + qcq->intx =PDS_CORE_INTR_INDEX_NOT_ASSIGNED ; +} + +static int pdsc_qcq_intr_alloc (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +char name[PDSC_INTR_NAME_MAX_SZ ]; +int index; + +if (!(qcq->flags &PDS_CORE_QCQ_F_INTR )) { + qcq->intx =PDS_CORE_INTR_INDEX_NOT_ASSIGNED ; +return 0 ; + } + +snprintf (name,sizeof (name), +"%s-%d-%s" ,PDS_CORE_DRV_NAME , pdsc->pdev->bus->number, qcq->q.name); + index =pdsc_intr_alloc (pdsc, name, pdsc_adminq_isr, qcq); +if (index <0 ) +return index; + qcq->intx = index; + +return 0 ; +} + +void pdsc_qcq_free (struct pdsc *pdsc,struct pdsc_qcq *qcq) +{ +struct device *dev = pdsc->dev; + +if (!(qcq && qcq->pdsc)) +return ; + +pdsc_debugfs_del_qcq (qcq); + +pdsc_qcq_intr_free (pdsc, qcq); + +if (qcq->q_base) { +dmam_free_coherent (dev, qcq->q_size, + qcq->q_base, qcq->q_base_pa); + qcq->q_base =NULL ; + qcq->q_base_pa =0 ; + } + +if (qcq->cq_base) { +dmam_free_coherent (dev, qcq->cq_size, qcq->cq_base, qcq->cq_base_pa); + qcq->cq_base =NULL ; + qcq->cq_base_pa =0 ; + } + +if (qcq->cq.info) { +vfree (qcq->cq.info); + qcq->cq.info =NULL ; + } +if (qcq->q.info) { +vfree (qcq->q.info); + qcq->q.info =NULL ; + } + + qcq->pdsc =NULL ; +memset (&qcq->q,0 ,sizeof (qcq->q)); +memset (&qcq->cq,0 ,sizeof (qcq->cq)); +} + +static void pdsc_q_map (struct pdsc_queue *q,void *base,dma_addr_t base_pa) +{ +struct pdsc_q_info *cur; +unsigned int i; + + q->base = base; + q->base_pa = base_pa; + +for (i =0 , cur = q->info; i < q->num_descs; i++, cur++) + cur->desc = base + (i * q->desc_size); +} + +static void pdsc_cq_map (struct pdsc_cq *cq,void *base,dma_addr_t base_pa) +{ +struct pdsc_cq_info *cur; +unsigned int i; + + cq->base = base; + cq->base_pa = base_pa; + +for (i =0 , cur = cq->info; i < cq->num_descs; i++, cur++) + cur->comp = base + (i * cq->desc_size); +} + +int pdsc_qcq_alloc (struct pdsc *pdsc,unsigned int type,unsigned int index, +const char *name,unsigned int flags,unsigned int num_descs, +unsigned int desc_size,unsigned int cq_desc_size, +unsigned int pid,struct pdsc_qcq *qcq) +{ +struct device *dev = pdsc->dev; +dma_addr_t cq_base_pa =0 ; +dma_addr_t q_base_pa =0 ; +void *q_base, *cq_base; +int err; + + qcq->q.info =vzalloc (num_descs *sizeof (*qcq->q.info)); +if (!qcq->q.info) { +dev_err (dev,"Cannot allocate %s queue info\n" , name); + err = -ENOMEM ; + goto err_out; + } + + qcq->pdsc = pdsc; + qcq->flags = flags; +INIT_WORK (&qcq->work, pdsc_work_thread); + + qcq->q.type = type; + qcq->q.index = index; + qcq->q.num_descs = num_descs; + qcq->q.desc_size = desc_size; + qcq->q.tail_idx =0 ; + qcq->q.head_idx =0 ; + qcq->q.pid = pid; +snprintf (qcq->q.name,sizeof (qcq->q.name),"%s%u" , name, index); + + err =pdsc_qcq_intr_alloc (pdsc, qcq); +if (err) + goto err_out_free_q_info; + + qcq->cq.info =vzalloc (num_descs *sizeof (*qcq->cq.info)); +if (!qcq->cq.info) { +dev_err (dev,"Cannot allocate %s completion queue info\n" , name); + err = -ENOMEM ; + goto err_out_free_irq; + } + + qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx]; + qcq->cq.num_descs = num_descs; + qcq->cq.desc_size = cq_desc_size; + qcq->cq.tail_idx =0 ; + qcq->cq.done_color =1 ; + +if (flags &PDS_CORE_QCQ_F_NOTIFYQ ) { +/* q & cq need to be contiguous in case of notifyq */ + qcq->q_size =PAGE_SIZE +ALIGN (num_descs * desc_size,PAGE_SIZE ) + +ALIGN (num_descs * cq_desc_size,PAGE_SIZE ); + qcq->q_base =dmam_alloc_coherent (dev, qcq->q_size + qcq->cq_size, + &qcq->q_base_pa, +GFP_KERNEL ); +if (!qcq->q_base) { +dev_err (dev,"Cannot allocate %s qcq DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_cq_info; + } + q_base =PTR_ALIGN (qcq->q_base,PAGE_SIZE ); + q_base_pa =ALIGN (qcq->q_base_pa,PAGE_SIZE ); +pdsc_q_map (&qcq->q, q_base, q_base_pa); + + cq_base =PTR_ALIGN (q_base + +ALIGN (num_descs * desc_size,PAGE_SIZE ),PAGE_SIZE ); + cq_base_pa =ALIGN (qcq->q_base_pa + +ALIGN (num_descs * desc_size,PAGE_SIZE ),PAGE_SIZE ); + + }else { +/* q DMA descriptors */ + qcq->q_size =PAGE_SIZE + (num_descs * desc_size); + qcq->q_base =dmam_alloc_coherent (dev, qcq->q_size, + &qcq->q_base_pa, +GFP_KERNEL ); +if (!qcq->q_base) { +dev_err (dev,"Cannot allocate %s queue DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_cq_info; + } + q_base =PTR_ALIGN (qcq->q_base,PAGE_SIZE ); + q_base_pa =ALIGN (qcq->q_base_pa,PAGE_SIZE ); +pdsc_q_map (&qcq->q, q_base, q_base_pa); + +/* cq DMA descriptors */ + qcq->cq_size =PAGE_SIZE + (num_descs * cq_desc_size); + qcq->cq_base =dmam_alloc_coherent (dev, qcq->cq_size, + &qcq->cq_base_pa, +GFP_KERNEL ); +if (!qcq->cq_base) { +dev_err (dev,"Cannot allocate %s cq DMA memory\n" , name); + err = -ENOMEM ; + goto err_out_free_q; + } + cq_base =PTR_ALIGN (qcq->cq_base,PAGE_SIZE ); + cq_base_pa =ALIGN (qcq->cq_base_pa,PAGE_SIZE ); + } + +pdsc_cq_map (&qcq->cq, cq_base, cq_base_pa); + qcq->cq.bound_q = &qcq->q; + +pdsc_debugfs_add_qcq (pdsc, qcq); + +return 0 ; + +err_out_free_q: +dmam_free_coherent (dev, qcq->q_size, qcq->q_base, qcq->q_base_pa); +err_out_free_cq_info: +vfree (qcq->cq.info); +err_out_free_irq: +pdsc_qcq_intr_free (pdsc, qcq); +err_out_free_q_info: +vfree (qcq->q.info); +memset (qcq,0 ,sizeof (*qcq)); +err_out: +dev_err (dev,"qcq alloc of %s%d failed %d\n" , name, index, err); +return err; +} + +static int pdsc_core_init (struct pdsc *pdsc) +{ +union pds_core_dev_comp comp = {0 }; +union pds_core_dev_cmd cmd = { + .init.opcode =PDS_CORE_CMD_INIT , + }; +struct pds_core_dev_init_data_out cido; +struct pds_core_dev_init_data_in cidi; +u32 dbid_count; +u32 dbpage_num; +size_t sz; +int err; + + cidi.adminq_q_base =cpu_to_le64 (pdsc->adminqcq.q_base_pa); + cidi.adminq_cq_base =cpu_to_le64 (pdsc->adminqcq.cq_base_pa); + cidi.notifyq_cq_base =cpu_to_le64 (pdsc->notifyqcq.cq.base_pa); + cidi.flags =cpu_to_le32 (PDS_CORE_QINIT_F_IRQ |PDS_CORE_QINIT_F_ENA ); + cidi.intr_index =cpu_to_le16 (pdsc->adminqcq.intx); + cidi.adminq_ring_size =ilog2 (pdsc->adminqcq.q.num_descs); + cidi.notifyq_ring_size =ilog2 (pdsc->notifyqcq.q.num_descs); + +mutex_lock (&pdsc->devcmd_lock); + + sz =min_t (size_t,sizeof (cidi),sizeof (pdsc->cmd_regs->data)); +memcpy_toio (&pdsc->cmd_regs->data, &cidi, sz); + + err =pdsc_devcmd_locked (pdsc, &cmd, &comp, pdsc->devcmd_timeout); + + sz =min_t (size_t,sizeof (cido),sizeof (pdsc->cmd_regs->data)); +memcpy_fromio (&cido, &pdsc->cmd_regs->data, sz); + +mutex_unlock (&pdsc->devcmd_lock); + + pdsc->hw_index =le32_to_cpu (cido.core_hw_index); + + dbid_count =le32_to_cpu (pdsc->dev_ident.ndbpgs_per_lif); + dbpage_num = pdsc->hw_index * dbid_count; + pdsc->kern_dbpage =pdsc_map_dbpage (pdsc, dbpage_num); +if (!pdsc->kern_dbpage) { +dev_err (pdsc->dev,"Cannot map dbpage, aborting\n" ); +return -ENOMEM ; + } + + pdsc->adminqcq.q.hw_type = cido.adminq_hw_type; + pdsc->adminqcq.q.hw_index =le32_to_cpu (cido.adminq_hw_index); + pdsc->adminqcq.q.dbval =PDS_CORE_DBELL_QID (pdsc->adminqcq.q.hw_index); + + pdsc->notifyqcq.q.hw_type = cido.notifyq_hw_type; + pdsc->notifyqcq.q.hw_index =le32_to_cpu (cido.notifyq_hw_index); + pdsc->notifyqcq.q.dbval =PDS_CORE_DBELL_QID (pdsc->notifyqcq.q.hw_index); + + pdsc->last_eid =0 ; + +return err; +} +int pdsc_setup (struct pdsc *pdsc,bool init) { +int numdescs;int err =0 ;if (init)
@@ -21,17 +381,60 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (err)return err; +/* Scale the descriptor ring length based on number of CPUs and VFs */ + numdescs =max_t (int,PDSC_ADMINQ_MIN_LENGTH ,num_online_cpus ()); + numdescs +=2 *pci_sriov_get_totalvfs (pdsc->pdev); + numdescs =roundup_pow_of_two (numdescs); + err =pdsc_qcq_alloc (pdsc,PDS_CORE_QTYPE_ADMINQ ,0 ,"adminq" , +PDS_CORE_QCQ_F_CORE |PDS_CORE_QCQ_F_INTR , + numdescs, +sizeof (union pds_core_adminq_cmd ), +sizeof (union pds_core_adminq_comp ), +0 , &pdsc->adminqcq); +if (err) + goto err_out_teardown; + + err =pdsc_qcq_alloc (pdsc,PDS_CORE_QTYPE_NOTIFYQ ,0 ,"notifyq" , +PDS_CORE_QCQ_F_NOTIFYQ , +PDSC_NOTIFYQ_LENGTH , +sizeof (struct pds_core_notifyq_cmd ), +sizeof (union pds_core_notifyq_comp ), +0 , &pdsc->notifyqcq); +if (err) + goto err_out_teardown; + +/* NotifyQ rides on the AdminQ interrupt */ + pdsc->notifyqcq.intx = pdsc->adminqcq.intx; + +/* Set up the Core with the AdminQ and NotifyQ info */ + err =pdsc_core_init (pdsc); +if (err) + goto err_out_teardown; +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ; + +err_out_teardown: +pdsc_teardown (pdsc, init); +return err; }void pdsc_teardown (struct pdsc *pdsc,bool removing) { +int i; +pdsc_devcmd_reset (pdsc); +pdsc_qcq_free (pdsc, &pdsc->notifyqcq); +pdsc_qcq_free (pdsc, &pdsc->adminqcq); -if (removing && pdsc->intr_info) { -devm_kfree (pdsc->dev, pdsc->intr_info); - pdsc->intr_info =NULL ; +if (pdsc->intr_info) { +for (i =0 ; i < pdsc->nintrs; i++) +pdsc_intr_free (pdsc, i); + +if (removing) { +devm_kfree (pdsc->dev, pdsc->intr_info); + pdsc->intr_info =NULL ; + } }if (pdsc->kern_dbpage) {
@@ -42,6 +445,36 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)set_bit (PDSC_S_FW_DEAD , &pdsc->state); } +int pdsc_start (struct pdsc *pdsc) +{ +pds_core_intr_mask (&pdsc->intr_ctrl[pdsc->adminqcq.intx], +PDS_CORE_INTR_MASK_CLEAR ); + +return 0 ; +} + +static void pdsc_mask_interrupts (struct pdsc *pdsc) +{ +int i; + +if (!pdsc->intr_info) +return ; + +/* Mask interrupts that are in use */ +for (i =0 ; i < pdsc->nintrs; i++) +if (pdsc->intr_info[i].vector) +pds_core_intr_mask (&pdsc->intr_ctrl[i], +PDS_CORE_INTR_MASK_SET ); +} + +void pdsc_stop (struct pdsc *pdsc) +{ +if (pdsc->wq) +flush_workqueue (pdsc->wq); + +pdsc_mask_interrupts (pdsc); +} +static void pdsc_fw_down (struct pdsc *pdsc) {mutex_lock (&pdsc->config_lock);
@@ -52,6 +485,7 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );mutex_unlock (&pdsc->config_lock);
@@ -73,6 +507,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; + err =pdsc_start (pdsc); +if (err) + goto err_out; +mutex_unlock (&pdsc->config_lock);return ;
drivers/net/ethernet/pensando/pds_core/core.h +149 -0
206 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/debugfs.c +125 -0
129 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/devlink.c +54 -1
63 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/main.c +14 -0
@@ -158,6 +158,13 @@ static int pdsc_map_bars(struct pdsc *pdsc)return err; } +void __iomem *pdsc_map_dbpage (struct pdsc *pdsc,int page_num) +{ +return pci_iomap_range (pdsc->pdev, + pdsc->bars[PDS_CORE_PCI_BAR_DBELL ].res_index, + (u64 )page_num <<PAGE_SHIFT ,PAGE_SIZE ); +} +static DEFINE_IDA (pdsc_pf_ida);#define PDSC_WQ_NAME_LEN 24
@@ -227,11 +234,15 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)/* PDS device setup */ mutex_init (&pdsc->devcmd_lock);mutex_init (&pdsc->config_lock); +spin_lock_init (&pdsc->adminq_lock);mutex_lock (&pdsc->config_lock); err =pdsc_setup (pdsc,PDSC_SETUP_INIT );if (err) goto err_out_unmap_bars; + err =pdsc_start (pdsc); +if (err) + goto err_out_teardown;/* publish devlink device */ err =pdsc_dl_register (pdsc);
@@ -251,6 +262,8 @@ static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return 0 ; err_out: +pdsc_stop (pdsc); +err_out_teardown:pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING ); err_out_unmap_bars:del_timer_sync (&pdsc->wdtimer);
@@ -299,6 +312,7 @@ static void pdsc_remove(struct pci_dev *pdev) }/* Device teardown */ +pdsc_stop (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_REMOVING );pdsc_debugfs_del_dev (pdsc);mutex_unlock (&pdsc->config_lock);
include/linux/pds/pds_adminq.h new file +641 -0
641 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 5/19 pds_core: Add adminq processing and commands
In reply to Shannon NelsonAdd the service routines for submitting and processing the adminq messages and for handling notifyq events.
--- .../net/ethernet/pensando/pds_core/Makefile | 1 + .../net/ethernet/pensando/pds_core/adminq.c | 282 ++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.c | 11 - drivers/net/ethernet/pensando/pds_core/core.h | 6 + include/linux/pds/pds_adminq.h | 2 + 5 files changed, 291 insertions(+), 11 deletions(-) create mode 100644 drivers/net/ethernet/pensando/pds_core/adminq.c
drivers/net/ethernet/pensando/pds_core/Makefile +1 -0
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ dev.o \ + adminq.o \ core.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/pensando/pds_core/adminq.c new file +282 -0
282 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/core.c +0 -11
@@ -12,17 +12,6 @@#include <linux/pds/pds_adminq.h> -void pdsc_work_thread (struct work_struct *work) -{ -/* stub */ -} - -irqreturn_t pdsc_adminq_isr (int irq,void *data) -{ -/* stub */ -return IRQ_HANDLED ; -} -void pdsc_intr_free (struct pdsc *pdsc,int index) {struct pdsc_intr_info *intr_info;
drivers/net/ethernet/pensando/pds_core/core.h +6 -0
@@ -263,6 +263,12 @@ int pds_devcmd_vf_start(struct pdsc *pdsc);int pdsc_dev_reinit (struct pdsc *pdsc);int pdsc_dev_init (struct pdsc *pdsc); +int pdsc_adminq_post (struct pdsc *pdsc, +struct pdsc_qcq *qcq, +union pds_core_adminq_cmd *cmd, +union pds_core_adminq_comp *comp, +bool fast_poll); +int pdsc_intr_alloc (struct pdsc *pdsc,char *name,irq_handler_t handler,void *data);void pdsc_intr_free (struct pdsc *pdsc,int index);
include/linux/pds/pds_adminq.h +2 -0
@@ -4,6 +4,8 @@#ifndef _PDS_CORE_ADMINQ_H_#define _PDS_CORE_ADMINQ_H_ +#define PDSC_ADMINQ_MAX_POLL_INTERVAL 256 +enum pds_core_adminq_flags {PDS_AQ_FLAG_FASTPOLL =BIT (1 ),/* poll for completion at 1ms intervals */ };
Signature · 1 line
2.17.1
RFC PATCH 6/19 pds_core: add FW update feature to devlink
In reply to Shannon NelsonAdd in the support for doing firmware updates, and for selecting the next firmware image to boot on, and tie them into the devlink flash and parameter handling. The FW flash is the same as in the ionic driver. However, this device has the ability to report what is in the firmware slots on the device and allows you to select the slot to use on the next device boot.
--- .../net/ethernet/pensando/pds_core/Makefile | 3 +- drivers/net/ethernet/pensando/pds_core/core.h | 8 + .../net/ethernet/pensando/pds_core/devlink.c | 103 ++++++++++ drivers/net/ethernet/pensando/pds_core/fw.c | 192 ++++++++++++++++++ 4 files changed, 305 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/pensando/pds_core/fw.c
drivers/net/ethernet/pensando/pds_core/Makefile +2 -1
@@ -7,6 +7,7 @@ pds_core-y := main.o \ devlink.o \ dev.o \ adminq.o \ - core.o + core.o \ + fw.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/pensando/pds_core/core.h +8 -0
@@ -123,6 +123,11 @@ struct pdsc_qcq {struct dentry *dentry; }; +enum pdsc_devlink_param_id { +PDSC_DEVLINK_PARAM_ID_BASE =DEVLINK_PARAM_GENERIC_ID_MAX , +PDSC_DEVLINK_PARAM_ID_FW_BOOT , +}; +/* No state flags set means we are in a steady running state */ enum pdsc_state_flags {PDSC_S_FW_DEAD ,/* fw stopped, waiting for startup or recovery */
@@ -287,4 +292,7 @@ void pdsc_process_adminq(struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data); +int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw, +struct netlink_ext_ack *extack); +#endif /* _PDSC_H_ */
drivers/net/ethernet/pensando/pds_core/devlink.c +103 -0
125 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/fw.c new file +192 -0
192 lines not on this page · raw message
Signature · 1 line
2.17.1
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
Quoted 6 lines · On Fri, 18 Nov 2022 14:56:43 -0800 Shannon Nelson wrote:
Add in the support for doing firmware updates, and for selecting the next firmware image to boot on, and tie them into the devlink flash and parameter handling. The FW flash is the same as in the ionic driver. However, this device has the ability to report what is in the firmware slots on the device and allows you to select the slot to use on the next device boot.
This is hardly vendor specific. Intel does a similar thing, IIUC. Please work on a common interface.
Re: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
Quoted 10 lines · Jakub Kicinski
Quoted 6 lines · On Fri, 18 Nov 2022 14:56:43 -0800 Shannon Nelson wrote:
Add in the support for doing firmware updates, and for selecting the next firmware image to boot on, and tie them into the devlink flash and parameter handling. The FW flash is the same as in the ionic driver. However, this device has the ability to report what is in the firmware slots on the device and allows you to select the slot to use on the next device boot.
This is hardly vendor specific. Intel does a similar thing, IIUC. Please work on a common interface.
I don't think Intel selects which FW image to boot, but it looks like mlxsw and nfp use the PARAM_GENERIC_FW_LOAD_POLICY to select between DRIVER, FLASH, or DISK. Shall I add a couple of generic SLOT_x items to the enum devlink_param_fw_load_policy_value and use this API? For example:
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_0, DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_1, DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_2, DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_3,
I could then modify the devlink dev info printed to refer to fw_slot_0, fw.slot_1, and fw.slot_2 instead of our vendor specific names.
Cheers,
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
Quoted 9 lines · On Mon, 28 Nov 2022 14:25:46 -0800 Shannon Nelson wrote:
I don't think Intel selects which FW image to boot, but it looks like mlxsw and nfp use the
PARAM_GENERIC_FW_LOAD_POLICYto select between DRIVER, FLASH, or DISK. Shall I add a couple of genericSLOT_xitems to the enumdevlink_param_fw_load_policy_valueand use this API? For example:
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_0,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_1,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_2,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_3,
Not the worst idea, although I presume normal FW flashing should switch between slots to activate the new image by default? Which means the action of fw flashing would alter the policy set by the user. A little awkward from an API purist standpoint.
I'd just expose the active "bank" via netlink directly.
Quoted 2 lines
I could then modify the devlink dev info printed to refer to
fw_slot_0, fw.slot_1, and fw.slot_2 instead of our vendor specific names.
Jake, didn't you have a similar capability in ice?
Knowing my memory I may have acquiesced to something in another driver already. That said - I think it's cleaner if we just list the stored versions per bank, no?
Re: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
Quoted 11 lines · Jakub Kicinski
Quoted 9 lines · On Mon, 28 Nov 2022 14:25:46 -0800 Shannon Nelson wrote:
I don't think Intel selects which FW image to boot, but it looks like mlxsw and nfp use the
PARAM_GENERIC_FW_LOAD_POLICYto select between DRIVER, FLASH, or DISK. Shall I add a couple of genericSLOT_xitems to the enumdevlink_param_fw_load_policy_valueand use this API? For example:
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_0,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_1,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_2,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_3,
Quoted 4 lines
Not the worst idea, although I presume normal FW flashing should switch between slots to activate the new image by default? Which means the action of fw flashing would alter the policy set by the user. A little awkward from an API purist standpoint.
Yes, the action of flashing will set the new bank/slot to use on the next boot. However, we have the ability to select from multiple valid images and we want to pass this flexibility to the user rather than force them to go through a whole flash sequence just to get to the other bank.
Quoted 11 lines
I'd just expose the active "bank" via netlink directly.
Quoted 2 lines
I could then modify the devlink dev info printed to refer to
fw_slot_0, fw.slot_1, and fw.slot_2 instead of our vendor specific names.Jake, didn't you have a similar capability in ice?
Knowing my memory I may have acquiesced to something in another driver already. That said - I think it's cleaner if we just list the stored versions per bank, no?
We are listing the stored images in the devlink dev info output, just want to let the user choose which of those valid images to use next.
Cheers,
sln
RE: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
Quoted 13 lines
-----Original Message-----
From: Shannon Nelson <[email protected]>
Sent: Monday, November 28, 2022 3:46 PM
To: Jakub Kicinski <[email protected]>
Cc: Shannon Nelson <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Keller, Jacob E <[email protected]>
Subject: Re: [RFC PATCH net-next 06/19]pds_core: add FW update feature to devlink
Quoted 17 lines
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Mon, 28 Nov 2022 14:25:46 -0800 Shannon Nelson wrote:
I don't think Intel selects which FW image to boot, but it looks like mlxsw and nfp use the
PARAM_GENERIC_FW_LOAD_POLICYto select between DRIVER, FLASH, or DISK. Shall I add a couple of genericSLOT_xitems to the enumdevlink_param_fw_load_policy_valueand use this API? Forexample:
Quoted 10 lines
Quoted 5 lines
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_0,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_1,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_2,
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_3,Not the worst idea, although I presume normal FW flashing should switch between slots to activate the new image by default? Which means the action of fw flashing would alter the policy set by the user. A little awkward from an API purist standpoint.
This could potentially be handled by having DELVINK_PARAM_FW_LOAD_POLICY_FLASH be the automatic "select best version", and if a user has set a manual value then don't allow flashing until a reboot or the value is set back to POLICY_FLASH?
Quoted 19 lines
Yes, the action of flashing will set the new bank/slot to use on the next boot. However, we have the ability to select from multiple valid images and we want to pass this flexibility to the user rather than force them to go through a whole flash sequence just to get to the other bank.
Quoted 11 lines
I'd just expose the active "bank" via netlink directly.
Quoted 2 lines
I could then modify the devlink dev info printed to refer to
fw_slot_0, fw.slot_1, and fw.slot_2 instead of our vendor specific names.Jake, didn't you have a similar capability in ice?
Knowing my memory I may have acquiesced to something in another driver already. That said - I think it's cleaner if we just list the stored versions per bank, no?
Quoted 5 lines
We are listing the stored images in the devlink dev info output, just want to let the user choose which of those valid images to use next.
Cheers,
sln
Technically I think we could do something similar in ice to switch between the banks, at least as long as there is a valid image in the bank. The big trick is that I am not sure we can verify ahead of time whether we have a valid image and if you happen to boot into an invalid or blank image. There is some recovery firmware that should activate in that case, but I think our current driver doesn't implement enough of a recovery mode to actually handle this case to allow user to switch back.
Still, I think the ability to select the bank is valuable, and finding the right way to expose it is good.
Thanks,
Jake
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [RFC PATCH net-next 06/19] pds_core: add FW update feature to devlink
In reply to Jakub KicinskiQuoted 29 lines
-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Monday, November 28, 2022 3:33 PM
To: Shannon Nelson <[email protected]>
Cc: Shannon Nelson <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Keller, Jacob E <[email protected]>
Subject: Re: [RFC PATCH net-next 06/19]pds_core: add FW update feature to devlinkQuoted 9 lines · On Mon, 28 Nov 2022 14:25:46 -0800 Shannon Nelson wrote:
I don't think Intel selects which FW image to boot, but it looks like mlxsw and nfp use the
PARAM_GENERIC_FW_LOAD_POLICYto select between DRIVER, FLASH, or DISK. Shall I add a couple of genericSLOT_xitems to the enumdevlink_param_fw_load_policy_valueand use this API? For example:
DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_0,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_1,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_2,DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_3,Not the worst idea, although I presume normal FW flashing should switch between slots to activate the new image by default? Which means the action of fw flashing would alter the policy set by the user. A little awkward from an API purist standpoint.
Quoted 7 lines
I'd just expose the active "bank" via netlink directly.
Quoted 2 lines
I could then modify the devlink dev info printed to refer to
fw_slot_0, fw.slot_1, and fw.slot_2 instead of our vendor specific names.Jake, didn't you have a similar capability in ice?
We have two banks of flash, the active bank, and an inactive bank used for updates. We can determine the active bank from the Shadow RAM contents which are generated as the EMP firmware boots up.
Quoted 3 lines
Knowing my memory I may have acquiesced to something in another driver already. That said - I think it's cleaner if we just list the stored versions per bank, no?
I think it would make sense to store them per bank and make the bank number some index instead of something separate as like this DEVLINK_PARAM_FW_LOAD_POLICY_VALUE_SLOT_<X> where each <X> makes a separate parameter.
Currently devlink info reports "stored" and "active", which aligns with our current use of the active vs inactive flash bank. We could be explicit and indicate which bank it is, though its a bit tricky since most of the firmware interface deals with it in terms of "active" and "inactive" rather than the absolute position of "bank 0 or bank 1".
Especially if another device has more than 2 banks I think its a good extension to devlink info, and we could probably get away with something like a new info attribute that specifies which bank index it is, and an attribute which indicates whether its active or not.
Thanks,
Jake
RFC PATCH 7/19 pds_core: set up the VIF definitions and defaults
In reply to Shannon NelsonThe Virtual Interfaces (VIFs) supported by the DSC's configuration (VFio Live Migration, vDPA, etc) are reported in the dev_ident struct.
--- drivers/net/ethernet/pensando/pds_core/core.c | 54 +++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.h | 14 +++++ .../net/ethernet/pensando/pds_core/debugfs.c | 23 ++++++++ include/linux/pds/pds_common.h | 19 +++++++ 4 files changed, 110 insertions(+)
drivers/net/ethernet/pensando/pds_core/core.c +54 -0
@@ -358,6 +358,47 @@ static int pdsc_core_init(struct pdsc *pdsc)return err; } +static struct pdsc_viftype pdsc_viftype_defaults[] = { + [PDS_DEV_TYPE_VDPA ] = { .name =PDS_DEV_TYPE_VDPA_STR , + .vif_id =PDS_DEV_TYPE_VDPA , + .dl_id =DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET }, + [PDS_DEV_TYPE_LM ] = { .name =PDS_DEV_TYPE_LM_STR , + .vif_id =PDS_DEV_TYPE_LM , + .dl_id =PDSC_DEVLINK_PARAM_ID_LM }, + [PDS_DEV_TYPE_MAX ] = {0 } +}; + +static int pdsc_viftypes_init (struct pdsc *pdsc) +{ +enum pds_core_vif_types vt; + + pdsc->viftype_status =devm_kzalloc (pdsc->dev, +sizeof (pdsc_viftype_defaults), +GFP_KERNEL ); +if (!pdsc->viftype_status) +return -ENOMEM ; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +bool vt_support; + +if (!pdsc_viftype_defaults[vt].name) +continue ; + +/* Grab the defaults */ + pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt]; + +/* See what the Core device has for support */ + vt_support = !!le16_to_cpu (pdsc->dev_ident.vif_types[vt]); +dev_dbg (pdsc->dev,"VIF %s is %ssupported\n" , + pdsc->viftype_status[vt].name, + vt_support ?"" :"not " ); + + pdsc->viftype_status[vt].supported = vt_support; + } + +return 0 ; +} +int pdsc_setup (struct pdsc *pdsc,bool init) {int numdescs;
@@ -400,6 +441,14 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (err) goto err_out_teardown; +/* Set up the VIFs */ + err =pdsc_viftypes_init (pdsc); +if (err) + goto err_out_teardown; + +if (init) +pdsc_debugfs_add_viftype (pdsc); +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ;
@@ -416,6 +465,11 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)pdsc_qcq_free (pdsc, &pdsc->notifyqcq);pdsc_qcq_free (pdsc, &pdsc->adminqcq); +if (pdsc->viftype_status) { +devm_kfree (pdsc->dev, pdsc->viftype_status); + pdsc->viftype_status =NULL ; + } +if (pdsc->intr_info) {for (i =0 ; i < pdsc->nintrs; i++)pdsc_intr_free (pdsc, i);
drivers/net/ethernet/pensando/pds_core/core.h +14 -0
@@ -123,8 +123,18 @@ struct pdsc_qcq {struct dentry *dentry; }; +struct pdsc_viftype { +char *name; +bool supported; +bool enabled; +int dl_id; +int vif_id; +struct pds_auxiliary_dev *padev; +}; +enum pdsc_devlink_param_id {PDSC_DEVLINK_PARAM_ID_BASE =DEVLINK_PARAM_GENERIC_ID_MAX , +PDSC_DEVLINK_PARAM_ID_LM ,PDSC_DEVLINK_PARAM_ID_FW_BOOT , };
@@ -178,6 +188,7 @@ struct pdsc {struct pdsc_qcq adminqcq;struct pdsc_qcq notifyqcq;u64 last_eid; +struct pdsc_viftype *viftype_status; };/** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -232,6 +243,7 @@ struct pdsc *pdsc_dl_alloc(struct device *dev);void pdsc_dl_free (struct pdsc *pdsc);int pdsc_dl_register (struct pdsc *pdsc);void pdsc_dl_unregister (struct pdsc *pdsc); +int pdsc_dl_vif_add (struct pdsc *pdsc,enum pds_core_vif_types vt,const char *name);#ifdef CONFIG_DEBUG_FS void pdsc_debugfs_create (void );
@@ -239,6 +251,7 @@ void pdsc_debugfs_destroy(void);void pdsc_debugfs_add_dev (struct pdsc *pdsc);void pdsc_debugfs_del_dev (struct pdsc *pdsc);void pdsc_debugfs_add_ident (struct pdsc *pdsc); +void pdsc_debugfs_add_viftype (struct pdsc *pdsc);void pdsc_debugfs_add_irqs (struct pdsc *pdsc);void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq);void pdsc_debugfs_del_qcq (struct pdsc_qcq *qcq);
@@ -248,6 +261,7 @@ static inline void pdsc_debugfs_destroy(void) { }static inline void pdsc_debugfs_add_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_del_dev (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_ident (struct pdsc *pdsc) { } +static inline void pdsc_debugfs_add_viftype (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_irqs (struct pdsc *pdsc) { }static inline void pdsc_debugfs_add_qcq (struct pdsc *pdsc,struct pdsc_qcq *qcq) { }static inline void pdsc_debugfs_del_qcq (struct pdsc_qcq *qcq) { }
drivers/net/ethernet/pensando/pds_core/debugfs.c +23 -0
@@ -82,6 +82,29 @@ void pdsc_debugfs_add_ident(struct pdsc *pdsc)debugfs_create_file ("identity" ,0400 , pdsc->dentry, pdsc, &identity_fops); } +static int viftype_show (struct seq_file *seq,void *v) +{ +struct pdsc *pdsc = seq->private; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].name) +continue ; + +seq_printf (seq,"%s\t%d supported %d enabled\n" , + pdsc->viftype_status[vt].name, + pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (viftype); + +void pdsc_debugfs_add_viftype (struct pdsc *pdsc) +{ +debugfs_create_file ("viftypes" ,0400 , pdsc->dentry, pdsc, &viftype_fops); +} +static int irqs_show (struct seq_file *seq,void *v) {struct pdsc *pdsc = seq->private;
include/linux/pds/pds_common.h +19 -0
@@ -50,6 +50,25 @@ enum pds_core_driver_type { PDS_DRIVER_ESXI = 6, }; +enum pds_core_vif_types { +PDS_DEV_TYPE_CORE =0 , +PDS_DEV_TYPE_VDPA =1 , +PDS_DEV_TYPE_VFIO =2 , +PDS_DEV_TYPE_ETH =3 , +PDS_DEV_TYPE_RDMA =4 , +PDS_DEV_TYPE_LM =5 , + +/* new ones added before this line */ +PDS_DEV_TYPE_MAX =16 /* don't change - used in struct size */ +}; + +#define PDS_DEV_TYPE_CORE_STR "Core" +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_DEV_TYPE_VFIO_STR "VFio" +#define PDS_DEV_TYPE_ETH_STR "Eth" +#define PDS_DEV_TYPE_RDMA_STR "RDMA" +#define PDS_DEV_TYPE_LM_STR "LM" +/* PDSC interface uses identity version 1 and PDSC uses 2 */ #define PDSC_IDENTITY_VERSION_1 1#define PDSC_IDENTITY_VERSION_2 2
Signature · 1 line
2.17.1
RFC PATCH 8/19 pds_core: initial VF configuration
In reply to Shannon NelsonIn order to manage the VFs associated with thie Core PF we need a PF netdev. This netdev is a simple representor to make available the "ip link set <if> vf ..." commands that are useful for setting attributes used by the vDPA VF. There is no packet handling in this netdev as the Core device has no Tx/Rx queues.
--- .../net/ethernet/pensando/pds_core/Makefile | 1 + drivers/net/ethernet/pensando/pds_core/core.c | 17 + drivers/net/ethernet/pensando/pds_core/core.h | 24 + drivers/net/ethernet/pensando/pds_core/main.c | 65 +++ .../net/ethernet/pensando/pds_core/netdev.c | 504 ++++++++++++++++++ 5 files changed, 611 insertions(+) create mode 100644 drivers/net/ethernet/pensando/pds_core/netdev.c
drivers/net/ethernet/pensando/pds_core/Makefile +1 -0
@@ -8,6 +8,7 @@ pds_core-y := main.o \ dev.o \ adminq.o \ core.o \ + netdev.o \ fw.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/pensando/pds_core/core.c +17 -0
@@ -449,6 +449,12 @@ int pdsc_setup(struct pdsc *pdsc, bool init)if (init)pdsc_debugfs_add_viftype (pdsc); +if (init) { + err =pdsc_init_netdev (pdsc); +if (err) + goto err_out_teardown; + } +clear_bit (PDSC_S_FW_DEAD , &pdsc->state);return 0 ;
@@ -461,6 +467,12 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing) {int i; +if (removing && pdsc->netdev) { +unregister_netdev (pdsc->netdev); +free_netdev (pdsc->netdev); + pdsc->netdev =NULL ; + } +pdsc_devcmd_reset (pdsc);pdsc_qcq_free (pdsc, &pdsc->notifyqcq);pdsc_qcq_free (pdsc, &pdsc->adminqcq);
@@ -528,6 +540,7 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +netif_device_detach (pdsc->netdev);pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );
@@ -554,8 +567,12 @@ static void pdsc_fw_up(struct pdsc *pdsc)if (err) goto err_out; +netif_device_attach (pdsc->netdev); +mutex_unlock (&pdsc->config_lock); +pdsc_vf_attr_replay (pdsc); +return ; err_out:
drivers/net/ethernet/pensando/pds_core/core.h +24 -0
@@ -30,6 +30,22 @@ struct pdsc_dev_bar {int res_index; }; +struct pdsc_vf { +u16 index; +u8 macaddr[6 ]; +__le32 maxrate; +__le16 vlanid; +u8 spoofchk; +u8 trusted; +u8 linkstate; +__le16 vif_types[PDS_DEV_TYPE_MAX ]; + +struct pds_core_vf_stats stats; +dma_addr_t stats_pa; + +struct pds_auxiliary_dev *padev; +}; +struct pdsc_devinfo {u8 asic_type;u8 asic_rev;
@@ -153,6 +169,7 @@ struct pdsc {struct dentry *dentry;struct device *dev;struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX ]; +struct pdsc_vf *vfs;int num_vfs;int hw_index;int id;
@@ -172,6 +189,8 @@ struct pdsc {struct pdsc_intr_info *intr_info;/* array of nintrs elements */ struct workqueue_struct *wq; +struct net_device *netdev; +struct rw_semaphore vf_op_lock;/* lock for VF operations */ unsigned int devcmd_timeout;struct mutex devcmd_lock;/* lock for dev_cmd operations */
@@ -309,4 +328,9 @@ irqreturn_t pdsc_adminq_isr(int irq, void *data);int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw,struct netlink_ext_ack *extack); +int pdsc_init_netdev (struct pdsc *pdsc); +int pdsc_set_vf_config (struct pdsc *pdsc,int vf, +struct pds_core_vf_setattr_cmd *vfc); +void pdsc_vf_attr_replay (struct pdsc *pdsc); +#endif /* _PDSC_H_ */
drivers/net/ethernet/pensando/pds_core/main.c +65 -0
89 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/netdev.c new file +504 -0
504 lines not on this page · raw message
Signature · 1 line
2.17.1
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
drivers/net/ethernet/pensando/pds_core/netdev.c line 400
Quoted 8 lines · On Fri, 18 Nov 2022 14:56:45 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/netdev.c new file +8
-0+ .ndo_set_vf_vlan = pdsc_set_vf_vlan, + .ndo_set_vf_mac = pdsc_set_vf_mac, + .ndo_set_vf_trust = pdsc_set_vf_trust, + .ndo_set_vf_rate = pdsc_set_vf_rate, + .ndo_set_vf_spoofchk = pdsc_set_vf_spoofchk, + .ndo_set_vf_link_state = pdsc_set_vf_link_state, + .ndo_get_vf_config = pdsc_get_vf_config, + .ndo_get_vf_stats = pdsc_get_vf_stats,
These are legacy, you're adding a fancy SmartNIC (or whatever your marketing decided to call it) driver. Please don't use these at all.
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 10 lines · Jakub Kicinski
drivers/net/ethernet/pensando/pds_core/netdev.c line 400
Quoted 8 lines · On Fri, 18 Nov 2022 14:56:45 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/netdev.c new file +8
-0+ .ndo_set_vf_vlan = pdsc_set_vf_vlan, + .ndo_set_vf_mac = pdsc_set_vf_mac, + .ndo_set_vf_trust = pdsc_set_vf_trust, + .ndo_set_vf_rate = pdsc_set_vf_rate, + .ndo_set_vf_spoofchk = pdsc_set_vf_spoofchk, + .ndo_set_vf_link_state = pdsc_set_vf_link_state, + .ndo_get_vf_config = pdsc_get_vf_config, + .ndo_get_vf_stats = pdsc_get_vf_stats,
Quoted 2 lines
These are legacy, you're adding a fancy SmartNIC (or whatever your marketing decided to call it) driver. Please don't use these at all.
Since these are the existing APIs that I am aware of for doing this kind of VF configuration, it seemed to be the right choice. I'm not aware of any other obvious solutions. Do you have an alternate suggestion?
Cheers,
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 11 lines · On Mon, 28 Nov 2022 14:25:56 -0800 Shannon Nelson wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 18 Nov 2022 14:56:45 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/netdev.c new file +8
-0+ .ndo_set_vf_vlan = pdsc_set_vf_vlan, + .ndo_set_vf_mac = pdsc_set_vf_mac, + .ndo_set_vf_trust = pdsc_set_vf_trust, + .ndo_set_vf_rate = pdsc_set_vf_rate, + .ndo_set_vf_spoofchk = pdsc_set_vf_spoofchk, + .ndo_set_vf_link_state = pdsc_set_vf_link_state, + .ndo_get_vf_config = pdsc_get_vf_config, + .ndo_get_vf_stats = pdsc_get_vf_stats,
Quoted 6 lines
Quoted 2 lines
These are legacy, you're adding a fancy SmartNIC (or whatever your marketing decided to call it) driver. Please don't use these at all.
Since these are the existing APIs that I am aware of for doing this kind of VF configuration, it seemed to be the right choice. I'm not aware of any other obvious solutions. Do you have an alternate suggestion?
If this is a "SmartNIC" there should be alternative solution based on representors for each of those callbacks, and the device should support forwarding using proper netdev constructs like bridge, routing, or tc.
This has been our high level guidance for a few years now. It's quite hard to move the ball forward since all major vendors have a single driver for multiple generations of HW :(
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 19 lines · Jakub Kicinski
Quoted 11 lines · On Mon, 28 Nov 2022 14:25:56 -0800 Shannon Nelson wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 18 Nov 2022 14:56:45 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/netdev.c new file +8
-0+ .ndo_set_vf_vlan = pdsc_set_vf_vlan, + .ndo_set_vf_mac = pdsc_set_vf_mac, + .ndo_set_vf_trust = pdsc_set_vf_trust, + .ndo_set_vf_rate = pdsc_set_vf_rate, + .ndo_set_vf_spoofchk = pdsc_set_vf_spoofchk, + .ndo_set_vf_link_state = pdsc_set_vf_link_state, + .ndo_get_vf_config = pdsc_get_vf_config, + .ndo_get_vf_stats = pdsc_get_vf_stats,Quoted 6 lines
Quoted 2 lines
These are legacy, you're adding a fancy SmartNIC (or whatever your marketing decided to call it) driver. Please don't use these at all.
Since these are the existing APIs that I am aware of for doing this kind of VF configuration, it seemed to be the right choice. I'm not aware of any other obvious solutions. Do you have an alternate suggestion?
Quoted 7 lines
If this is a "SmartNIC" there should be alternative solution based on representors for each of those callbacks, and the device should support forwarding using proper netdev constructs like bridge, routing, or tc.
This has been our high level guidance for a few years now. It's quite hard to move the ball forward since all major vendors have a single driver for multiple generations of HW :(
Absolutely, if the device presented to the host is a SmartNIC and has these bridge and router capabilities, by all means it should use the newer APIs, but that's not the case here.
In this case we are making devices available to baremetal platforms in a cloud vendor setting where the majority of the network configuration is controlled outside of the host machine's purview. There is no bridging, routing, or filtering control available to the baremetal client other than the basic VF configurations.
The device model presented to the host is a simple PF with VFs, not a SmartNIC, thus the pds_core driver sets up a simple PF netdev "representor" for using the existing VF control API: easy to use, everyone knows how to use it, keeps code simple.
I suppose we could have the PF create a representor netdev for each individual VF to set mac address and read stats, but that seems redundant, and as far as I know that still would be missing the other VF controls. Do we have alternate ways for the user to set things like trust and spoofchk?
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 12 lines · On Mon, 28 Nov 2022 16:37:45 -0800 Shannon Nelson wrote:
Quoted 7 lines
If this is a "SmartNIC" there should be alternative solution based on representors for each of those callbacks, and the device should support forwarding using proper netdev constructs like bridge, routing, or tc.
This has been our high level guidance for a few years now. It's quite hard to move the ball forward since all major vendors have a single driver for multiple generations of HW :(
Absolutely, if the device presented to the host is a SmartNIC and has these bridge and router capabilities, by all means it should use the newer APIs, but that's not the case here.
Quoted 5 lines
In this case we are making devices available to baremetal platforms in a cloud vendor setting where the majority of the network configuration is controlled outside of the host machine's purview. There is no bridging, routing, or filtering control available to the baremetal client other than the basic VF configurations.
Don't even start with the "our device is simple and only needs the legacy API" line of arguing :/
Quoted 7 lines
The device model presented to the host is a simple PF with VFs, not a SmartNIC, thus the
pds_coredriver sets up a simple PF netdev "representor" for using the existing VF control API: easy to use, everyone knows how to use it, keeps code simple.I suppose we could have the PF create a representor netdev for each individual VF to set mac address and read stats, but that seems
Oh, so the "representor" you mention in the cover letter is for the PF?
Quoted 3 lines
redundant, and as far as I know that still would be missing the other VF controls. Do we have alternate ways for the user to set things like trust and spoofchk?
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 13 lines · Jakub Kicinski
Quoted 12 lines · On Mon, 28 Nov 2022 16:37:45 -0800 Shannon Nelson wrote:
Quoted 7 lines
If this is a "SmartNIC" there should be alternative solution based on representors for each of those callbacks, and the device should support forwarding using proper netdev constructs like bridge, routing, or tc.
This has been our high level guidance for a few years now. It's quite hard to move the ball forward since all major vendors have a single driver for multiple generations of HW :(
Absolutely, if the device presented to the host is a SmartNIC and has these bridge and router capabilities, by all means it should use the newer APIs, but that's not the case here.
Quoted 8 lines
Quoted 5 lines
In this case we are making devices available to baremetal platforms in a cloud vendor setting where the majority of the network configuration is controlled outside of the host machine's purview. There is no bridging, routing, or filtering control available to the baremetal client other than the basic VF configurations.
Don't even start with the "our device is simple and only needs the legacy API" line of arguing :/
I'm not sure what else to say here - yes, we have a fancy and complex piece of hardware plugged into the PCI slot, but the device that shows up on the PCI bus is a very constrained model that doesn't know anything about switchdev kinds of things.
Quoted 10 lines
Quoted 8 lines
The device model presented to the host is a simple PF with VFs, not a SmartNIC, thus the
pds_coredriver sets up a simple PF netdev "representor" for using the existing VF control API: easy to use, everyone knows how to use it, keeps code simple.I suppose we could have the PF create a representor netdev for each individual VF to set mac address and read stats, but that seems
Oh, so the "representor" you mention in the cover letter is for the PF?
Yes, a PF representor simply so we can get access to the .ndo_set_vf_xxx interfaces. There is no network traffic running through the PF.
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 7 lines · On Mon, 28 Nov 2022 17:08:28 -0800 Shannon Nelson wrote:
Quoted 2 lines
Don't even start with the "our device is simple and only needs the legacy API" line of arguing :/
I'm not sure what else to say here - yes, we have a fancy and complex piece of hardware plugged into the PCI slot, but the device that shows up on the PCI bus is a very constrained model that doesn't know anything about switchdev kinds of things.
Today it is, but I presume it's all FW underneath. So a year from now you'll be back asking for extensions because FW devs added features.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The device model presented to the host is a simple PF with VFs, not a SmartNIC, thus the
pds_coredriver sets up a simple PF netdev "representor" for using the existing VF control API: easy to use, everyone knows how to use it, keeps code simple.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I suppose we could have the PF create a representor netdev for each individual VF to set mac address and read stats, but that seems
Oh, so the "representor" you mention in the cover letter is for the PF?
Yes, a PF representor simply so we can get access to the .ndo_set_vf_xxx interfaces. There is no network traffic running through the PF.
In that case not only have you come up with your own name for a SmartNIC, you also managed to misuse one of our existing terms in your own way! It can't pass any traffic it's just a dummy to hook the legacy vf ndos to. It's the opposite of what a repr is.
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 11 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 28 Nov 2022 17:08:28 -0800 Shannon Nelson wrote:
Quoted 2 lines
Don't even start with the "our device is simple and only needs the legacy API" line of arguing :/
I'm not sure what else to say here - yes, we have a fancy and complex piece of hardware plugged into the PCI slot, but the device that shows up on the PCI bus is a very constrained model that doesn't know anything about switchdev kinds of things.
Today it is, but I presume it's all FW underneath. So a year from now you'll be back asking for extensions because FW devs added features.
Sure, and that will be the time to add the APIs and code for handling the more complex switching and filtering needs. We leave it out for now so as to not have unneeded code waiting for future features that might never actually appear, as driver writers are often reminded.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
The device model presented to the host is a simple PF with VFs, not a SmartNIC, thus the
pds_coredriver sets up a simple PF netdev "representor" for using the existing VF control API: easy to use, everyone knows how to use it, keeps code simple.I suppose we could have the PF create a representor netdev for each individual VF to set mac address and read stats, but that seems
Oh, so the "representor" you mention in the cover letter is for the PF?
Quoted 7 lines
Quoted 2 lines
Yes, a PF representor simply so we can get access to the .ndo_set_vf_xxx interfaces. There is no network traffic running through the PF.
In that case not only have you come up with your own name for a SmartNIC, you also managed to misuse one of our existing terms in your own way! It can't pass any traffic it's just a dummy to hook the legacy vf ndos to. It's the opposite of what a repr is.
Sorry, this seemed to me an reasonable use of the term. Is there an alternative wording we should use for this case?
Are there other existing methods we can use for getting the VF configurations from the user, or does this make sense to keep in our current simple model?
Thanks,
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 8 lines · On Tue, 29 Nov 2022 09:57:25 -0800 Shannon Nelson wrote:
Quoted 7 lines
Quoted 2 lines
Yes, a PF representor simply so we can get access to the .ndo_set_vf_xxx interfaces. There is no network traffic running through the PF.
In that case not only have you come up with your own name for a SmartNIC, you also managed to misuse one of our existing terms in your own way! It can't pass any traffic it's just a dummy to hook the legacy vf ndos to. It's the opposite of what a repr is.
Quoted 6 lines
Sorry, this seemed to me an reasonable use of the term. Is there an alternative wording we should use for this case?
Are there other existing methods we can use for getting the VF configurations from the user, or does this make sense to keep in our current simple model?
Enough back and forth. I'm not going to come up with a special model just for you when a model already exists, and you present no technical argument against it.
I am against merging your code, if you want to override find other vendors and senior upstream reviewers who will side with you.
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 16 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 29 Nov 2022 09:57:25 -0800 Shannon Nelson wrote:
Quoted 7 lines
Quoted 2 lines
Yes, a PF representor simply so we can get access to the .ndo_set_vf_xxx interfaces. There is no network traffic running through the PF.
In that case not only have you come up with your own name for a SmartNIC, you also managed to misuse one of our existing terms in your own way! It can't pass any traffic it's just a dummy to hook the legacy vf ndos to. It's the opposite of what a repr is.
Quoted 6 lines
Sorry, this seemed to me an reasonable use of the term. Is there an alternative wording we should use for this case?
Are there other existing methods we can use for getting the VF configurations from the user, or does this make sense to keep in our current simple model?
Quoted 6 lines
Enough back and forth. I'm not going to come up with a special model just for you when a model already exists, and you present no technical argument against it.
I am against merging your code, if you want to override find other vendors and senior upstream reviewers who will side with you.
We're not asking for a special model, just to use the PF interface to configure VFs as has been the practice in the past.
Anyway, this feature can wait and we can work out alternatives later. For now, we'll drop the netdev portion from the driver and rework the other bits as discussed in other messages. I'll likely have a v2 for comments sometime next week.
Thanks for your help,
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 9 lines · On Wed, 30 Nov 2022 16:12:23 -0800 Shannon Nelson wrote:
Quoted 6 lines
Enough back and forth. I'm not going to come up with a special model just for you when a model already exists, and you present no technical argument against it.
I am against merging your code, if you want to override find other vendors and senior upstream reviewers who will side with you.
We're not asking for a special model, just to use the PF interface to configure VFs as has been the practice in the past.
It simply does not compute for me. You're exposing a very advanced vDPA interface, and yet you say you don't need any network configuration beyond what Niantic had.
There are no upstream-minded users of IPUs, if it was up to me I'd flat out ban them from the kernel.
Quoted 4 lines
Anyway, this feature can wait and we can work out alternatives later. For now, we'll drop the netdev portion from the driver and rework the other bits as discussed in other messages. I'll likely have a v2 for comments sometime next week.
Seems reasonable, if it doesn't live in networking and doesn't use any networking APIs it won't matter to me.
Re: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 30 Nov 2022 16:12:23 -0800 Shannon Nelson wrote:
We're not asking for a special model, just to use the PF interface to configure VFs as has been the practice in the past.
It simply does not compute for me. You're exposing a very advanced vDPA interface, and yet you say you don't need any network configuration beyond what Niantic had.
Would you have the same responses if we were trying to do this same kind of PF netdev on a simple Niantic-like device (simple sr-iov support, little filtering capability)?
Quoted 3 lines
There are no upstream-minded users of IPUs, if it was up to me I'd flat out ban them from the kernel.
Yeah, there's a lot of hidden magic going on behind the PCI devices presented to the host, and a lot of it depends on the use cases attempting to be addressed by the different product vendors and their various cloud and enterprise customers. I tend to think that the most friction here comes from us being more familiar and comfortable with the enterprise use cases where we typically own the whole host, and not so comfortable these newer cloud use cases with control and configuration coming from outside the host.
sln
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 08/19] pds_core: initial VF configuration
Quoted 7 lines · On Thu, 1 Dec 2022 11:19:51 -0800 Shannon Nelson wrote:
Quoted 3 lines
It simply does not compute for me. You're exposing a very advanced vDPA interface, and yet you say you don't need any network configuration beyond what Niantic had.
Would you have the same responses if we were trying to do this same kind of PF netdev on a simple Niantic-like device (simple sr-iov support, little filtering capability)?
It is really hard for me to imagine someone building a Niantic-like device today.
Recently I was thought-experiment-designing simplest Niantic-like device for container workloads. And my conclusion was that yes, TC would probably be the best way to control forwarding. (Sorry not really an answer to your question, I don't know of any real Niantics of the day)
Quoted 11 lines
Quoted 2 lines
There are no upstream-minded users of IPUs, if it was up to me I'd flat out ban them from the kernel.
Yeah, there's a lot of hidden magic going on behind the PCI devices presented to the host, and a lot of it depends on the use cases attempting to be addressed by the different product vendors and their various cloud and enterprise customers. I tend to think that the most friction here comes from us being more familiar and comfortable with the enterprise use cases where we typically own the whole host, and not so comfortable these newer cloud use cases with control and configuration coming from outside the host.
I know about cloud as much as I know about enterprise, being a Meta's employee. But those who do have public clouds seem to develop all the meaningful tech behind closed doors, under NDAs. And at best "bless" us with a code dump which is under an open source license.
The community is where various developers should come together and design together. If you do a full design internally and then come upstream to just ship the code then it's a SW distribution channel, not an open source project. That's what I am not comfortable with.
RFC PATCH 9/19 pds_core: add auxiliary_bus devices
In reply to Shannon NelsonAn auxiliary_bus device is created for each VF, and the device name is made up of the PF driver name, VIF name, and PCI BDF of the VF.
--- .../net/ethernet/pensando/pds_core/Makefile | 1 + .../net/ethernet/pensando/pds_core/auxbus.c | 126 ++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.h | 3 + drivers/net/ethernet/pensando/pds_core/main.c | 18 +++ include/linux/pds/pds_auxbus.h | 37 +++++ 5 files changed, 185 insertions(+) create mode 100644 drivers/net/ethernet/pensando/pds_core/auxbus.c create mode 100644 include/linux/pds/pds_auxbus.h
drivers/net/ethernet/pensando/pds_core/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o pds_core-y := main.o \ devlink.o \ + auxbus.o \ dev.o \ adminq.o \ core.o \
drivers/net/ethernet/pensando/pds_core/auxbus.c new file +126 -0
126 lines not on this page · raw message
drivers/net/ethernet/pensando/pds_core/core.h +3 -0
@@ -321,6 +321,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +int pdsc_auxbus_dev_add_vf (struct pdsc *pdsc,int vf_id); +int pdsc_auxbus_dev_del_vf (struct pdsc *pdsc,int vf_id); +void pdsc_process_adminq (struct pdsc_qcq *qcq);void pdsc_work_thread (struct work_struct *work);irqreturn_t pdsc_adminq_isr (int irq,void *data);
drivers/net/ethernet/pensando/pds_core/main.c +18 -0
@@ -170,6 +170,8 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)struct pds_core_vf_setattr_cmd vfc = { .attr =PDS_CORE_VF_ATTR_STATSADDR };struct pdsc *pdsc =pci_get_drvdata (pdev);struct device *dev = pdsc->dev; +enum pds_core_vif_types vt; +bool enabled = false;struct pdsc_vf *v;int ret =0 ;int i;
@@ -200,9 +202,21 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs) goto no_vfs; } +/* If any VF types are enabled, start the VF aux devices */ +for (vt =0 ; vt <PDS_DEV_TYPE_MAX && !enabled; vt++) + enabled = pdsc->viftype_status[vt].supported && + pdsc->viftype_status[vt].enabled; +if (enabled) +for (i =0 ; i < num_vfs; i++) +pdsc_auxbus_dev_add_vf (pdsc, i); +return num_vfs; } + i =pci_num_vf (pdev); +while (i--) +pdsc_auxbus_dev_del_vf (pdsc, i); + no_vfs:pci_disable_sriov (pdev);
@@ -362,6 +376,10 @@ static void pdsc_remove(struct pci_dev *pdev)*/ pdsc_dl_unregister (pdsc); +/* Remove the VFs and their aux_bus connections before other +* cleanup so that the clients can use the AdminQ to cleanly +* shut themselves down. +*/ pdsc_sriov_configure (pdev,0 );/* Now we can lock it up and tear it down */
include/linux/pds/pds_auxbus.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev ; + +struct pds_auxiliary_drv { + +/* .event_handler() - callback for receiving events +* padev: ptr to the client device info +* event: ptr to event data +* The client can provide an event handler callback that can +* receive DSC events. The Core driver may generate its +* own events which can notify the client of changes in the +* DSC status, such as a RESET event generated when the Core +* has lost contact with the FW - in this case the event.eid +* field will be 0. +*/ +void (*event_handler)(struct pds_auxiliary_dev *padev, +union pds_core_notifyq_comp *event); +}; + +struct pds_auxiliary_dev { +struct auxiliary_device aux_dev; +struct pci_dev *pcidev; +u32 id; +u16 client_id; +void (*event_handler)(struct pds_auxiliary_dev *padev, +union pds_core_notifyq_comp *event); +void *priv; +}; +#endif /* _PDSC_AUXBUS_H_ */
Signature · 1 line
2.17.1
RFC PATCH 10/19 pds_core: devlink params for enabling VIF support
In reply to Shannon NelsonNow that we have the code to start and stop the VFs and set up the auxiliary_bus devices, let's add the devlink parameter switches so the user can enable the features.
--- .../net/ethernet/pensando/pds_core/devlink.c | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+)
drivers/net/ethernet/pensando/pds_core/devlink.c +99 -0
@@ -8,6 +8,75 @@#include "core.h" +static struct pdsc_viftype *pdsc_dl_find_viftype_by_id (struct pdsc *pdsc, +enum devlink_param_type dl_id) +{ +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (pdsc->viftype_status[vt].dl_id == dl_id) +return &pdsc->viftype_status[vt]; + } + +return NULL ; +} + +static int pdsc_dl_enable_get (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry) +return -ENOENT ; + + ctx->val.vbool = vt_entry->enabled; + +return 0 ; +} + +static int pdsc_dl_enable_set (struct devlink *dl,u32 id, +struct devlink_param_gset_ctx *ctx) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; +int err =0 ; +int vf; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry || !vt_entry->supported) +return -EOPNOTSUPP ; + +if (vt_entry->enabled == ctx->val.vbool) +return 0 ; + + vt_entry->enabled = ctx->val.vbool; +for (vf =0 ; vf < pdsc->num_vfs; vf++) { + err = ctx->val.vbool ?pdsc_auxbus_dev_add_vf (pdsc, vf) : +pdsc_auxbus_dev_del_vf (pdsc, vf); + } + +return err; +} + +static int pdsc_dl_enable_validate (struct devlink *dl,u32 id, +union devlink_param_value val, +struct netlink_ext_ack *extack) +{ +struct pdsc *pdsc =devlink_priv (dl); +struct pdsc_viftype *vt_entry; + + vt_entry =pdsc_dl_find_viftype_by_id (pdsc, id); +if (!vt_entry || !vt_entry->supported) +return -EOPNOTSUPP ; + +if (!pdsc->viftype_status[vt_entry->vif_id].supported) +return -ENODEV ; + +return 0 ; +} +static char *slot_labels[] = {"fw.gold" ,"fw.mainfwa" ,"fw.mainfwb" }; static int pdsc_dl_fw_boot_get(struct devlink *dl, u32 id,
@@ -84,6 +153,18 @@ static int pdsc_dl_fw_boot_validate(struct devlink *dl, u32 id, }static const struct devlink_param pdsc_dl_params[] = { +DEVLINK_PARAM_GENERIC (ENABLE_VNET , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate), +DEVLINK_PARAM_DRIVER (PDSC_DEVLINK_PARAM_ID_LM , +"enable_lm" , +DEVLINK_PARAM_TYPE_BOOL , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate), DEVLINK_PARAM_DRIVER(PDSC_DEVLINK_PARAM_ID_FW_BOOT, "boot_fw", DEVLINK_PARAM_TYPE_STRING,
@@ -93,6 +174,23 @@ static const struct devlink_param pdsc_dl_params[] = { pdsc_dl_fw_boot_validate), }; +static void pdsc_dl_set_params_init_values (struct devlink *dl) +{ +struct pdsc *pdsc =devlink_priv (dl); +union devlink_param_value value; +int vt; + +for (vt =0 ; vt <PDS_DEV_TYPE_MAX ; vt++) { +if (!pdsc->viftype_status[vt].dl_id) +continue ; + + value.vbool = pdsc->viftype_status[vt].enabled; +devlink_param_driverinit_value_set (dl, + pdsc->viftype_status[vt].dl_id, + value); + } +} +static int pdsc_dl_flash_update (struct devlink *dl,struct devlink_flash_update_params *params,struct netlink_ext_ack *extack)
@@ -195,6 +293,7 @@ int pdsc_dl_register(struct pdsc *pdsc) ARRAY_SIZE(pdsc_dl_params));if (err)return err; +pdsc_dl_set_params_init_values (dl);devlink_register (dl);
Signature · 1 line
2.17.1
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
drivers/net/ethernet/pensando/pds_core/devlink.c static int pdsc_dl_fw_boot_validate(struct devlink *dl, u32 id, · line 167
Quoted 7 lines · On Fri, 18 Nov 2022 14:56:47 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/devlink.c +7
-0+DEVLINK_PARAM_DRIVER (PDSC_DEVLINK_PARAM_ID_LM , +"enable_lm" , +DEVLINK_PARAM_TYPE_BOOL , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate),
Terrible name, not vendor specific.
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 10 lines · Jakub Kicinski
drivers/net/ethernet/pensando/pds_core/devlink.c static int pdsc_dl_fw_boot_validate(struct devlink *dl, u32 id, · line 167
Quoted 7 lines · On Fri, 18 Nov 2022 14:56:47 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/devlink.c +7
-0+DEVLINK_PARAM_DRIVER (PDSC_DEVLINK_PARAM_ID_LM , +"enable_lm" , +DEVLINK_PARAM_TYPE_BOOL , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate),Terrible name, not vendor specific.
... but useful for starting a conversation.
How about we add
DEVLINK_PARAM_GENERIC_ID_ENABLE_LM,
to live along with the existing
DEVLINK_PARAM_GENERIC_ID_ENABLE_RDMA,
DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET,
By the way, thanks for your time looking through these patches.
Cheers,
sln
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 10 lines · Shannon Nelson
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Fri, 18 Nov 2022 14:56:47 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/devlink.c +7
-0+DEVLINK_PARAM_DRIVER (PDSC_DEVLINK_PARAM_ID_LM , +"enable_lm" , +DEVLINK_PARAM_TYPE_BOOL , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate),
Quoted 6 lines
Quoted 1 line
Terrible name, not vendor specific.
... but useful for starting a conversation.
How about we add
DEVLINK_PARAM_GENERIC_ID_ENABLE_LM,
I know we are running short of short acronyms and we have to recycle them, rfc5513 and all, so could you actually use DEVLINK_PARAM_GENERIC_ID_ENABLE_LIST_MANAGER making it clear your Smart NIC is running majordomo and will soon replace vger.
Andrew
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 18 lines · Andrew Lunn
Quoted 10 lines · Shannon Nelson
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Fri, 18 Nov 2022 14:56:47 -0800 Shannon Nelson wrote:
drivers/net/ethernet/pensando/pds_core/devlink.c +7
-0+DEVLINK_PARAM_DRIVER (PDSC_DEVLINK_PARAM_ID_LM , +"enable_lm" , +DEVLINK_PARAM_TYPE_BOOL , +BIT (DEVLINK_PARAM_CMODE_RUNTIME ), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate),Quoted 6 lines
Quoted 1 line
Terrible name, not vendor specific.
... but useful for starting a conversation.
How about we add
DEVLINK_PARAM_GENERIC_ID_ENABLE_LM,
Quoted 6 lines
I know we are running short of short acronyms and we have to recycle them, rfc5513 and all, so could you actually use
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIST_MANAGERmaking it clear your Smart NIC is running majordomo and will soon replace vger.Andrew
Oh, hush, someone might hear you speak of our plan to take over the email world! You never know who might be listening...
On the other hand, "LM" could be expanded to "LIVE_MIGRATION", but that is soooo many letters to type... perhaps I'll need to set up some additional vim macros.
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATION
Cheers,
sln
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 9 lines
Quoted 6 lines
I know we are running short of short acronyms and we have to recycle them, rfc5513 and all, so could you actually use
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIST_MANAGERmaking it clear your Smart NIC is running majordomo and will soon replace vger.Andrew
Oh, hush, someone might hear you speak of our plan to take over the email world!
It seems like something a Smart NIC would be ideal to do. Here is an email body and 10,000 email addresses i recently acquired, go send spam to them at line rate.
Quoted 2 lines
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATION
Much better.
Andrew
Jakub Kicinski
maintainerRe: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 4 lines · On Tue, 29 Nov 2022 00:29:42 +0100 Andrew Lunn wrote:
Quoted 2 lines
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATIONMuch better.
+1, although I care much less about the define name which is stupidly long anyway and more about the actual value that the user will see
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 29 Nov 2022 00:29:42 +0100 Andrew Lunn wrote:
Quoted 2 lines
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATIONMuch better.
+1, although I care much less about the define name which is stupidly long anyway and more about the actual value that the user will see
We have enable/disable devlink live migration knob in our queue. Saeed thought to send it next week.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
In reply to Jakub KicinskiQuoted 8 lines · Tue, Nov 29, 2022 at 12:39:22AM CET, [email protected] wrote:
Quoted 4 lines · On Tue, 29 Nov 2022 00:29:42 +0100 Andrew Lunn wrote:
Quoted 2 lines
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATIONMuch better.
+1, although I care much less about the define name which is stupidly long anyway and more about the actual value that the user will see
We have patches that introduce live migration as a generic port function capability bit. It is an attribute of the function.
Re: [RFC PATCH net-next 10/19] pds_core: devlink params for enabling VIF support
Quoted 5 lines · Jiri Pirko
Quoted 4 lines · Tue, Nov 29, 2022 at 12:39:22AM CET, [email protected] wrote:
Quoted 3 lines · On Tue, 29 Nov 2022 00:29:42 +0100 Andrew Lunn wrote:
Quoted 2 lines
How about:
DEVLINK_PARAM_GENERIC_ID_ENABLE_LIVE_MIGRATION
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Much better.
+1, although I care much less about the define name which is stupidly long anyway and more about the actual value that the user will see
We have patches that introduce live migration as a generic port function capability bit. It is an attribute of the function.
Thanks Leon and Jiri, we'll keep an eye out for it.
sln
RFC PATCH 11/19 pds_core: add the aux client API
In reply to Shannon NelsonAdd the client API operations for registering, unregistering, and running adminq commands. We expect to add additional operations for other clients, including requesting additional private adminqs and IRQs, but don't have the need yet,
--- .../net/ethernet/pensando/pds_core/auxbus.c | 144 +++++++++++++++++- include/linux/pds/pds_auxbus.h | 51 +++++++ 2 files changed, 193 insertions(+), 2 deletions(-)
drivers/net/ethernet/pensando/pds_core/auxbus.c +142 -2
@@ -11,6 +11,144 @@#include <linux/pds/pds_adminq.h> #include <linux/pds/pds_auxbus.h> +/** +* pds_client_register - Register the client with the device +* @padev: ptr to the client device info +* @padrv: ptr to the client driver info +* +* Register the client with the core and with the DSC. The core +* will fill in the client padev->client_id for use in calls +* to the DSC AdminQ +*/ +static int pds_client_register (struct pds_auxiliary_dev *padev, +struct pds_auxiliary_drv *padrv) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev; +struct pdsc *pdsc; +int err; +u16 ci; + + pdsc = (struct pdsc *)dev_get_drvdata (padev->aux_dev.dev.parent); + dev = pdsc->dev; + +if (pdsc->state) +return -ENXIO ; + + cmd.client_reg.opcode =PDS_AQ_CMD_CLIENT_REG ; +strscpy (cmd.client_reg.devname,dev_name (&padev->aux_dev.dev), +sizeof (cmd.client_reg.devname)); + + err =pdsc_adminq_post (pdsc, &pdsc->adminqcq, &cmd, &comp, false); +if (err) { +dev_info (dev,"register dev_name %s with DSC failed, status %d: %pe\n" , +dev_name (&padev->aux_dev.dev), comp.status,ERR_PTR (err)); +return err; + } + + ci =le16_to_cpu (comp.client_reg.client_id); +if (!ci) { +dev_err (dev,"%s: device returned null client_id\n" , __func__); +return -EIO ; + } + + padev->client_id = ci; + padev->event_handler = padrv->event_handler; + +return 0 ; +} + +/** +* pds_client_unregister - Disconnect the client from the device +* @padev: ptr to the client device info +* +* Disconnect the client from the core and with the DSC. +*/ +static int pds_client_unregister (struct pds_auxiliary_dev *padev) +{ +union pds_core_adminq_comp comp = {0 }; +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev; +struct pdsc *pdsc; +int err; + + pdsc = (struct pdsc *)dev_get_drvdata (padev->aux_dev.dev.parent); + dev = pdsc->dev; + +if (pdsc->state) +return -ENXIO ; + + cmd.client_unreg.opcode =PDS_AQ_CMD_CLIENT_UNREG ; + cmd.client_unreg.client_id =cpu_to_le16 (padev->client_id); + + err =pdsc_adminq_post (pdsc, &pdsc->adminqcq, &cmd, &comp, false); +if (err) +dev_info (dev,"unregister dev_name %s failed, status %d: %pe\n" , +dev_name (&padev->aux_dev.dev), comp.status,ERR_PTR (err)); + + padev->client_id =0 ; + +return err; +} + +/** +* pds_client_adminq_cmd - Process an adminq request for the client +* @padev: ptr to the client device +* @req: ptr to buffer with request +* @req_len: length of actual struct used for request +* @resp: ptr to buffer where answer is to be copied +* @flags: optional flags from pds_core_adminq_flags +* +* Return: 0 on success, or +* negative for error +* +* Client sends pointers to request and response buffers +* Core copies request data into pds_core_client_request_cmd +* Core sets other fields as needed +* Core posts to AdminQ +* Core copies completion data into response buffer +*/ +static int pds_client_adminq_cmd (struct pds_auxiliary_dev *padev, +union pds_core_adminq_cmd *req, +size_t req_len, +union pds_core_adminq_comp *resp, +u64 flags) +{ +union pds_core_adminq_cmd cmd = {0 }; +struct device *dev; +struct pdsc *pdsc; +size_t cp_len; +int err; + + pdsc = (struct pdsc *)dev_get_drvdata (padev->aux_dev.dev.parent); + dev = pdsc->dev; + +dev_dbg (dev,"%s: %s opcode %d\n" , + __func__,dev_name (&padev->aux_dev.dev), req->opcode); + +if (pdsc->state) +return -ENXIO ; + +/* Wrap the client's request */ + cmd.client_request.opcode =PDS_AQ_CMD_CLIENT_CMD ; + cmd.client_request.client_id =cpu_to_le16 (padev->client_id); + cp_len =min_t (size_t, req_len,sizeof (cmd.client_request.client_cmd)); +memcpy (cmd.client_request.client_cmd, req, cp_len); + + err =pdsc_adminq_post (pdsc, &pdsc->adminqcq, &cmd, resp, !!(flags &PDS_AQ_FLAG_FASTPOLL )); +if (err && err != -EAGAIN ) +dev_info (dev,"client admin cmd failed: %pe\n" ,ERR_PTR (err)); + +return err; +} + +static struct pds_core_ops pds_core_ops = { + .register_client = pds_client_register, + .unregister_client = pds_client_unregister, + .adminq_cmd = pds_client_adminq_cmd, +}; +static void pdsc_auxbus_dev_release (struct device *dev) { struct pds_auxiliary_dev *padev =
@@ -21,7 +159,8 @@ static void pdsc_auxbus_dev_release(struct device *dev)static struct pds_auxiliary_dev *pdsc_auxbus_dev_register (struct pdsc *pdsc,char *name,u32 id, -struct pci_dev *client_dev) +struct pci_dev *client_dev, +struct pds_core_ops *ops) {struct auxiliary_device *aux_dev;struct pds_auxiliary_dev *padev;
@@ -31,6 +170,7 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *pdsc,if (!padev)return NULL ; + padev->ops = ops; padev->pcidev = client_dev; aux_dev = &padev->aux_dev;
@@ -99,7 +239,7 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *pdsc, int vf_id) id =PCI_DEVID (pdsc->pdev->bus->number,pci_iov_virtfn_devfn (pdsc->pdev, vf_id)); padev =pdsc_auxbus_dev_register (pdsc, pdsc->viftype_status[vt].name, id, - pdsc->pdev); + pdsc->pdev, &pds_core_ops); pdsc->vfs[vf_id].padev = padev;/* We only support a single type per VF, so jump out here */
include/linux/pds/pds_auxbus.h +51 -0
61 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 12/19 pds_core: publish events to the clients
In reply to Shannon NelsonWhen the Core device gets an event from the device, or notices the device FW to be up or down, it needs to send those events on to the clients that have an event handler.
--- .../net/ethernet/pensando/pds_core/adminq.c | 17 ++++++++ .../net/ethernet/pensando/pds_core/auxbus.c | 40 +++++++++++++++++++ drivers/net/ethernet/pensando/pds_core/core.c | 15 +++++++ drivers/net/ethernet/pensando/pds_core/core.h | 3 ++ 4 files changed, 75 insertions(+)
drivers/net/ethernet/pensando/pds_core/adminq.c +17 -0
@@ -34,11 +34,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)case PDS_EVENT_LINK_CHANGE :dev_info (pdsc->dev,"NotifyQ LINK_CHANGE ecode %d eid %lld\n" , ecode, eid); +pdsc_auxbus_publish (pdsc,PDSC_ALL_CLIENT_IDS , comp);break ;case PDS_EVENT_RESET :dev_info (pdsc->dev,"NotifyQ RESET ecode %d eid %lld\n" , ecode, eid); +pdsc_auxbus_publish (pdsc,PDSC_ALL_CLIENT_IDS , comp);break ;case PDS_EVENT_XCVR :
@@ -46,6 +48,21 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq) ecode, eid);break ; +case PDS_EVENT_CLIENT : + { +struct pds_core_client_event *ce; +union pds_core_notifyq_comp *cc; +u16 client_id; + + ce = (struct pds_core_client_event *)comp; + cc = (union pds_core_notifyq_comp *)&ce->client_event; + client_id =le16_to_cpu (ce->client_id); +dev_info (pdsc->dev,"NotifyQ CLIENT %d ecode %d eid %lld cc->ecode %d\n" , + client_id, ecode, eid,le16_to_cpu (cc->ecode)); +pdsc_auxbus_publish (pdsc, client_id, cc); +break ; + } +default :dev_info (pdsc->dev,"NotifyQ ecode %d eid %lld\n" , ecode, eid);
drivers/net/ethernet/pensando/pds_core/auxbus.c +40 -0
@@ -205,6 +205,46 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *pdsc,return NULL ; } +static int pdsc_core_match (struct device *dev,const void *data) +{ +struct pds_auxiliary_dev *curr_padev; +struct pdsc *curr_pdsc; +const struct pdsc *pdsc; + +/* Match the core device searching for its clients */ + curr_padev = container_of(dev, struct pds_auxiliary_dev, aux_dev.dev); + curr_pdsc = (struct pdsc *)dev_get_drvdata (curr_padev->aux_dev.dev.parent); + pdsc = data; + +if (curr_pdsc == pdsc) +return 1 ; + +return 0 ; +} + +int pdsc_auxbus_publish (struct pdsc *pdsc,u16 client_id, +union pds_core_notifyq_comp *event) +{ +struct pds_auxiliary_dev *padev; +struct auxiliary_device *aux_dev; + +/* Search aux bus for this core's devices */ + aux_dev =auxiliary_find_device (NULL , pdsc, pdsc_core_match); +while (aux_dev) { + padev = container_of(aux_dev, struct pds_auxiliary_dev, aux_dev); +if ((padev->client_id == client_id || + client_id ==PDSC_ALL_CLIENT_IDS ) && + padev->event_handler) + padev->event_handler (padev, event); +put_device (&aux_dev->dev); + + aux_dev =auxiliary_find_device (&aux_dev->dev, + pdsc, pdsc_core_match); + } + +return 0 ; +} +int pdsc_auxbus_dev_add_vf (struct pdsc *pdsc,int vf_id) {struct pds_auxiliary_dev *padev;
drivers/net/ethernet/pensando/pds_core/core.c +15 -0
@@ -532,6 +532,11 @@ void pdsc_stop(struct pdsc *pdsc)static void pdsc_fw_down (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =0 , + }; +mutex_lock (&pdsc->config_lock);if (test_and_set_bit (PDSC_S_FW_DEAD , &pdsc->state)) {
@@ -540,6 +545,9 @@ static void pdsc_fw_down(struct pdsc *pdsc)return ; } +/* Notify clients of fw_down */ +pdsc_auxbus_publish (pdsc,PDSC_ALL_CLIENT_IDS , &reset_event); +netif_device_detach (pdsc->netdev);pdsc_mask_interrupts (pdsc);pdsc_teardown (pdsc,PDSC_TEARDOWN_RECOVERY );
@@ -549,6 +557,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)static void pdsc_fw_up (struct pdsc *pdsc) { +union pds_core_notifyq_comp reset_event = { + .reset.ecode =cpu_to_le16 (PDS_EVENT_RESET ), + .reset.state =1 , + };int err;mutex_lock (&pdsc->config_lock);
@@ -573,6 +585,9 @@ static void pdsc_fw_up(struct pdsc *pdsc)pdsc_vf_attr_replay (pdsc); +/* Notify clients of fw_up */ +pdsc_auxbus_publish (pdsc,PDSC_ALL_CLIENT_IDS , &reset_event); +return ; err_out:
drivers/net/ethernet/pensando/pds_core/core.h +3 -0
@@ -321,6 +321,9 @@ int pdsc_start(struct pdsc *pdsc);void pdsc_stop (struct pdsc *pdsc);void pdsc_health_thread (struct work_struct *work); +#define PDSC_ALL_CLIENT_IDS 0xffff +int pdsc_auxbus_publish (struct pdsc *pdsc,u16 client_id, +union pds_core_notifyq_comp *event);int pdsc_auxbus_dev_add_vf (struct pdsc *pdsc,int vf_id);int pdsc_auxbus_dev_del_vf (struct pdsc *pdsc,int vf_id);
Signature · 1 line
2.17.1
RFC PATCH 13/19 pds_core: Kconfig and pds_core.rst
In reply to Shannon NelsonDocumentation and Kconfig hook for building the driver.
--- .../ethernet/pensando/pds_core.rst | 162 ++++++++++++++++++ MAINTAINERS | 3 +- drivers/net/ethernet/pensando/Kconfig | 12 ++ 3 files changed, 176 insertions(+), 1 deletion(-) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_core.rst
Documentation/networking/device_drivers/ethernet/pensando/pds_core.rst new file +162 -0
@@ -0,0 +1,162 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +======================================================== +Linux Driver for the Pensando(R) DSC adapter family +======================================================== + +Pensando Linux Core driver. +Copyright(c) 2022 Pensando Systems, Inc + +Identifying the Adapter +======================= + +To find if one or more Pensando PCI Core devices are installed on the +host, check for the PCI devices:: + + # lspci -d 1dd8:100c + 39:00.0 Processing accelerators: Pensando Systems Device 100c + 3a:00.0 Processing accelerators: Pensando Systems Device 100c + +If such devices are listed as above, then the pds_core.ko driver should find +and configure them for use. There should be log entries in the kernel +messages such as these:: + + $ dmesg | grep pds_core + pds_core 0000:b5:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link) + pds_core 0000:b5:00.0: FW: 1.51.0-73 + pds_core 0000:b6:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link) + pds_core 0000:b5:00.0: FW: 1.51.0-73 + +Driver and firmware version information can be gathered with devlink:: + + $ devlink dev info pci/0000:b5:00.0 + pci/0000:b5:00.0: + driver pds_core + serial_number FLM18420073 + versions: + fixed: + asic.id 0x0 + asic.rev 0x0 + running: + fw 1.51.0-73 + stored: + fw.goldfw 1.15.9-C-22 + fw.mainfwa 1.51.0-73 + fw.mainfwb 1.51.0-57 + + +Info versions +============= + +The ``pds_core`` driver reports the following versions + +.. list-table:: devlink info versions implemented + :widths: 5 5 90 + + * - Name + - Type + - Description + * - ``fw`` + - running + - Version of firmware running on the device + * - ``fw.goldfw`` + - stored + - Version of firmware stored in the goldfw slot + * - ``fw.mainfwa`` + - stored + - Version of firmware stored in the mainfwa slot + * - ``fw.mainfwb`` + - stored + - Version of firmware stored in the mainfwb slot + * - ``asic.id`` + - fixed + - The ASIC type for this device + * - ``asic.rev`` + - fixed + - The revision of the ASIC for this device + + +Parameters +========== + +The ``pds_core`` driver implements the following generic +parameters for controlling the functionality to be made available +as auxiliary_bus devices. + +.. list-table:: Generic parameters implemented + :widths: 5 5 8 82 + + * - Name + - Mode + - Type + - Description + * - ``enable_vnet`` + - runtime + - Boolean + - Enables vDPA functionality through an auxiliary_bus device + + +The ``pds_core`` driver also implements the following driver-specific +parameters for similar uses, as well as for selecting the next boot firmware: + +.. list-table:: Driver-specific parameters implemented + :widths: 5 5 8 82 + + * - Name + - Mode + - Type + - Description + * - ``enable_lm`` + - runtime + - Boolean + - Enables Live Migration functionality through an auxiliary_bus device + * - ``boot_fw`` + - runtime + - String + - Selects the Firmware slot to use for the next DSC boot + + +Firmware Management +=================== + +Using the ``devlink`` utility's ``flash`` command the DSC firmware can be +updated. The downloaded firmware will be loaded into either of mainfwa or +mainfwb firmware slots, whichever is not currrently in use, and that slot +will be then selected for the next boot. The firmware currently in use can +be found by inspecting the ``running`` firmware from the devlink dev info. + +The ``boot_fw`` parameter can inspect and select the firmware slot to be +used in the next DSC boot up. The mainfwa and mainfwb slots are used for +for normal operations, and the goldfw slot should only be selected for +recovery purposes if both the other slots have bad or corrupted firmware. + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> Pensando devices + -> Pensando Ethernet PDS_CORE Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]
MAINTAINERS +2 -1
@@ -16150,8 +16150,9 @@ M: Shannon Nelson <[email protected]> M: [email protected] L: [email protected] S: Supported -F: Documentation/networking/device_drivers/ethernet/pensando/ionic.rst +F: Documentation/networking/device_drivers/ethernet/pensando/ F: drivers/net/ethernet/pensando/ +F: include/linux/pds/ PER-CPU MEMORY ALLOCATOR M: Dennis Zhou <[email protected]>
drivers/net/ethernet/pensando/Kconfig +12 -0
@@ -17,6 +17,18 @@ config NET_VENDOR_PENSANDO if NET_VENDOR_PENSANDO +config PDS_CORE + tristate "Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/pensando/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + config IONIC tristate "Pensando Ethernet IONIC Support" depends on 64BIT && PCI
Signature · 1 line
2.17.1
RFC PATCH 14/19 pds_vdpa: Add new PCI VF device for PDS vDPA services
In reply to Shannon NelsonThis is the initial PCI driver framework for the new pds_vdpa VF device driver, an auxiliary_bus client of the pds_core driver. This does the very basics of registering for the new PCI device 1dd8:100b, setting up debugfs entries, and registering with devlink.
The new PCI device id has not made it to the official PCI ID Repository yet, but will soon be registered there.
--- drivers/vdpa/pds/Makefile | 7 + drivers/vdpa/pds/debugfs.c | 44 +++++++ drivers/vdpa/pds/debugfs.h | 22 ++++ drivers/vdpa/pds/pci_drv.c | 143 +++++++++++++++++++++ drivers/vdpa/pds/pci_drv.h | 46 +++++++ include/linux/pds/pds_core_if.h | 1 + include/linux/pds/pds_vdpa.h | 219 ++++++++++++++++++++++++++++++++ 7 files changed, 482 insertions(+) create mode 100644 drivers/vdpa/pds/Makefile create mode 100644 drivers/vdpa/pds/debugfs.c create mode 100644 drivers/vdpa/pds/debugfs.h create mode 100644 drivers/vdpa/pds/pci_drv.c create mode 100644 drivers/vdpa/pds/pci_drv.h create mode 100644 include/linux/pds/pds_vdpa.h
drivers/vdpa/pds/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright(c) 2022 Pensando Systems, Inc + +obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o + +pds_vdpa-y := pci_drv.o \ + debugfs.o
drivers/vdpa/pds/debugfs.c new file +44 -0
44 lines not on this page · raw message
drivers/vdpa/pds/debugfs.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_VDPA_DEBUGFS_H_ +#define _PDS_VDPA_DEBUGFS_H_ + +#include <linux/debugfs.h> + +#ifdef CONFIG_DEBUG_FS + +void pds_vdpa_debugfs_create (void ); +void pds_vdpa_debugfs_destroy (void ); +void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +#else +static inline void pds_vdpa_debugfs_create (void ) { } +static inline void pds_vdpa_debugfs_destroy (void ) { } +static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { } +static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { } +#endif + +#endif /* _PDS_VDPA_DEBUGFS_H_ */
drivers/vdpa/pds/pci_drv.c new file +143 -0
143 lines not on this page · raw message
drivers/vdpa/pds/pci_drv.h new file +46 -0
46 lines not on this page · raw message
include/linux/pds/pds_core_if.h +1 -0
@@ -8,6 +8,7 @@#define PCI_VENDOR_ID_PENSANDO 0x1dd8#define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c +#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b#define PDS_CORE_BARS_MAX 4#define PDS_CORE_PCI_BAR_DBELL 1
include/linux/pds/pds_vdpa.h new file +219 -0
219 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH net-next 14/19] pds_vdpa: Add new PCI VF device for PDS vDPA services
Quoted 133 lines · Shannon Nelson
This is the initial PCI driver framework for the new
pds_vdpaVF device driver, anauxiliary_busclient of thepds_coredriver. This does the very basics of registering for the new PCI device 1dd8:100b, setting up debugfs entries, and registering with devlink.The new PCI device id has not made it to the official PCI ID Repository yet, but will soon be registered there.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 7 + drivers/vdpa/pds/debugfs.c | 44 +++++++ drivers/vdpa/pds/debugfs.h | 22 ++++ drivers/vdpa/pds/pci_drv.c | 143 +++++++++++++++++++++ drivers/vdpa/pds/pci_drv.h | 46 +++++++ include/linux/pds/pds_core_if.h | 1 + include/linux/pds/pds_vdpa.h | 219 ++++++++++++++++++++++++++++++++ 7 files changed, 482 insertions(+)create mode 100644
drivers/vdpa/pds/Makefile
create mode 100644drivers/vdpa/pds/debugfs.c
create mode 100644drivers/vdpa/pds/debugfs.h
create mode 100644drivers/vdpa/pds/pci_drv.c
create mode 100644drivers/vdpa/pds/pci_drv.h
create mode 100644include/linux/pds/pds_vdpa.hdrivers/vdpa/pds/Makefile new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright(c) 2022 Pensando Systems, Inc + +obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o + +pds_vdpa-y := pci_drv.o \ + debugfs.odrivers/vdpa/pds/debugfs.c new file +44
-0@@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/types.h> + +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_vdpa.h> + +#include "pci_drv.h" +#include "debugfs.h" + +#ifdef CONFIG_DEBUG_FS + +static struct dentry *dbfs_dir; + +void +pds_vdpa_debugfs_create (void ) +{ + dbfs_dir =debugfs_create_dir (PDS_VDPA_DRV_NAME ,NULL ); +} + +void +pds_vdpa_debugfs_destroy (void ) +{ +debugfs_remove_recursive (dbfs_dir); + dbfs_dir =NULL ; +} + +void +pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) +{ + vdpa_pdev->dentry =debugfs_create_dir (pci_name (vdpa_pdev->pdev), dbfs_dir); +} + +void +pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) +{ +debugfs_remove_recursive (vdpa_pdev->dentry); + vdpa_pdev->dentry =NULL ; +} + +#endif /* CONFIG_DEBUG_FS */ drivers/vdpa/pds/debugfs.h new file +22
-0@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_VDPA_DEBUGFS_H_ +#define _PDS_VDPA_DEBUGFS_H_ + +#include <linux/debugfs.h> + +#ifdef CONFIG_DEBUG_FS + +void pds_vdpa_debugfs_create (void ); +void pds_vdpa_debugfs_destroy (void ); +void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +#else +static inline void pds_vdpa_debugfs_create (void ) { } +static inline void pds_vdpa_debugfs_destroy (void ) { } +static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { } +static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { } +#endif + +#endif /* _PDS_VDPA_DEBUGFS_H_ */ drivers/vdpa/pds/pci_drv.c new file +10
-0@@ -0,0 +1,143 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/aer.h> +#include <linux/types.h> +#include <linux/vdpa.h> + +#include <linux/pds/pds_core_if.h>
drivers/vdpa/pds/pci_drv.c line 20
Quoted 10 lines
drivers/vdpa/pds/pci_drv.c new file +10
-0+#include <linux/pds/pds_vdpa.h> + +#include "pci_drv.h" +#include "debugfs.h" + +static void +pds_vdpa_dma_action (void *data) +{ +pci_free_irq_vectors ((struct pci_dev *)data); +}
Nit: since we're release irq vectors, it might be better to use "pds_vdpa_irq_action"
Quoted 45 lines
drivers/vdpa/pds/pci_drv.c new file +45
-0+ +static int +pds_vdpa_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +struct pds_vdpa_pci_device *vdpa_pdev; +struct device *dev = &pdev->dev; +int err; + + vdpa_pdev =kzalloc (sizeof (*vdpa_pdev),GFP_KERNEL ); +if (!vdpa_pdev) +return -ENOMEM ; +pci_set_drvdata (pdev, vdpa_pdev); + + vdpa_pdev->pdev = pdev; + vdpa_pdev->vf_id =pci_iov_vf_id (pdev); + vdpa_pdev->pci_id =PCI_DEVID (pdev->bus->number, pdev->devfn); + +/* Query system for DMA addressing limitation for the device. */ + err =dma_set_mask_and_coherent (dev,DMA_BIT_MASK (PDS_CORE_ADDR_LEN )); +if (err) { +dev_err (dev,"Unable to obtain 64-bit DMA for consistent allocations, aborting. %pe\n" , +ERR_PTR (err)); + goto err_out_free_mem; + } + +pci_enable_pcie_error_reporting (pdev); + +/* Use devres management */ + err =pcim_enable_device (pdev); +if (err) { +dev_err (dev,"Cannot enable PCI device: %pe\n" ,ERR_PTR (err)); + goto err_out_free_mem; + } + + err =devm_add_action_or_reset (dev, pds_vdpa_dma_action, pdev); +if (err) { +dev_err (dev,"Failed adding devres for freeing irq vectors: %pe\n" , +ERR_PTR (err)); + goto err_out_pci_release_device; + } + +pci_set_master (pdev); + +pds_vdpa_debugfs_add_pcidev (vdpa_pdev);
drivers/vdpa/pds/pci_drv.c line 75
Quoted 10 lines
drivers/vdpa/pds/pci_drv.c new file +10
-0+ +dev_info (dev,"%s: PF %#04x VF %#04x (%d) vf_id %d domain %d vdpa_aux %p vdpa_pdev %p\n" , + __func__,pci_dev_id (vdpa_pdev->pdev->physfn), + vdpa_pdev->pci_id, vdpa_pdev->pci_id, vdpa_pdev->vf_id, +pci_domain_nr (pdev->bus), vdpa_pdev->vdpa_aux, vdpa_pdev); + +return 0 ; + +err_out_pci_release_device: +pci_disable_device (pdev);
Do we still need to care about this consider we use devres/pcim_enable_device()?
Quoted 184 lines
drivers/vdpa/pds/pci_drv.c new file +68
-0+err_out_free_mem: +pci_disable_pcie_error_reporting (pdev); +kfree (vdpa_pdev); +return err; +} + +static void +pds_vdpa_pci_remove (struct pci_dev *pdev) +{ +struct pds_vdpa_pci_device *vdpa_pdev =pci_get_drvdata (pdev); + +pds_vdpa_debugfs_del_pcidev (vdpa_pdev); +pci_clear_master (pdev); +pci_disable_pcie_error_reporting (pdev); +pci_disable_device (pdev); +kfree (vdpa_pdev); + +dev_info (&pdev->dev,"Removed\n" ); +} + +static const struct pci_device_id +pds_vdpa_pci_table[] = { + {PCI_VDEVICE (PENSANDO ,PCI_DEVICE_ID_PENSANDO_VDPA_VF ) }, + {0 , } +}; +MODULE_DEVICE_TABLE (pci, pds_vdpa_pci_table); + +static struct pci_driver +pds_vdpa_pci_driver = { + .name =PDS_VDPA_DRV_NAME , + .id_table = pds_vdpa_pci_table, + .probe = pds_vdpa_pci_probe, + .remove = pds_vdpa_pci_remove +}; + +static void __exit +pds_vdpa_pci_cleanup (void ) +{ +pci_unregister_driver (&pds_vdpa_pci_driver); + +pds_vdpa_debugfs_destroy (); +} +module_exit (pds_vdpa_pci_cleanup); + +static int __init +pds_vdpa_pci_init (void ) +{ +int err; + +pds_vdpa_debugfs_create (); + + err =pci_register_driver (&pds_vdpa_pci_driver); +if (err) { +pr_err ("%s: pci driver register failed: %pe\n" , __func__,ERR_PTR (err)); + goto err_pci; + } + +return 0 ; + +err_pci: +pds_vdpa_debugfs_destroy (); +return err; +} +module_init (pds_vdpa_pci_init); + +MODULE_DESCRIPTION (PDS_VDPA_DRV_DESCRIPTION ); +MODULE_AUTHOR ("Pensando Systems, Inc" ); +MODULE_LICENSE ("GPL" );drivers/vdpa/pds/pci_drv.h new file +46
-0@@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PCI_DRV_H +#define _PCI_DRV_H + +#include <linux/pci.h> +#include <linux/virtio_pci_modern.h> + +#define PDS_VDPA_DRV_NAME "pds_vdpa" +#define PDS_VDPA_DRV_DESCRIPTION "Pensando vDPA VF Device Driver" + +#define PDS_VDPA_BAR_BASE 0 +#define PDS_VDPA_BAR_INTR 2 +#define PDS_VDPA_BAR_DBELL 4 + +struct pds_dev_bar { +int index; +void __iomem *vaddr; +phys_addr_t pa; +unsigned long len; +}; + +struct pds_vdpa_intr_info { +int index; +int irq; +int qid; +char name[32 ]; +}; + +struct pds_vdpa_pci_device { +struct pci_dev *pdev; +struct pds_vdpa_aux *vdpa_aux; + +int vf_id; +int pci_id; + +int nintrs; +struct pds_vdpa_intr_info *intrs; + +struct dentry *dentry; + +struct virtio_pci_modern_device vd_mdev; +}; + +#endif /* _PCI_DRV_H */ include/linux/pds/pds_core_if.h +1
-0@@ -8,6 +8,7 @@#define PCI_VENDOR_ID_PENSANDO 0x1dd8#define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c +#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b#define PDS_CORE_BARS_MAX 4#define PDS_CORE_PCI_BAR_DBELL 1include/linux/pds/pds_vdpa.h new file +46
-0@@ -0,0 +1,219 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_VDPA_IF_H_ +#define _PDS_VDPA_IF_H_ + +#include <linux/pds/pds_common.h> + +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_VDPA_DEV_NAME PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_VDPA_STR + +/* + * enum pds_vdpa_cmd_opcode - vDPA Device commands + */ +enum pds_vdpa_cmd_opcode { + PDS_VDPA_CMD_INIT = 48, + PDS_VDPA_CMD_IDENT = 49, + PDS_VDPA_CMD_RESET = 51, + PDS_VDPA_CMD_VQ_RESET = 52, + PDS_VDPA_CMD_VQ_INIT = 53, + PDS_VDPA_CMD_STATUS_UPDATE = 54, + PDS_VDPA_CMD_SET_FEATURES = 55, + PDS_VDPA_CMD_SET_ATTR = 56, +}; + +/** + * struct pds_vdpa_cmd - generic command + * @opcode: Opcode + * @vdpa_index: Index for vdpa subdevice + * @vf_id: VF id + */ +struct pds_vdpa_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; +}; + +/** + * struct pds_vdpa_comp - generic command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @color: Color bit + */ +struct pds_vdpa_comp { + u8 status; + u8 rsvd[14];
include/linux/pds/pds_vdpa.h line 56
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+u8 color; +}; + +/** +* struct pds_vdpa_init_cmd - INIT command +* @opcode: Opcode PDS_VDPA_CMD_INIT +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @len: length of config info DMA space +* @config_pa: address for DMA of virtio_net_config struct
Looks like the structure is not specific to net, if yes, we may tweak the above comment to say it's the address of the device configuration space.
Quoted 5 lines
include/linux/pds/pds_vdpa.h new file +5
-0+*/ +struct pds_vdpa_init_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id;
include/linux/pds/pds_vdpa.h line 71
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+__le32 len; +__le64 config_pa; +}; + +/** +* struct pds_vdpa_ident - vDPA identification data +* @hw_features: vDPA features supported by device +* @max_vqs: max queues available (2 queues for a single queuepair) +* @max_qlen: log(2) of maximum number of descriptors +* @min_qlen: log(2) of minimum number of descriptors
Note that is you have the plan to support packed virtqueue, the qlen is not necessarily the power of 2.
Quoted 64 lines
include/linux/pds/pds_vdpa.h new file +64
-0+* +* This struct is used in a DMA block that is set up for the PDS_VDPA_CMD_IDENT +* transaction. Set up the DMA block and send the address in the IDENT cmd +* data, the DSC will write the ident information, then we can remove the DMA +* block after reading the answer. If the completion status is 0, then there +* is valid information, else there was an error and the data should be invalid. +*/ +struct pds_vdpa_ident { +__le64 hw_features; +__le16 max_vqs; +__le16 max_qlen; +__le16 min_qlen; +}; + +/** +* struct pds_vdpa_ident_cmd - IDENT command +* @opcode: Opcode PDS_VDPA_CMD_IDENT +* @rsvd: Word boundary padding +* @vf_id: VF id +* @len: length of ident info DMA space +* @ident_pa: address for DMA of ident info (struct pds_vdpa_ident) +* only used for this transaction, then forgotten by DSC +*/ +struct pds_vdpa_ident_cmd { +u8 opcode; +u8 rsvd; +__le16 vf_id; +__le32 len; +__le64 ident_pa; +}; + +/** +* struct pds_vdpa_status_cmd - STATUS_UPDATE command +* @opcode: Opcode PDS_VDPA_CMD_STATUS_UPDATE +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @status: new status bits +*/ +struct pds_vdpa_status_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id; +u8 status; +}; + +/** +* enum pds_vdpa_attr - List of VDPA device attributes +* @PDS_VDPA_ATTR_MAC: MAC address +* @PDS_VDPA_ATTR_MAX_VQ_PAIRS: Max virtqueue pairs +*/ +enum pds_vdpa_attr { +PDS_VDPA_ATTR_MAC =1 , +PDS_VDPA_ATTR_MAX_VQ_PAIRS =2 , +}; + +/** +* struct pds_vdpa_setattr_cmd - SET_ATTR command +* @opcode: Opcode PDS_VDPA_CMD_SET_ATTR +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @attr: attribute to be changed (enum pds_vdpa_attr) +* @pad: Word boundary padding +* @mac: new mac address to be assigned as vdpa device address +* @max_vq_pairs: new limit of virtqueue pairs
include/linux/pds/pds_vdpa.h line 145
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+*/ +struct pds_vdpa_setattr_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id; +u8 attr; +u8 pad[3 ]; +union { +u8 mac[6 ]; +__le16 max_vq_pairs;
So does this mean if we want to set both mac and max_vq_paris, we need two commands? The seems to be less efficient, since the mgmt layer could provision more attributes here. Can we pack all attributes into a single command?
include/linux/pds/pds_vdpa.h line 154
Quoted 9 lines
include/linux/pds/pds_vdpa.h new file +9
-0+ } __packed; +}; + +/** +* struct pds_vdpa_vq_init_cmd - queue init command +* @opcode: Opcode PDS_VDPA_CMD_VQ_INIT +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @qid: Queue id (bit0 clear = rx, bit0 set = tx, qid=N is ctrlq)
I wonder any reason we need to design it like this, it would be better to make it general to be used by other type of virtio devices.
include/linux/pds/pds_vdpa.h line 159
Quoted 5 lines
include/linux/pds/pds_vdpa.h new file +5
-0+* @len: log(2) of max descriptor count +* @desc_addr: DMA address of descriptor area +* @avail_addr: DMA address of available descriptors (aka driver area) +* @used_addr: DMA address of used descriptors (aka device area) +* @intr_index: interrupt index
Is this something like MSI-X vector?
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+*/ +struct pds_vdpa_vq_init_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id; +__le16 qid; +__le16 len; +__le64 desc_addr; +__le64 avail_addr; +__le64 used_addr;
include/linux/pds/pds_vdpa.h line 179
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+__le16 intr_index; +}; + +/** +* struct pds_vdpa_vq_init_comp - queue init completion +* @status: Status of the command (enum pds_core_status_code) +* @hw_qtype: HW queue type, used in doorbell selection +* @hw_qindex: HW queue index, used in doorbell selection +* @rsvd: Word boundary padding +* @color: Color bit
More comment is needed to know the how to use this color bit.
include/linux/pds/pds_vdpa.h line 191
Quoted 12 lines
include/linux/pds/pds_vdpa.h new file +12
-0+*/ +struct pds_vdpa_vq_init_comp { +u8 status; +u8 hw_qtype; +__le16 hw_qindex; +u8 rsvd[11 ]; +u8 color; +}; + +/** +* struct pds_vdpa_vq_reset_cmd - queue reset command +* @opcode: Opcode PDS_VDPA_CMD_VQ_RESET
Is there a chance that we could have more type of opcode here?
Thanks
Quoted 18 lines
include/linux/pds/pds_vdpa.h new file +18
-0+* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @qid: Queue id +*/ +struct pds_vdpa_vq_reset_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id; +__le16 qid; +}; + +/** +* struct pds_vdpa_set_features_cmd - set hw features +* @opcode: Opcode PDS_VDPA_CMD_SET_FEATURES +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @rsvd: Word boundary padding +* @features: Feature bit mask
include/linux/pds/pds_vdpa.h line 219
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+*/ +struct pds_vdpa_set_features_cmd { +u8 opcode; +u8 vdpa_index; +__le16 vf_id; +__le32 rsvd; +__le64 features; +}; + +#endif /* _PDS_VDPA_IF_H_ */
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 14/19] pds_vdpa: Add new PCI VF device for PDS vDPA services
Quoted 143 lines · Jason Wang
Quoted 142 lines · Shannon Nelson
This is the initial PCI driver framework for the new
pds_vdpaVF device driver, anauxiliary_busclient of thepds_coredriver. This does the very basics of registering for the new PCI device 1dd8:100b, setting up debugfs entries, and registering with devlink.The new PCI device id has not made it to the official PCI ID Repository yet, but will soon be registered there.
Signed-off-by Shannon Nelson <[email protected]>---
drivers/vdpa/pds/Makefile | 7 +drivers/vdpa/pds/debugfs.c | 44 +++++++ drivers/vdpa/pds/debugfs.h | 22 ++++ drivers/vdpa/pds/pci_drv.c | 143 +++++++++++++++++++++ drivers/vdpa/pds/pci_drv.h | 46 +++++++ include/linux/pds/pds_core_if.h | 1 + include/linux/pds/pds_vdpa.h | 219 ++++++++++++++++++++++++++++++++ 7 files changed, 482 insertions(+)create mode 100644
drivers/vdpa/pds/Makefile
create mode 100644drivers/vdpa/pds/debugfs.c
create mode 100644drivers/vdpa/pds/debugfs.h
create mode 100644drivers/vdpa/pds/pci_drv.c
create mode 100644drivers/vdpa/pds/pci_drv.h
create mode 100644include/linux/pds/pds_vdpa.hdrivers/vdpa/pds/Makefile new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright(c) 2022 Pensando Systems, Inc + +obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o + +pds_vdpa-y := pci_drv.o \ + debugfs.odrivers/vdpa/pds/debugfs.c new file +34
-0@@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/types.h> + +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_vdpa.h> + +#include "pci_drv.h" +#include "debugfs.h" + +#ifdef CONFIG_DEBUG_FS + +static struct dentry *dbfs_dir; + +void +pds_vdpa_debugfs_create (void ) +{ + dbfs_dir = debugfs_create_dir(PDS_VDPA_DRV_NAME, NULL); +} + +void +pds_vdpa_debugfs_destroy (void ) +{ + debugfs_remove_recursive(dbfs_dir); + dbfs_dir = NULL; +} + +void +pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) +{ + vdpa_pdev->dentry =
debugfs_create_dir(pci_name(vdpa_pdev->pdev), dbfs_dir);drivers/vdpa/pds/debugfs.c new file +10
-0+} + +void +pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) +{ + debugfs_remove_recursive(vdpa_pdev->dentry); + vdpa_pdev->dentry = NULL; +} + +#endif /* CONFIG_DEBUG_FS */ drivers/vdpa/pds/debugfs.h new file +18
-0@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_VDPA_DEBUGFS_H_ +#define _PDS_VDPA_DEBUGFS_H_ + +#include <linux/debugfs.h> + +#ifdef CONFIG_DEBUG_FS + +void pds_vdpa_debugfs_create (void ); +void pds_vdpa_debugfs_destroy (void ); +void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +#else +static inline void pds_vdpa_debugfs_create (void ) { } +static inline void pds_vdpa_debugfs_destroy (void ) { } +static inline void pds_vdpa_debugfs_add_pcidev(struct
pds_vdpa_pci_device*vdpa_pdev) { }
+static inline voidpds_vdpa_debugfs_del_pcidev(struct pds_vdpa_pci_device *vdpa_pdev){ }drivers/vdpa/pds/debugfs.h new file +3
-0+#endif + +#endif /* _PDS_VDPA_DEBUGFS_H_ */ drivers/vdpa/pds/pci_drv.c new file +16
-0@@ -0,0 +1,143 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/aer.h> +#include <linux/types.h> +#include <linux/vdpa.h> + +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_vdpa.h> + +#include "pci_drv.h" +#include "debugfs.h" + +static void
Quoted 8 lines
drivers/vdpa/pds/pci_drv.c line 20
Quoted 4 lines
drivers/vdpa/pds/pci_drv.c new file +4
-0+pds_vdpa_dma_action(void *data) +{ + pci_free_irq_vectors((struct pci_dev *)data); +}Nit: since we're release irq vectors, it might be better to use "
pds_vdpa_irq_action"
Sure.
Quoted 58 lines
Quoted 58 lines
drivers/vdpa/pds/pci_drv.c new file +20
-0+ +static int +pds_vdpa_pci_probe (struct pci_dev *pdev, + const struct pci_device_id *id) +{ + struct pds_vdpa_pci_device *vdpa_pdev; + struct device *dev = &pdev->dev; + int err; + + vdpa_pdev = kzalloc(sizeof(*vdpa_pdev), GFP_KERNEL); + if (!vdpa_pdev) + return -ENOMEM; + pci_set_drvdata(pdev, vdpa_pdev); + + vdpa_pdev->pdev = pdev; + vdpa_pdev->vf_id = pci_iov_vf_id(pdev); + vdpa_pdev->pci_id = PCI_DEVID(pdev->bus->number, pdev->devfn); + + /* Query system for DMA addressing limitation for the device. */ + err = dma_set_mask_and_coherent(dev,
DMA_BIT_MASK(PDS_CORE_ADDR_LEN));
+ if (err) {
+ dev_err(dev, "Unable to obtain 64-bit DMA for consistent allocations, aborting. %pe\n",
+ERR_PTR(err));
+ gotoerr_out_free_mem;
+ }drivers/vdpa/pds/pci_drv.c new file +7
-0+ + pci_enable_pcie_error_reporting(pdev); + + /* Use devres management */ + err = pcim_enable_device(pdev); + if (err) { + dev_err(dev, "Cannot enable PCI device: %pe\n",
ERR_PTR(err));
+ gotoerr_out_free_mem;
+ }
+
+ err =devm_add_action_or_reset(dev, pds_vdpa_dma_action, pdev); + if (err) {
+ dev_err(dev, "Failed adding devres for freeing irq
vectors: %pe\n",
+ERR_PTR(err));
+ gotoerr_out_pci_release_device;
+ }drivers/vdpa/pds/pci_drv.c new file +6
-0+ + pci_set_master(pdev); + + pds_vdpa_debugfs_add_pcidev(vdpa_pdev); + + dev_info(dev, "%s: PF %#04x VF %#04x (%d) vf_id %d domain %d
vdpa_aux%pvdpa_pdev%p\n",
+__func__,pci_dev_id(vdpa_pdev->pdev->physfn), +vdpa_pdev->pci_id,vdpa_pdev->pci_id,vdpa_pdev->vf_id, +pci_domain_nr(pdev->bus),vdpa_pdev->vdpa_aux,vdpa_pdev); +
Quoted 8 lines
drivers/vdpa/pds/pci_drv.c line 75
Quoted 4 lines
+ return 0;
drivers/vdpa/pds/pci_drv.c new file +3
-0+ +err_out_pci_release_device: + pci_disable_device(pdev);Do we still need to care about this consider we use devres/pcim_enable_device()?
It isn't absolutely necessary... I like how the pcim/devm stuff will clean up lost items at removal time, but I like to try to keep lost items from happening in the first place.
Quoted 193 lines
Quoted 193 lines
drivers/vdpa/pds/pci_drv.c new file +54
-0+err_out_free_mem: + pci_disable_pcie_error_reporting(pdev); + kfree(vdpa_pdev); + return err; +} + +static void +pds_vdpa_pci_remove (struct pci_dev *pdev) +{ + struct pds_vdpa_pci_device *vdpa_pdev = pci_get_drvdata(pdev); + + pds_vdpa_debugfs_del_pcidev(vdpa_pdev); + pci_clear_master(pdev); + pci_disable_pcie_error_reporting(pdev); + pci_disable_device(pdev); + kfree(vdpa_pdev); + + dev_info(&pdev->dev, "Removed\n"); +} + +static const struct pci_device_id +pds_vdpa_pci_table[] = { + { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_VDPA_VF) }, + { 0, } +}; +MODULE_DEVICE_TABLE (pci, pds_vdpa_pci_table); + +static struct pci_driver +pds_vdpa_pci_driver = { + .name = PDS_VDPA_DRV_NAME, + .id_table = pds_vdpa_pci_table, + .probe = pds_vdpa_pci_probe, + .remove = pds_vdpa_pci_remove +}; + +static void __exit +pds_vdpa_pci_cleanup (void ) +{ + pci_unregister_driver(&pds_vdpa_pci_driver); + + pds_vdpa_debugfs_destroy(); +} +module_exit (pds_vdpa_pci_cleanup); + +static int __init +pds_vdpa_pci_init (void ) +{ + int err; + + pds_vdpa_debugfs_create(); + + err = pci_register_driver(&pds_vdpa_pci_driver); + if (err) { + pr_err("%s: pci driver register failed: %pe\n",
__func__,ERR_PTR(err));
+ gotoerr_pci;
+ }drivers/vdpa/pds/pci_drv.c new file +12
-0+ + return 0; + +err_pci: + pds_vdpa_debugfs_destroy(); + return err; +} +module_init (pds_vdpa_pci_init); + +MODULE_DESCRIPTION (PDS_VDPA_DRV_DESCRIPTION ); +MODULE_AUTHOR ("Pensando Systems, Inc" ); +MODULE_LICENSE ("GPL" );drivers/vdpa/pds/pci_drv.h new file +46
-0@@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PCI_DRV_H +#define _PCI_DRV_H + +#include <linux/pci.h> +#include <linux/virtio_pci_modern.h> + +#define PDS_VDPA_DRV_NAME "pds_vdpa" +#define PDS_VDPA_DRV_DESCRIPTION "Pensando vDPA VF Device Driver" + +#define PDS_VDPA_BAR_BASE 0 +#define PDS_VDPA_BAR_INTR 2 +#define PDS_VDPA_BAR_DBELL 4 + +struct pds_dev_bar { + int index; + void __iomem *vaddr; + phys_addr_t pa; + unsigned long len; +}; + +struct pds_vdpa_intr_info { + int index; + int irq; + int qid; + char name[32]; +}; + +struct pds_vdpa_pci_device { + struct pci_dev *pdev; + struct pds_vdpa_aux *vdpa_aux; + + int vf_id; + int pci_id; + + int nintrs; + struct pds_vdpa_intr_info *intrs; + + struct dentry *dentry; + + struct virtio_pci_modern_device vd_mdev; +}; + +#endif /* _PCI_DRV_H */ include/linux/pds/pds_core_if.h +1
-0@@ -8,6 +8,7 @@ #define PCI_VENDOR_ID_PENSANDO 0x1dd8 #define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c +#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b #define PDS_CORE_BARS_MAX 4 #define PDS_CORE_PCI_BAR_DBELL 1include/linux/pds/pds_vdpa.h new file +52
-0@@ -0,0 +1,219 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_VDPA_IF_H_ +#define _PDS_VDPA_IF_H_ + +#include <linux/pds/pds_common.h> + +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_VDPA_DEV_NAME PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_VDPA_STR + +/* +* enum pds_vdpa_cmd_opcode - vDPA Device commands +*/ +enum pds_vdpa_cmd_opcode { + PDS_VDPA_CMD_INIT = 48, + PDS_VDPA_CMD_IDENT = 49, + PDS_VDPA_CMD_RESET = 51, + PDS_VDPA_CMD_VQ_RESET = 52, + PDS_VDPA_CMD_VQ_INIT = 53, + PDS_VDPA_CMD_STATUS_UPDATE = 54, + PDS_VDPA_CMD_SET_FEATURES = 55, + PDS_VDPA_CMD_SET_ATTR = 56, +}; + +/** +* struct pds_vdpa_cmd - generic command +* @opcode: Opcode +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +*/ +struct pds_vdpa_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; +}; + +/** +* struct pds_vdpa_comp - generic command completion +* @status: Status of the command (enum pds_core_status_code) +* @rsvd: Word boundary padding +* @color: Color bit +*/ +struct pds_vdpa_comp { + u8 status; + u8 rsvd[14]; + u8 color; +}; + +/** +* struct pds_vdpa_init_cmd - INIT command +* @opcode: Opcode PDS_VDPA_CMD_INIT
Quoted 8 lines
include/linux/pds/pds_vdpa.h line 56
Quoted 4 lines
include/linux/pds/pds_vdpa.h new file +4
-0+* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @len: length of config info DMA space +* @config_pa: address for DMA of virtio_net_config struct Looks like the structure is not specific to net, if yes, we may tweak the above comment to say it's the address of the device configuration space.
We're not expecting to do anything other than net, but yes we can update this comment.
Quoted 14 lines
Quoted 14 lines
include/linux/pds/pds_vdpa.h new file +12
-0+*/ +struct pds_vdpa_init_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; + __le32 len; + __le64 config_pa; +}; + +/** +* struct pds_vdpa_ident - vDPA identification data +* @hw_features: vDPA features supported by device
Quoted 8 lines
Quoted 4 lines
include/linux/pds/pds_vdpa.h new file +1
-0+* @max_vqs: max queues available (2 queues for a single queuepair)
+ * @max_qlen: log(2) of maximum number of descriptors
+ * @min_qlen: log(2) of minimum number of descriptorsNote that is you have the plan to support packed virtqueue, the qlen is not necessarily the power of 2.
Shouldn't be a problem - this is only a way of giving us the max queue len in an 8-bit value, it doesn't mean we can only deal with power-of-2 actual use values.
Quoted 82 lines
Quoted 82 lines
+ *
+ * This struct is used in a DMA block that is set up for thePDS_VDPA_CMD_IDENT
+ * transaction. Set up the DMA block and send the address in the IDENT cmd
+ * data, the DSC will write the ident information, then we can remove the DMA
+ * block after reading the answer. If the completion status is 0, then there
+ * is valid information, else there was an error and the data should be invalid.
+ */include/linux/pds/pds_vdpa.h new file +14
-0+struct pds_vdpa_ident { + __le64 hw_features; + __le16 max_vqs; + __le16 max_qlen; + __le16 min_qlen; +}; + +/** +* struct pds_vdpa_ident_cmd - IDENT command +* @opcode: Opcode PDS_VDPA_CMD_IDENT +* @rsvd: Word boundary padding +* @vf_id: VF id +* @len: length of ident info DMA space +* @ident_pa: address for DMA of ident info (struct
pds_vdpa_ident)
+ * only used for this transaction, then forgotten by DSC
+ */include/linux/pds/pds_vdpa.h new file +40
-0+struct pds_vdpa_ident_cmd { + u8 opcode; + u8 rsvd; + __le16 vf_id; + __le32 len; + __le64 ident_pa; +}; + +/** +* struct pds_vdpa_status_cmd - STATUS_UPDATE command +* @opcode: Opcode PDS_VDPA_CMD_STATUS_UPDATE +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @status: new status bits +*/ +struct pds_vdpa_status_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; + u8 status; +}; + +/** +* enum pds_vdpa_attr - List of VDPA device attributes +* @PDS_VDPA_ATTR_MAC: MAC address +* @PDS_VDPA_ATTR_MAX_VQ_PAIRS: Max virtqueue pairs +*/ +enum pds_vdpa_attr { + PDS_VDPA_ATTR_MAC = 1, + PDS_VDPA_ATTR_MAX_VQ_PAIRS = 2, +}; + +/** +* struct pds_vdpa_setattr_cmd - SET_ATTR command +* @opcode: Opcode PDS_VDPA_CMD_SET_ATTR +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @attr: attribute to be changed (enum pds_vdpa_attr) +* @pad: Word boundary padding +* @mac: new mac address to be assigned as vdpa device address
+ * @max_vq_pairs: new limit of virtqueue pairs
+ */
+structpds_vdpa_setattr_cmd{
+ u8 opcode;
+ u8vdpa_index;
+__le16vf_id;
+ u8 attr;
+ u8 pad[3];
+ union {
Quoted 8 lines
Quoted 2 lines
+ u8 mac[6];
+__le16max_vq_pairs;So does this mean if we want to set both mac and
max_vq_paris, we need two commands? The seems to be less efficient, since the mgmt layer could provision more attributes here. Can we pack all attributes into a single command?
Yes, that is how the cmd is set up, similar to other setattr commands work in our firmware. This was driven originally by our ionic's struct ionic_lif_setattr_cmd where you can see that the combined fields in the union would be much greater than the available 64 bytes for the request.
This is a new device, but we wanted to keep the cmd operations similar.
Quoted 15 lines
include/linux/pds/pds_vdpa.h line 154
Quoted 11 lines
+ }
__packed;include/linux/pds/pds_vdpa.h new file +8
-0+}; + +/** +* struct pds_vdpa_vq_init_cmd - queue init command +* @opcode: Opcode PDS_VDPA_CMD_VQ_INIT +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @qid: Queue id (bit0 clear = rx, bit0 set = tx, qid=N is ctrlq) I wonder any reason we need to design it like this, it would be better to make it general to be used by other type of virtio devices.
There's no plan to go beyond net devices, but if we find we need to revisit that, we can version the interface and add what is needed.
Quoted 11 lines
Quoted 8 lines
+ * @len: log(2) of max descriptor count
+ * @desc_addr: DMA address of descriptor area
+ * @avail_addr: DMA address of available descriptors (aka driver area)
+ * @used_addr: DMA address of used descriptors (aka device area)
+ * @intr_index: interrupt indexIs this something like MSI-X vector?
It is an index into the VF's list of MSI-x vectors
Quoted 17 lines
Quoted 17 lines
+ */
include/linux/pds/pds_vdpa.h new file +14
-0+struct pds_vdpa_vq_init_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; + __le16 qid; + __le16 len; + __le64 desc_addr; + __le64 avail_addr; + __le64 used_addr; + __le16 intr_index; +}; + +/** +* struct pds_vdpa_vq_init_comp - queue init completion
Quoted 8 lines
Quoted 5 lines
+ * @status: Status of the command (enum
pds_core_status_code)
+ * @hw_qtype: HW queue type, used in doorbell selection
+ * @hw_qindex: HW queue index, used in doorbell selection
+ * @rsvd: Word boundary padding
+ * @color: Color bitMore comment is needed to know the how to use this color bit.
Sure. I'll add something somewhere about how this is used in place of a tail pointer.
Quoted 17 lines
Quoted 4 lines
+ */
include/linux/pds/pds_vdpa.h new file +1
-0+struct pds_vdpa_vq_init_comp {include/linux/pds/pds_vdpa.h line 191
Quoted 10 lines
include/linux/pds/pds_vdpa.h new file +10
-0+ u8 status; + u8 hw_qtype; + __le16 hw_qindex; + u8 rsvd[11]; + u8 color; +}; + +/** +* struct pds_vdpa_vq_reset_cmd - queue reset command +* @opcode: Opcode PDS_VDPA_CMD_VQ_RESET Is there a chance that we could have more type of opcode here?
I'm not sure I understand this question... yes it is possible we can add to the enum pds_vdpa_cmd_opcode list.
sln
Quoted 33 lines
Thanks
Quoted 28 lines
+ * @vdpa_index: Index for vdpa subdevice
+ * @vf_id: VF id
+ * @qid: Queue id
+ */include/linux/pds/pds_vdpa.h new file +24
-0+struct pds_vdpa_vq_reset_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; + __le16 qid; +}; + +/** +* struct pds_vdpa_set_features_cmd - set hw features +* @opcode: Opcode PDS_VDPA_CMD_SET_FEATURES +* @vdpa_index: Index for vdpa subdevice +* @vf_id: VF id +* @rsvd: Word boundary padding +* @features: Feature bit mask +*/ +struct pds_vdpa_set_features_cmd { + u8 opcode; + u8 vdpa_index; + __le16 vf_id; + __le32 rsvd; + __le64 features; +}; + +#endif /* _PDS_VDPA_IF_H_ */
RFC PATCH 15/19 pds_vdpa: virtio bar setup for vdpa
In reply to Shannon NelsonThe PDS vDPA device has a virtio BAR for describing itself, and the pds_vdpa driver needs to access it. Here we copy liberally from the existing drivers/virtio/virtio_pci_modern_dev.c as it has what we need, but we need to modify it so that it can work with our device id and so we can use our own DMA mask.
We suspect there is room for discussion here about making the existing code a little more flexible, but we thought we'd at least start the discussion here.
--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/pci_drv.c | 10 ++ drivers/vdpa/pds/pci_drv.h | 2 + drivers/vdpa/pds/virtio_pci.c | 283 ++++++++++++++++++++++++++++++++++ 4 files changed, 297 insertions(+), 1 deletion(-) create mode 100644 drivers/vdpa/pds/virtio_pci.c
drivers/vdpa/pds/Makefile +2 -1
@@ -4,4 +4,5 @@ obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o pds_vdpa-y := pci_drv.o \ - debugfs.o + debugfs.o \ + virtio_pci.o
drivers/vdpa/pds/pci_drv.c +10 -0
@@ -44,6 +44,14 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, goto err_out_free_mem; } + vdpa_pdev->vd_mdev.pci_dev = pdev; + err =pds_vdpa_probe_virtio (&vdpa_pdev->vd_mdev); +if (err) { +dev_err (dev,"Unable to probe for virtio configuration: %pe\n" , +ERR_PTR (err)); + goto err_out_free_mem; + } +pci_enable_pcie_error_reporting (pdev);/* Use devres management */
@@ -74,6 +82,7 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, err_out_pci_release_device:pci_disable_device (pdev); err_out_free_mem: +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);pci_disable_pcie_error_reporting (pdev);kfree (vdpa_pdev);return err;
@@ -88,6 +97,7 @@ pds_vdpa_pci_remove(struct pci_dev *pdev)pci_clear_master (pdev);pci_disable_pcie_error_reporting (pdev);pci_disable_device (pdev); +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);kfree (vdpa_pdev);dev_info (&pdev->dev,"Removed\n" );
drivers/vdpa/pds/pci_drv.h +2 -0
@@ -43,4 +43,6 @@ struct pds_vdpa_pci_device { struct virtio_pci_modern_device vd_mdev; }; +int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev); +void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev);#endif /* _PCI_DRV_H */
drivers/vdpa/pds/virtio_pci.c new file +283 -0
283 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH net-next 15/19] pds_vdpa: virtio bar setup for vdpa
Quoted 9 lines · Shannon Nelson
The PDS vDPA device has a virtio BAR for describing itself, and the
pds_vdpadriver needs to access it. Here we copy liberally from the existingdrivers/virtio/virtio_pci_modern_dev.cas it has what we need, but we need to modify it so that it can work with our device id and so we can use our own DMA mask.We suspect there is room for discussion here about making the existing code a little more flexible, but we thought we'd at least start the discussion here.
Exactly, since the virtio_pci_modern_dev.c is a library, we could tweak it to allow the caller to pass the device_id with the DMA mask. Then we can avoid code/bug duplication here.
Thanks
Quoted 346 lines
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/pci_drv.c | 10 ++ drivers/vdpa/pds/pci_drv.h | 2 + drivers/vdpa/pds/virtio_pci.c | 283 ++++++++++++++++++++++++++++++++++ 4 files changed, 297 insertions(+), 1 deletion(-)create mode 100644
drivers/vdpa/pds/virtio_pci.cdrivers/vdpa/pds/Makefile +2
-1@@ -4,4 +4,5 @@ obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o pds_vdpa-y := pci_drv.o \ - debugfs.o + debugfs.o \ + virtio_pci.odrivers/vdpa/pds/pci_drv.c +10
-0@@ -44,6 +44,14 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, goto err_out_free_mem; } + vdpa_pdev->vd_mdev.pci_dev = pdev; + err =pds_vdpa_probe_virtio (&vdpa_pdev->vd_mdev); +if (err) { +dev_err (dev,"Unable to probe for virtio configuration: %pe\n" , +ERR_PTR (err)); + goto err_out_free_mem; + } +pci_enable_pcie_error_reporting (pdev);/* Use devres management */ @@ -74,6 +82,7 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, err_out_pci_release_device:pci_disable_device (pdev); err_out_free_mem: +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);pci_disable_pcie_error_reporting (pdev);kfree (vdpa_pdev);return err;@@ -88,6 +97,7 @@ pds_vdpa_pci_remove(struct pci_dev *pdev)pci_clear_master (pdev);pci_disable_pcie_error_reporting (pdev);pci_disable_device (pdev); +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);kfree (vdpa_pdev);dev_info (&pdev->dev,"Removed\n" );drivers/vdpa/pds/pci_drv.h +2
-0@@ -43,4 +43,6 @@ struct pds_vdpa_pci_device { struct virtio_pci_modern_device vd_mdev; }; +int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev); +void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev);#endif /* _PCI_DRV_H */ drivers/vdpa/pds/virtio_pci.c new file +273
-0@@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +/* +* adapted from drivers/virtio/virtio_pci_modern_dev.c, v6.0-rc1 +*/ + +#include <linux/virtio_pci_modern.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/delay.h> + +#include "pci_drv.h" + +/* +* pds_vdpa_map_capability - map a part of virtio pci capability +* @mdev: the modern virtio-pci device +* @off: offset of the capability +* @minlen: minimal length of the capability +* @align: align requirement +* @start: start from the capability +* @size: map size +* @len: the length that is actually mapped +* @pa: physical address of the capability +* +* Returns the io address of for the part of the capability +*/ +static void __iomem * +pds_vdpa_map_capability (struct virtio_pci_modern_device *mdev,int off, +size_t minlen,u32 align,u32 start,u32 size, +size_t *len,resource_size_t *pa) +{ +struct pci_dev *dev = mdev->pci_dev; +u8 bar; +u32 offset, length; +void __iomem *p; + +pci_read_config_byte (dev, off + offsetof(struct virtio_pci_cap , + bar), + &bar); +pci_read_config_dword (dev, off + offsetof(struct virtio_pci_cap , offset), + &offset); +pci_read_config_dword (dev, off + offsetof(struct virtio_pci_cap , length), + &length); + +/* Check if the BAR may have changed since we requested the region. */ +if (bar >=PCI_STD_NUM_BARS || !(mdev->modern_bars & (1 << bar))) { +dev_err (&dev->dev, +"virtio_pci: bar unexpectedly changed to %u\n" , bar); +return NULL ; + } + +if (length <= start) { +dev_err (&dev->dev, +"virtio_pci: bad capability len %u (>%u expected)\n" , + length, start); +return NULL ; + } + +if (length - start < minlen) { +dev_err (&dev->dev, +"virtio_pci: bad capability len %u (>=%zu expected)\n" , + length, minlen); +return NULL ; + } + + length -= start; + +if (start + offset < offset) { +dev_err (&dev->dev, +"virtio_pci: map wrap-around %u+%u\n" , + start, offset); +return NULL ; + } + + offset += start; + +if (offset & (align -1 )) { +dev_err (&dev->dev, +"virtio_pci: offset %u not aligned to %u\n" , + offset, align); +return NULL ; + } + +if (length > size) + length = size; + +if (len) + *len = length; + +if (minlen + offset < minlen || + minlen + offset >pci_resource_len (dev, bar)) { +dev_err (&dev->dev, +"virtio_pci: map virtio %zu@%u out of range on bar %i length %lu\n" , + minlen, offset, + bar, (unsigned long )pci_resource_len (dev, bar)); +return NULL ; + } + + p =pci_iomap_range (dev, bar, offset, length); +if (!p) +dev_err (&dev->dev, +"virtio_pci: unable to map virtio %u@%u on bar %i\n" , + length, offset, bar); +else if (pa) + *pa =pci_resource_start (dev, bar) + offset; + +return p; +} + +/** +* virtio_pci_find_capability - walk capabilities to find device info. +* @dev: the pci device +* @cfg_type: the VIRTIO_PCI_CAP_* value we seek +* @ioresource_types: IORESOURCE_MEM and/or IORESOURCE_IO. +* @bars: the bitmask of BARs +* +* Returns offset of the capability, or 0. +*/ +static inline int virtio_pci_find_capability (struct pci_dev *dev,u8 cfg_type, +u32 ioresource_types,int *bars) +{ +int pos; + +for (pos =pci_find_capability (dev,PCI_CAP_ID_VNDR ); + pos >0 ; + pos =pci_find_next_capability (dev, pos,PCI_CAP_ID_VNDR )) { +u8 type, bar; + +pci_read_config_byte (dev, pos + offsetof(struct virtio_pci_cap , + cfg_type), + &type); +pci_read_config_byte (dev, pos + offsetof(struct virtio_pci_cap , + bar), + &bar); + +/* Ignore structures with reserved BAR values */ +if (bar >=PCI_STD_NUM_BARS ) +continue ; + +if (type == cfg_type) { +if (pci_resource_len (dev, bar) && +pci_resource_flags (dev, bar) & ioresource_types) { + *bars |= (1 << bar); +return pos; + } + } + } +return 0 ; +} + +/* +* pds_vdpa_probe_virtio: probe the modern virtio pci device, note that the +* caller is required to enable PCI device before calling this function. +* @mdev: the modern virtio-pci device +* +* Return 0 on succeed otherwise fail +*/ +int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev) +{ +struct pci_dev *pci_dev = mdev->pci_dev; +int err, common, isr, notify, device; +u32 notify_length; +u32 notify_offset; + +/* check for a common config: if not, use legacy mode (bar 0). */ + common =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_COMMON_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); +if (!common) { +dev_info (&pci_dev->dev, +"virtio_pci: missing common config\n" ); +return -ENODEV ; + } + +/* If common is there, these should be too... */ + isr =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_ISR_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); + notify =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_NOTIFY_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); +if (!isr || !notify) { +dev_err (&pci_dev->dev, +"virtio_pci: missing capabilities %i/%i/%i\n" , + common, isr, notify); +return -EINVAL ; + } + +/* Device capability is only mandatory for devices that have +* device-specific configuration. +*/ + device =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_DEVICE_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); + + err =pci_request_selected_regions (pci_dev, mdev->modern_bars, +"virtio-pci-modern" ); +if (err) +return err; + + err = -EINVAL ; + mdev->common =pds_vdpa_map_capability (mdev, common, +sizeof (struct virtio_pci_common_cfg ),4 , +0 ,sizeof (struct virtio_pci_common_cfg ), +NULL ,NULL ); +if (!mdev->common) + goto err_map_common; + mdev->isr =pds_vdpa_map_capability (mdev, isr,sizeof (u8),1 , +0 ,1 , +NULL ,NULL ); +if (!mdev->isr) + goto err_map_isr; + +/* Read notify_off_multiplier from config space. */ +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + notify_off_multiplier), + &mdev->notify_offset_multiplier); +/* Read notify length and offset from config space. */ +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + cap.length), + ¬ify_length); + +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + cap.offset), + ¬ify_offset); + +/* We don't know how many VQs we'll map, ahead of the time. +* If notify length is small, map it all now. +* Otherwise, map each VQ individually later. +*/ +if ((u64 )notify_length + (notify_offset %PAGE_SIZE ) <=PAGE_SIZE ) { + mdev->notify_base =pds_vdpa_map_capability (mdev, notify, +2 ,2 , +0 , notify_length, + &mdev->notify_len, + &mdev->notify_pa); +if (!mdev->notify_base) + goto err_map_notify; + }else { + mdev->notify_map_cap = notify; + } + +/* Again, we don't know how much we should map, but PAGE_SIZE +* is more than enough for all existing devices. +*/ +if (device) { + mdev->device =pds_vdpa_map_capability (mdev, device,0 ,4 , +0 ,PAGE_SIZE , + &mdev->device_len, +NULL ); +if (!mdev->device) + goto err_map_device; + } + +return 0 ; + +err_map_device: +if (mdev->notify_base) +pci_iounmap (pci_dev, mdev->notify_base); +err_map_notify: +pci_iounmap (pci_dev, mdev->isr); +err_map_isr: +pci_iounmap (pci_dev, mdev->common); +err_map_common: +pci_release_selected_regions (pci_dev, mdev->modern_bars); +return err; +} + +void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev) +{
drivers/vdpa/pds/virtio_pci.c line 283
Quoted 10 lines
drivers/vdpa/pds/virtio_pci.c new file +10
-0+struct pci_dev *pci_dev = mdev->pci_dev; + +if (mdev->device) +pci_iounmap (pci_dev, mdev->device); +if (mdev->notify_base) +pci_iounmap (pci_dev, mdev->notify_base); +pci_iounmap (pci_dev, mdev->isr); +pci_iounmap (pci_dev, mdev->common); +pci_release_selected_regions (pci_dev, mdev->modern_bars); +}
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 15/19] pds_vdpa: virtio bar setup for vdpa
Quoted 9 lines · Jason Wang
Quoted 6 lines · Shannon Nelson
The PDS vDPA device has a virtio BAR for describing itself, and the
pds_vdpadriver needs to access it. Here we copy liberally from the existingdrivers/virtio/virtio_pci_modern_dev.cas it has what we need, but we need to modify it so that it can work with our device id and so we can use our own DMA mask.
Quoted 8 lines
Quoted 3 lines
We suspect there is room for discussion here about making the existing code a little more flexible, but we thought we'd at least start the discussion here.
Exactly, since the
virtio_pci_modern_dev.cis a library, we could tweak it to allow the caller to pass thedevice_idwith the DMA mask. Then we can avoid code/bug duplication here.
Btw, I found only isr/notification were used but not the others? If this is true, we can avoid mapping those capabilities.
Thanks
Quoted 352 lines
Thanks
Quoted 348 lines
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/pci_drv.c | 10 ++ drivers/vdpa/pds/pci_drv.h | 2 + drivers/vdpa/pds/virtio_pci.c | 283 ++++++++++++++++++++++++++++++++++ 4 files changed, 297 insertions(+), 1 deletion(-)create mode 100644
drivers/vdpa/pds/virtio_pci.cdrivers/vdpa/pds/Makefile +2
-1@@ -4,4 +4,5 @@ obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o pds_vdpa-y := pci_drv.o \ - debugfs.o + debugfs.o \ + virtio_pci.odrivers/vdpa/pds/pci_drv.c +10
-0@@ -44,6 +44,14 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, goto err_out_free_mem; } + vdpa_pdev->vd_mdev.pci_dev = pdev; + err =pds_vdpa_probe_virtio (&vdpa_pdev->vd_mdev); +if (err) { +dev_err (dev,"Unable to probe for virtio configuration: %pe\n" , +ERR_PTR (err)); + goto err_out_free_mem; + } +pci_enable_pcie_error_reporting (pdev);/* Use devres management */ @@ -74,6 +82,7 @@ pds_vdpa_pci_probe(struct pci_dev *pdev, err_out_pci_release_device:pci_disable_device (pdev); err_out_free_mem: +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);pci_disable_pcie_error_reporting (pdev);kfree (vdpa_pdev);return err;@@ -88,6 +97,7 @@ pds_vdpa_pci_remove(struct pci_dev *pdev)pci_clear_master (pdev);pci_disable_pcie_error_reporting (pdev);pci_disable_device (pdev); +pds_vdpa_remove_virtio (&vdpa_pdev->vd_mdev);kfree (vdpa_pdev);dev_info (&pdev->dev,"Removed\n" );drivers/vdpa/pds/pci_drv.h +2
-0@@ -43,4 +43,6 @@ struct pds_vdpa_pci_device { struct virtio_pci_modern_device vd_mdev; }; +int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev); +void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev);#endif /* _PCI_DRV_H */ drivers/vdpa/pds/virtio_pci.c new file +275
-0@@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +/* +* adapted from drivers/virtio/virtio_pci_modern_dev.c, v6.0-rc1 +*/ + +#include <linux/virtio_pci_modern.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/delay.h> + +#include "pci_drv.h" + +/* +* pds_vdpa_map_capability - map a part of virtio pci capability +* @mdev: the modern virtio-pci device +* @off: offset of the capability +* @minlen: minimal length of the capability +* @align: align requirement +* @start: start from the capability +* @size: map size +* @len: the length that is actually mapped +* @pa: physical address of the capability +* +* Returns the io address of for the part of the capability +*/ +static void __iomem * +pds_vdpa_map_capability (struct virtio_pci_modern_device *mdev,int off, +size_t minlen,u32 align,u32 start,u32 size, +size_t *len,resource_size_t *pa) +{ +struct pci_dev *dev = mdev->pci_dev; +u8 bar; +u32 offset, length; +void __iomem *p; + +pci_read_config_byte (dev, off + offsetof(struct virtio_pci_cap , + bar), + &bar); +pci_read_config_dword (dev, off + offsetof(struct virtio_pci_cap , offset), + &offset); +pci_read_config_dword (dev, off + offsetof(struct virtio_pci_cap , length), + &length); + +/* Check if the BAR may have changed since we requested the region. */ +if (bar >=PCI_STD_NUM_BARS || !(mdev->modern_bars & (1 << bar))) { +dev_err (&dev->dev, +"virtio_pci: bar unexpectedly changed to %u\n" , bar); +return NULL ; + } + +if (length <= start) { +dev_err (&dev->dev, +"virtio_pci: bad capability len %u (>%u expected)\n" , + length, start); +return NULL ; + } + +if (length - start < minlen) { +dev_err (&dev->dev, +"virtio_pci: bad capability len %u (>=%zu expected)\n" , + length, minlen); +return NULL ; + } + + length -= start; + +if (start + offset < offset) { +dev_err (&dev->dev, +"virtio_pci: map wrap-around %u+%u\n" , + start, offset); +return NULL ; + } + + offset += start; + +if (offset & (align -1 )) { +dev_err (&dev->dev, +"virtio_pci: offset %u not aligned to %u\n" , + offset, align); +return NULL ; + } + +if (length > size) + length = size; + +if (len) + *len = length; + +if (minlen + offset < minlen || + minlen + offset >pci_resource_len (dev, bar)) { +dev_err (&dev->dev, +"virtio_pci: map virtio %zu@%u out of range on bar %i length %lu\n" , + minlen, offset, + bar, (unsigned long )pci_resource_len (dev, bar)); +return NULL ; + } + + p =pci_iomap_range (dev, bar, offset, length); +if (!p) +dev_err (&dev->dev, +"virtio_pci: unable to map virtio %u@%u on bar %i\n" , + length, offset, bar); +else if (pa) + *pa =pci_resource_start (dev, bar) + offset; + +return p; +} + +/** +* virtio_pci_find_capability - walk capabilities to find device info. +* @dev: the pci device +* @cfg_type: the VIRTIO_PCI_CAP_* value we seek +* @ioresource_types: IORESOURCE_MEM and/or IORESOURCE_IO. +* @bars: the bitmask of BARs +* +* Returns offset of the capability, or 0. +*/ +static inline int virtio_pci_find_capability (struct pci_dev *dev,u8 cfg_type, +u32 ioresource_types,int *bars) +{ +int pos; + +for (pos =pci_find_capability (dev,PCI_CAP_ID_VNDR ); + pos >0 ; + pos =pci_find_next_capability (dev, pos,PCI_CAP_ID_VNDR )) { +u8 type, bar; + +pci_read_config_byte (dev, pos + offsetof(struct virtio_pci_cap , + cfg_type), + &type); +pci_read_config_byte (dev, pos + offsetof(struct virtio_pci_cap , + bar), + &bar); + +/* Ignore structures with reserved BAR values */ +if (bar >=PCI_STD_NUM_BARS ) +continue ; + +if (type == cfg_type) { +if (pci_resource_len (dev, bar) && +pci_resource_flags (dev, bar) & ioresource_types) { + *bars |= (1 << bar); +return pos; + } + } + } +return 0 ; +} + +/* +* pds_vdpa_probe_virtio: probe the modern virtio pci device, note that the +* caller is required to enable PCI device before calling this function. +* @mdev: the modern virtio-pci device +* +* Return 0 on succeed otherwise fail +*/ +int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev) +{ +struct pci_dev *pci_dev = mdev->pci_dev; +int err, common, isr, notify, device; +u32 notify_length; +u32 notify_offset; + +/* check for a common config: if not, use legacy mode (bar 0). */ + common =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_COMMON_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); +if (!common) { +dev_info (&pci_dev->dev, +"virtio_pci: missing common config\n" ); +return -ENODEV ; + } + +/* If common is there, these should be too... */ + isr =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_ISR_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); + notify =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_NOTIFY_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); +if (!isr || !notify) { +dev_err (&pci_dev->dev, +"virtio_pci: missing capabilities %i/%i/%i\n" , + common, isr, notify); +return -EINVAL ; + } + +/* Device capability is only mandatory for devices that have +* device-specific configuration. +*/ + device =virtio_pci_find_capability (pci_dev,VIRTIO_PCI_CAP_DEVICE_CFG , +IORESOURCE_IO |IORESOURCE_MEM , + &mdev->modern_bars); + + err =pci_request_selected_regions (pci_dev, mdev->modern_bars, +"virtio-pci-modern" ); +if (err) +return err; + + err = -EINVAL ; + mdev->common =pds_vdpa_map_capability (mdev, common, +sizeof (struct virtio_pci_common_cfg ),4 , +0 ,sizeof (struct virtio_pci_common_cfg ), +NULL ,NULL ); +if (!mdev->common) + goto err_map_common; + mdev->isr =pds_vdpa_map_capability (mdev, isr,sizeof (u8),1 , +0 ,1 , +NULL ,NULL ); +if (!mdev->isr) + goto err_map_isr; + +/* Read notify_off_multiplier from config space. */ +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + notify_off_multiplier), + &mdev->notify_offset_multiplier); +/* Read notify length and offset from config space. */ +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + cap.length), + ¬ify_length); + +pci_read_config_dword (pci_dev, + notify + offsetof(struct virtio_pci_notify_cap , + cap.offset), + ¬ify_offset); + +/* We don't know how many VQs we'll map, ahead of the time. +* If notify length is small, map it all now. +* Otherwise, map each VQ individually later. +*/ +if ((u64 )notify_length + (notify_offset %PAGE_SIZE ) <=PAGE_SIZE ) { + mdev->notify_base =pds_vdpa_map_capability (mdev, notify, +2 ,2 , +0 , notify_length, + &mdev->notify_len, + &mdev->notify_pa); +if (!mdev->notify_base) + goto err_map_notify; + }else { + mdev->notify_map_cap = notify; + } + +/* Again, we don't know how much we should map, but PAGE_SIZE +* is more than enough for all existing devices. +*/ +if (device) { + mdev->device =pds_vdpa_map_capability (mdev, device,0 ,4 , +0 ,PAGE_SIZE , + &mdev->device_len, +NULL ); +if (!mdev->device) + goto err_map_device; + } + +return 0 ; + +err_map_device: +if (mdev->notify_base) +pci_iounmap (pci_dev, mdev->notify_base); +err_map_notify: +pci_iounmap (pci_dev, mdev->isr); +err_map_isr: +pci_iounmap (pci_dev, mdev->common); +err_map_common: +pci_release_selected_regions (pci_dev, mdev->modern_bars); +return err; +} + +void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev) +{ +struct pci_dev *pci_dev = mdev->pci_dev; +
Quoted 8 lines
drivers/vdpa/pds/virtio_pci.c line 283
Quoted 8 lines
drivers/vdpa/pds/virtio_pci.c new file +8
-0+if (mdev->device) +pci_iounmap (pci_dev, mdev->device); +if (mdev->notify_base) +pci_iounmap (pci_dev, mdev->notify_base); +pci_iounmap (pci_dev, mdev->isr); +pci_iounmap (pci_dev, mdev->common); +pci_release_selected_regions (pci_dev, mdev->modern_bars); +}
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 15/19] pds_vdpa: virtio bar setup for vdpa
Quoted 8 lines · Jason Wang
Quoted 7 lines · Jason Wang
Quoted 6 lines · Shannon Nelson
The PDS vDPA device has a virtio BAR for describing itself, and the
pds_vdpadriver needs to access it. Here we copy liberally from the existingdrivers/virtio/virtio_pci_modern_dev.cas it has what we need, but we need to modify it so that it can work with our device id and so we can use our own DMA mask.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
We suspect there is room for discussion here about making the existing code a little more flexible, but we thought we'd at least start the discussion here.
Exactly, since the
virtio_pci_modern_dev.cis a library, we could tweak it to allow the caller to pass thedevice_idwith the DMA mask. Then we can avoid code/bug duplication here.
I'll look into possible mods for it, although I'm not sure how quickly I can get to that... maybe a v+1 along the way.
Quoted 3 lines
Btw, I found only isr/notification were used but not the others? If this is true, we can avoid mapping those capabilities.
I'll keep this in mind.
sln
RFC PATCH 16/19 pds_vdpa: add auxiliary driver
In reply to Shannon NelsonThe auxiliary_bus driver is registered after the PCI driver is loaded, and when the pds_core has created a device for it, after the VF has been enabled, this driver gets probed. It then registers itself with the DSC through the pds_core in order to start the firmware services for this device.
--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/aux_drv.c | 123 +++++++++++++++++++++++++++++++++++++ drivers/vdpa/pds/aux_drv.h | 28 +++++++++ drivers/vdpa/pds/debugfs.c | 23 +++++++ drivers/vdpa/pds/debugfs.h | 2 + drivers/vdpa/pds/pci_drv.c | 19 ++++++ drivers/vdpa/pds/pci_drv.h | 1 + 7 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 drivers/vdpa/pds/aux_drv.c create mode 100644 drivers/vdpa/pds/aux_drv.h
drivers/vdpa/pds/Makefile +2 -1
@@ -3,6 +3,7 @@ obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o -pds_vdpa-y := pci_drv.o \ +pds_vdpa-y := aux_drv.o \ + pci_drv.o \ debugfs.o \ virtio_pci.o
drivers/vdpa/pds/aux_drv.c new file +123 -0
123 lines not on this page · raw message
drivers/vdpa/pds/aux_drv.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _AUX_DRV_H_ +#define _AUX_DRV_H_ + +#include <linux/auxiliary_bus.h> +#include <linux/interrupt.h> +#include <linux/io.h> + +struct pds_vdpa_pci_device ; + +struct pds_vdpa_aux { +struct pds_auxiliary_dev *padev; +struct pds_auxiliary_drv padrv; + +struct pds_vdpa_pci_device *vdpa_vf; +struct vdpa_mgmt_dev vdpa_mdev; +struct pds_vdpa_device *pdsv; + +struct pds_vdpa_ident ident; +bool local_mac_bit; +}; + +struct auxiliary_driver * +pds_vdpa_aux_driver_info (void ); + +#endif /* _AUX_DRV_H_ */
drivers/vdpa/pds/debugfs.c +23 -0
@@ -4,10 +4,14 @@#include <linux/module.h> #include <linux/pci.h> #include <linux/types.h> +#include <linux/vdpa.h> #include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> #include <linux/pds/pds_vdpa.h> +#include "aux_drv.h" #include "pci_drv.h" #include "debugfs.h"
@@ -41,4 +45,23 @@ pds_vdpa_debugfs_del_pcidev(struct pds_vdpa_pci_device *vdpa_pdev) vdpa_pdev->dentry = NULL; } +static int +identity_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_aux *vdpa_aux = seq->private; + +seq_printf (seq,"aux_dev: %s\n" , +dev_name (&vdpa_aux->padev->aux_dev.dev)); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (identity); + +void +pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) +{ +debugfs_create_file ("identity" ,0400 , vdpa_aux->vdpa_vf->dentry, + vdpa_aux, &identity_fops); +} +#endif /* CONFIG_DEBUG_FS */
drivers/vdpa/pds/debugfs.h +2 -0
@@ -12,11 +12,13 @@ void pds_vdpa_debugfs_create(void);void pds_vdpa_debugfs_destroy (void );void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev); +void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux);#else static inline void pds_vdpa_debugfs_create (void ) { }static inline void pds_vdpa_debugfs_destroy (void ) { }static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { } +static inline void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) { }#endif #endif /* _PDS_VDPA_DEBUGFS_H_ */
drivers/vdpa/pds/pci_drv.c +19 -0
@@ -6,11 +6,15 @@#include <linux/aer.h> #include <linux/types.h> #include <linux/vdpa.h> +#include <linux/auxiliary_bus.h> #include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> #include <linux/pds/pds_vdpa.h> #include "pci_drv.h" +#include "aux_drv.h" #include "debugfs.h" static void
@@ -118,9 +122,16 @@ pds_vdpa_pci_driver = { .remove = pds_vdpa_pci_remove }; +bool +pds_vdpa_is_vdpa_pci_driver (struct pci_dev *pdev) +{ +return (to_pci_driver (pdev->dev.driver) == &pds_vdpa_pci_driver); +} + static void __exitpds_vdpa_pci_cleanup (void ) { +auxiliary_driver_unregister (pds_vdpa_aux_driver_info ());pci_unregister_driver (&pds_vdpa_pci_driver);pds_vdpa_debugfs_destroy ();
@@ -140,8 +151,16 @@ pds_vdpa_pci_init(void) goto err_pci; } + err =auxiliary_driver_register (pds_vdpa_aux_driver_info ()); +if (err) { +pr_err ("%s: aux driver register failed: %pe\n" , __func__,ERR_PTR (err)); + goto err_aux; + } +return 0 ; +err_aux: +pci_unregister_driver (&pds_vdpa_pci_driver); err_pci:pds_vdpa_debugfs_destroy ();return err;
drivers/vdpa/pds/pci_drv.h +1 -0
@@ -43,6 +43,7 @@ struct pds_vdpa_pci_device { struct virtio_pci_modern_device vd_mdev; }; +bool pds_vdpa_is_vdpa_pci_driver (struct pci_dev *pdev);int pds_vdpa_probe_virtio (struct virtio_pci_modern_device *mdev);void pds_vdpa_remove_virtio (struct virtio_pci_modern_device *mdev);#endif /* _PCI_DRV_H */
Signature · 1 line
2.17.1
RFC PATCH 17/19 pds_vdpa: add vdpa config client commands
In reply to Shannon NelsonThese are the adminq commands that will be needed for setting up and using the vDPA device.
--- drivers/vdpa/pds/Makefile | 1 + drivers/vdpa/pds/cmds.c | 266 ++++++++++++++++++++++++++++++++++++ drivers/vdpa/pds/cmds.h | 17 +++ drivers/vdpa/pds/vdpa_dev.h | 60 ++++++++ 4 files changed, 344 insertions(+) create mode 100644 drivers/vdpa/pds/cmds.c create mode 100644 drivers/vdpa/pds/cmds.h create mode 100644 drivers/vdpa/pds/vdpa_dev.h
drivers/vdpa/pds/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o pds_vdpa-y := aux_drv.o \ + cmds.o \ pci_drv.o \ debugfs.o \ virtio_pci.o
drivers/vdpa/pds/cmds.c new file +266 -0
266 lines not on this page · raw message
drivers/vdpa/pds/cmds.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _VDPA_CMDS_H_ +#define _VDPA_CMDS_H_ + +int pds_vdpa_init_hw (struct pds_vdpa_device *pdsv); + +int pds_vdpa_cmd_reset (struct pds_vdpa_device *pdsv); +int pds_vdpa_cmd_set_status (struct pds_vdpa_device *pdsv,u8 status); +int pds_vdpa_cmd_set_mac (struct pds_vdpa_device *pdsv,u8 *mac); +int pds_vdpa_cmd_set_max_vq_pairs (struct pds_vdpa_device *pdsv,u16 max_vqp); +int pds_vdpa_cmd_init_vq (struct pds_vdpa_device *pdsv,u16 qid, +struct pds_vdpa_vq_info *vq_info); +int pds_vdpa_cmd_reset_vq (struct pds_vdpa_device *pdsv,u16 qid); +int pds_vdpa_cmd_set_features (struct pds_vdpa_device *pdsv,u64 features); +#endif /* _VDPA_CMDS_H_ */
drivers/vdpa/pds/vdpa_dev.h new file +60 -0
60 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH net-next 17/19] pds_vdpa: add vdpa config client commands
Quoted 15 lines · Shannon Nelson
These are the adminq commands that will be needed for setting up and using the vDPA device.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 1 + drivers/vdpa/pds/cmds.c | 266 ++++++++++++++++++++++++++++++++++++ drivers/vdpa/pds/cmds.h | 17 +++ drivers/vdpa/pds/vdpa_dev.h | 60 ++++++++ 4 files changed, 344 insertions(+) create mode 100644 drivers/vdpa/pds/cmds.c create mode 100644 drivers/vdpa/pds/cmds.h create mode 100644 drivers/vdpa/pds/vdpa_dev.h
[...]
drivers/vdpa/pds/vdpa_dev.h line 54
Quoted 7 lines
drivers/vdpa/pds/vdpa_dev.h new file +7
-0+struct pds_vdpa_device { +struct vdpa_device vdpa_dev; +struct pds_vdpa_aux *vdpa_aux; +struct pds_vdpa_hw hw; + +struct virtio_net_config vn_config; +dma_addr_t vn_config_pa;
So this is the dma address not necessarily pa, we'd better drop the "pa" suffix.
Thanks
Quoted 9 lines
drivers/vdpa/pds/vdpa_dev.h new file +6
-1+ struct dentry *dentry; +}; + +int pds_vdpa_get_mgmt_info (struct pds_vdpa_aux *vdpa_aux); + +#endif /* _VDPA_DEV_H_ */ --2.17.1
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 17/19] pds_vdpa: add vdpa config client commands
Quoted 19 lines · Jason Wang
Quoted 15 lines · Shannon Nelson
These are the adminq commands that will be needed for setting up and using the vDPA device.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 1 + drivers/vdpa/pds/cmds.c | 266 ++++++++++++++++++++++++++++++++++++ drivers/vdpa/pds/cmds.h | 17 +++ drivers/vdpa/pds/vdpa_dev.h | 60 ++++++++ 4 files changed, 344 insertions(+)create mode 100644
drivers/vdpa/pds/cmds.c
create mode 100644drivers/vdpa/pds/cmds.h
create mode 100644drivers/vdpa/pds/vdpa_dev.h[...]
Quoted 9 lines
drivers/vdpa/pds/vdpa_dev.h line 54
Quoted 7 lines
drivers/vdpa/pds/vdpa_dev.h new file +7
-0+struct pds_vdpa_device { +struct vdpa_device vdpa_dev; +struct pds_vdpa_aux *vdpa_aux; +struct pds_vdpa_hw hw; + +struct virtio_net_config vn_config; +dma_addr_t vn_config_pa;So this is the dma address not necessarily pa, we'd better drop the "pa" suffix.
Yeah, strictly speaking I suppose it isn't necessarily pa, but _pa is the moniker we've used throughout our drivers for this kind of thing - maybe not perfect, but this is where we are for now.
sln
RFC PATCH 18/19 pds_vdpa: add support for vdpa and vdpamgmt interfaces
In reply to Shannon NelsonThis is the vDPA device support, where we advertise that we can support the virtio queues and deal with the configuration work through the pds_core's adminq.
--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/aux_drv.c | 33 ++ drivers/vdpa/pds/debugfs.c | 167 ++++++++ drivers/vdpa/pds/debugfs.h | 4 + drivers/vdpa/pds/vdpa_dev.c | 796 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1002 insertions(+), 1 deletion(-) create mode 100644 drivers/vdpa/pds/vdpa_dev.c
drivers/vdpa/pds/Makefile +2 -1
@@ -7,4 +7,5 @@ pds_vdpa-y := aux_drv.o \ cmds.o \ pci_drv.o \ debugfs.o \ - virtio_pci.o + virtio_pci.o \ + vdpa_dev.o
drivers/vdpa/pds/aux_drv.c +33 -0
@@ -12,6 +12,7 @@#include <linux/pds/pds_vdpa.h> #include "aux_drv.h" +#include "vdpa_dev.h" #include "pci_drv.h" #include "debugfs.h"
@@ -25,10 +26,25 @@ static void pds_vdpa_aux_notify_handler(struct pds_auxiliary_dev *padev, union pds_core_notifyq_comp *event) { + struct pds_vdpa_device *pdsv = padev->priv;struct device *dev = &padev->aux_dev.dev;u16 ecode =le16_to_cpu (event->ecode);dev_info (dev,"%s: event code %d\n" , __func__, ecode); + +/* Give the upper layers a hint that something interesting +* may have happened. It seems that the only thing this +* triggers in the virtio-net drivers above us is a check +* of link status. +* +* We don't set the NEEDS_RESET flag for EVENT_RESET +* because we're likely going through a recovery or +* fw_update and will be back up and running soon. +*/ +if (ecode ==PDS_EVENT_RESET || ecode ==PDS_EVENT_LINK_CHANGE ) { +if (pdsv->hw.config_cb.callback) + pdsv->hw.config_cb.callback (pdsv->hw.config_cb.private); + } } static int
@@ -80,10 +96,25 @@ pds_vdpa_aux_probe(struct auxiliary_device *aux_dev, goto err_register_client; } +/* Get device ident info and set up the vdpa_mgmt_dev */ + err =pds_vdpa_get_mgmt_info (vdpa_aux); +if (err) + goto err_register_client; + +/* Let vdpa know that we can provide devices */ + err =vdpa_mgmtdev_register (&vdpa_aux->vdpa_mdev); +if (err) { +dev_err (dev,"%s: Failed to initialize vdpa_mgmt interface: %pe\n" , + __func__,ERR_PTR (err)); + goto err_mgmt_reg; + } +pds_vdpa_debugfs_add_ident (vdpa_aux);return 0 ; +err_mgmt_reg: + padev->ops->unregister_client (padev); err_register_client:auxiliary_set_drvdata (aux_dev,NULL ); err_invalid_driver:
@@ -98,6 +129,8 @@ pds_vdpa_aux_remove(struct auxiliary_device *aux_dev)struct pds_vdpa_aux *vdpa_aux =auxiliary_get_drvdata (aux_dev);struct device *dev = &aux_dev->dev; +vdpa_mgmtdev_unregister (&vdpa_aux->vdpa_mdev); + vdpa_aux->padev->ops->unregister_client (vdpa_aux->padev);if (vdpa_aux->vdpa_vf)pci_dev_put (vdpa_aux->vdpa_vf->pdev);
drivers/vdpa/pds/debugfs.c +167 -0
193 lines not on this page · raw message
drivers/vdpa/pds/debugfs.h +4 -0
@@ -13,12 +13,16 @@ void pds_vdpa_debugfs_destroy(void);void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux); +void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv); +void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv);#else static inline void pds_vdpa_debugfs_create (void ) { }static inline void pds_vdpa_debugfs_destroy (void ) { }static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) { } +static inline void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) { } +static inline void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) { }#endif #endif /* _PDS_VDPA_DEBUGFS_H_ */
drivers/vdpa/pds/vdpa_dev.c new file +796 -0
796 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH net-next 18/19] pds_vdpa: add support for vdpa and vdpamgmt interfaces
Quoted 478 lines · Shannon Nelson
This is the vDPA device support, where we advertise that we can support the virtio queues and deal with the configuration work through the pds_core's adminq.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/aux_drv.c | 33 ++ drivers/vdpa/pds/debugfs.c | 167 ++++++++ drivers/vdpa/pds/debugfs.h | 4 + drivers/vdpa/pds/vdpa_dev.c | 796 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1002 insertions(+), 1 deletion(-) create mode 100644 drivers/vdpa/pds/vdpa_dev.cdrivers/vdpa/pds/Makefile +2
-1@@ -7,4 +7,5 @@ pds_vdpa-y := aux_drv.o \ cmds.o \ pci_drv.o \ debugfs.o \ - virtio_pci.o + virtio_pci.o \ + vdpa_dev.odrivers/vdpa/pds/aux_drv.c +33
-0@@ -12,6 +12,7 @@#include <linux/pds/pds_vdpa.h> #include "aux_drv.h" +#include "vdpa_dev.h" #include "pci_drv.h" #include "debugfs.h" @@ -25,10 +26,25 @@ static void pds_vdpa_aux_notify_handler(struct pds_auxiliary_dev *padev, union pds_core_notifyq_comp *event) { + struct pds_vdpa_device *pdsv = padev->priv;struct device *dev = &padev->aux_dev.dev;u16 ecode =le16_to_cpu (event->ecode);dev_info (dev,"%s: event code %d\n" , __func__, ecode); + +/* Give the upper layers a hint that something interesting +* may have happened. It seems that the only thing this +* triggers in the virtio-net drivers above us is a check +* of link status. +* +* We don't set the NEEDS_RESET flag for EVENT_RESET +* because we're likely going through a recovery or +* fw_update and will be back up and running soon. +*/ +if (ecode ==PDS_EVENT_RESET || ecode ==PDS_EVENT_LINK_CHANGE ) { +if (pdsv->hw.config_cb.callback) + pdsv->hw.config_cb.callback (pdsv->hw.config_cb.private); + } } static int@@ -80,10 +96,25 @@ pds_vdpa_aux_probe(struct auxiliary_device *aux_dev, goto err_register_client; } +/* Get device ident info and set up the vdpa_mgmt_dev */ + err =pds_vdpa_get_mgmt_info (vdpa_aux); +if (err) + goto err_register_client; + +/* Let vdpa know that we can provide devices */ + err =vdpa_mgmtdev_register (&vdpa_aux->vdpa_mdev); +if (err) { +dev_err (dev,"%s: Failed to initialize vdpa_mgmt interface: %pe\n" , + __func__,ERR_PTR (err)); + goto err_mgmt_reg; + } +pds_vdpa_debugfs_add_ident (vdpa_aux);return 0 ; +err_mgmt_reg: + padev->ops->unregister_client (padev); err_register_client:auxiliary_set_drvdata (aux_dev,NULL ); err_invalid_driver:@@ -98,6 +129,8 @@ pds_vdpa_aux_remove(struct auxiliary_device *aux_dev)struct pds_vdpa_aux *vdpa_aux =auxiliary_get_drvdata (aux_dev);struct device *dev = &aux_dev->dev; +vdpa_mgmtdev_unregister (&vdpa_aux->vdpa_mdev); + vdpa_aux->padev->ops->unregister_client (vdpa_aux->padev);if (vdpa_aux->vdpa_vf)pci_dev_put (vdpa_aux->vdpa_vf->pdev);drivers/vdpa/pds/debugfs.c +167
-0@@ -11,6 +11,7 @@#include <linux/pds/pds_auxbus.h> #include <linux/pds/pds_vdpa.h> +#include "vdpa_dev.h" #include "aux_drv.h" #include "pci_drv.h" #include "debugfs.h" @@ -19,6 +20,72 @@static struct dentry *dbfs_dir; +#define PRINT_SBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & __name) \ + seq_printf(__seq, " %s", &#__name[16]); \ + } while (0) + +static void +print_status_bits (struct seq_file *seq,u16 status) +{ +seq_puts (seq,"status:" ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_ACKNOWLEDGE ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FEATURES_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_NEEDS_RESET ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FAILED ); +seq_puts (seq,"\n" ); +} + +#define PRINT_FBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & BIT_ULL(__name)) \ + seq_printf(__seq, " %s", #__name); \ + } while (0) + +static void +print_feature_bits (struct seq_file *seq,u64 features) +{ +seq_puts (seq,"features:" ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_GUEST_OFFLOADS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MTU ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MAC ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MRG_RXBUF ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STATUS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VLAN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX_EXTRA ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ANNOUNCE ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_MAC_ADDR ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HASH_REPORT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSC_EXT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STANDBY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_SPEED_DUPLEX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_NOTIFY_ON_EMPTY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ANY_LAYOUT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_VERSION_1 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ACCESS_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_RING_PACKED ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ORDER_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_SR_IOV ); +seq_puts (seq,"\n" ); +} +void pds_vdpa_debugfs_create (void ) {@@ -49,10 +116,18 @@ static int identity_show(struct seq_file *seq, void *v) { struct pds_vdpa_aux *vdpa_aux = seq->private; +struct vdpa_mgmt_dev *mgmt;seq_printf (seq,"aux_dev: %s\n" ,dev_name (&vdpa_aux->padev->aux_dev.dev)); + mgmt = &vdpa_aux->vdpa_mdev; +seq_printf (seq,"max_vqs: %d\n" , mgmt->max_supported_vqs); +seq_printf (seq,"config_attr_mask: %#llx\n" , mgmt->config_attr_mask); +seq_printf (seq,"supported_features: %#llx\n" , mgmt->supported_features); +print_feature_bits (seq, mgmt->supported_features); +seq_printf (seq,"local_mac_bit: %d\n" , vdpa_aux->local_mac_bit); +return 0 ; }DEFINE_SHOW_ATTRIBUTE (identity);@@ -64,4 +139,96 @@ pds_vdpa_debugfs_add_ident(struct pds_vdpa_aux *vdpa_aux) vdpa_aux, &identity_fops); } +static int +config_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_device *pdsv = seq->private; +struct virtio_net_config *vc = &pdsv->vn_config; + +seq_printf (seq,"mac: %pM\n" , vc->mac); +seq_printf (seq,"max_virtqueue_pairs: %d\n" , +__virtio16_to_cpu (true, vc->max_virtqueue_pairs)); +seq_printf (seq,"mtu: %d\n" ,__virtio16_to_cpu (true, vc->mtu)); +seq_printf (seq,"speed: %d\n" ,le32_to_cpu (vc->speed)); +seq_printf (seq,"duplex: %d\n" , vc->duplex); +seq_printf (seq,"rss_max_key_size: %d\n" , vc->rss_max_key_size); +seq_printf (seq,"rss_max_indirection_table_length: %d\n" , +le16_to_cpu (vc->rss_max_indirection_table_length)); +seq_printf (seq,"supported_hash_types: %#x\n" , +le32_to_cpu (vc->supported_hash_types)); +seq_printf (seq,"vn_status: %#x\n" , +__virtio16_to_cpu (true, vc->status)); +print_status_bits (seq,__virtio16_to_cpu (true, vc->status)); + +seq_printf (seq,"hw_status: %#x\n" , pdsv->hw.status); +print_status_bits (seq, pdsv->hw.status); +seq_printf (seq,"req_features: %#llx\n" , pdsv->hw.req_features); +print_feature_bits (seq, pdsv->hw.req_features); +seq_printf (seq,"actual_features: %#llx\n" , pdsv->hw.actual_features); +print_feature_bits (seq, pdsv->hw.actual_features); +seq_printf (seq,"vdpa_index: %d\n" , pdsv->hw.vdpa_index); +seq_printf (seq,"num_vqs: %d\n" , pdsv->hw.num_vqs); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (config); + +static int +vq_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_vq_info *vq = seq->private; +struct pds_vdpa_intr_info *intrs; + +seq_printf (seq,"ready: %d\n" , vq->ready); +seq_printf (seq,"desc_addr: %#llx\n" , vq->desc_addr); +seq_printf (seq,"avail_addr: %#llx\n" , vq->avail_addr); +seq_printf (seq,"used_addr: %#llx\n" , vq->used_addr); +seq_printf (seq,"q_len: %d\n" , vq->q_len); +seq_printf (seq,"qid: %d\n" , vq->qid); + +seq_printf (seq,"doorbell: %#llx\n" , vq->doorbell); +seq_printf (seq,"avail_idx: %d\n" , vq->avail_idx); +seq_printf (seq,"used_idx: %d\n" , vq->used_idx); +seq_printf (seq,"intr_index: %d\n" , vq->intr_index); + + intrs = vq->pdsv->vdpa_aux->vdpa_vf->intrs; +seq_printf (seq,"irq: %d\n" , intrs[vq->intr_index].irq); +seq_printf (seq,"irq-name: %s\n" , intrs[vq->intr_index].name); + +seq_printf (seq,"hw_qtype: %d\n" , vq->hw_qtype); +seq_printf (seq,"hw_qindex: %d\n" , vq->hw_qindex); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (vq); + +void +pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) +{ +struct dentry *dentry; +const char *name; +int i; + + dentry = pdsv->vdpa_aux->vdpa_vf->dentry; + name =dev_name (&pdsv->vdpa_dev.dev); + + pdsv->dentry =debugfs_create_dir (name, dentry); + +debugfs_create_file ("config" ,0400 , pdsv->dentry, pdsv, &config_fops); + +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +char name[8 ]; + +snprintf (name,sizeof (name),"vq%02d" , i); +debugfs_create_file (name,0400 , pdsv->dentry, &pdsv->hw.vqs[i], &vq_fops); + } +} + +void +pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) +{ +debugfs_remove_recursive (pdsv->dentry); + pdsv->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */ drivers/vdpa/pds/debugfs.h +4
-0@@ -13,12 +13,16 @@ void pds_vdpa_debugfs_destroy(void);void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux); +void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv); +void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv);#else static inline void pds_vdpa_debugfs_create (void ) { }static inline void pds_vdpa_debugfs_destroy (void ) { }static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) { } +static inline void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) { } +static inline void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) { }#endif #endif /* _PDS_VDPA_DEBUGFS_H_ */ drivers/vdpa/pds/vdpa_dev.c new file +151
-0@@ -0,0 +1,796 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/sysfs.h> +#include <linux/types.h> +#include <linux/vdpa.h> +#include <uapi/linux/virtio_pci.h> +#include <uapi/linux/vdpa.h> + +#include <linux/pds/pds_intr.h> +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> +#include <linux/pds/pds_vdpa.h> + +#include "vdpa_dev.h" +#include "pci_drv.h" +#include "aux_drv.h" +#include "pci_drv.h" +#include "cmds.h" +#include "debugfs.h" + +static int +pds_vdpa_setup_driver (struct pds_vdpa_device *pdsv) +{ +struct device *dev = &pdsv->vdpa_dev.dev; +int err =0 ; +int i; + +/* Verify all vqs[] are in ready state */ +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +if (!pdsv->hw.vqs[i].ready) { +dev_warn (dev,"%s: qid %d not ready\n" , __func__, i); + err = -ENOENT ; + } + } + +return err; +} + +static struct pds_vdpa_device * +vdpa_to_pdsv (struct vdpa_device *vdpa_dev) +{ + return container_of(vdpa_dev, struct pds_vdpa_device, vdpa_dev); +} + +static struct pds_vdpa_hw * +vdpa_to_hw (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return &pdsv->hw; +} + +static int +pds_vdpa_set_vq_address (struct vdpa_device *vdpa_dev,u16 qid, +u64 desc_addr,u64 driver_addr,u64 device_addr) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].desc_addr = desc_addr; + hw->vqs[qid].avail_addr = driver_addr; + hw->vqs[qid].used_addr = device_addr; + +return 0 ; +} + +static void +pds_vdpa_set_vq_num (struct vdpa_device *vdpa_dev,u16 qid,u32 num) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].q_len = num; +} + +static void +pds_vdpa_kick_vq (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +iowrite16 (qid, pdsv->hw.vqs[qid].notify); +} + +static void +pds_vdpa_set_vq_cb (struct vdpa_device *vdpa_dev,u16 qid, +struct vdpa_callback *cb) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].event_cb = *cb; +} + +static irqreturn_t +pds_vdpa_isr (int irq,void *data) +{ +struct pds_core_intr __iomem *intr_ctrl; +struct pds_vdpa_device *pdsv; +struct pds_vdpa_vq_info *vq; + + vq = data; + pdsv = vq->pdsv; + +if (vq->event_cb.callback) + vq->event_cb.callback (vq->event_cb.private); + +/* Since we don't actually know how many vq descriptors are +* covered in this interrupt cycle, we simply clean all the +* credits and re-enable the interrupt. +*/ + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; +pds_core_intr_clean_flags (&intr_ctrl[vq->intr_index], +PDS_CORE_INTR_CRED_REARM ); + +return IRQ_HANDLED ; +} + +static void +pds_vdpa_release_irq (struct pds_vdpa_device *pdsv,int qid) +{ +struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_core_intr __iomem *intr_ctrl; +int index; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; + index = pdsv->hw.vqs[qid].intr_index; +if (index ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (intrs[index].irq ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (qid &0x1 ) { + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + }else { +pds_core_intr_mask (&intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); +devm_free_irq (&pdev->dev, intrs[index].irq, &pdsv->hw.vqs[qid]); + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + intrs[index].irq =VIRTIO_MSI_NO_VECTOR ; + } +} + +static void +pds_vdpa_set_vq_ready (struct vdpa_device *vdpa_dev,u16 qid,bool ready) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev);
drivers/vdpa/pds/vdpa_dev.c line 161
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+struct device *dev = &pdsv->vdpa_dev.dev; +struct pds_core_intr __iomem *intr_ctrl; +int err; + +dev_dbg (dev,"%s: qid %d ready %d => %d\n" , + __func__, qid, hw->vqs[qid].ready, ready); +if (ready == hw->vqs[qid].ready) +return ; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr;
It looks to me pds has a different layout/semantic for isr than virtio spec. I'd suggest to not reuse spec isr here to avoid confusion.
drivers/vdpa/pds/vdpa_dev.c line 162
Quoted 1 line
drivers/vdpa/pds/vdpa_dev.c new file +1
-0+if (ready) {
Spec said no interrupt before DRIVER_OK, it looks more simple if we move the interrupt setup to set_status():
E.g we can know if we have sufficient vectors and use different mapping policies in advance.
Quoted 25 lines
drivers/vdpa/pds/vdpa_dev.c new file +25
-0+struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +int index =VIRTIO_MSI_NO_VECTOR ; +int i; + +/* Tx and Rx queues share interrupts, and they start with +* even numbers, so only find an interrupt for the even numbered +* qid, and let the odd number use what the previous queue got. +*/ +if (qid &0x1 ) { +int even = qid & ~0x1 ; + + index = hw->vqs[even].intr_index; + }else { +for (i =0 ; i < pdsv->vdpa_aux->vdpa_vf->nintrs; i++) { +if (intrs[i].irq ==VIRTIO_MSI_NO_VECTOR ) { + index = i; +break ; + } + } + } + +if (qid &0x1 ) { + hw->vqs[qid].intr_index = index; + }else if (index !=VIRTIO_MSI_NO_VECTOR ) { +int irq;
drivers/vdpa/pds/vdpa_dev.c line 197
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ + irq =pci_irq_vector (pdev, index); +snprintf (intrs[index].name,sizeof (intrs[index].name), +"vdpa-%s-%d" ,dev_name (dev), index); + + err =devm_request_irq (&pdev->dev, irq, pds_vdpa_isr,0 , + intrs[index].name, &hw->vqs[qid]); +if (err) { +dev_info (dev,"%s: no irq for qid %d: %pe\n" , + __func__, qid,ERR_PTR (err));
Should we fail?
drivers/vdpa/pds/vdpa_dev.c line 202
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+ }else { + intrs[index].irq = irq; + hw->vqs[qid].intr_index = index; +pds_core_intr_mask (&intr_ctrl[index], +PDS_CORE_INTR_MASK_CLEAR );
I guess the reason that you don't simply use VF MSI-X is the DPU can support vDPA subdevice in the future?
drivers/vdpa/pds/vdpa_dev.c line 206
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ } + }else { +dev_info (dev,"%s: no intr slot for qid %d\n" , + __func__, qid);
Do we need to fail here?
Quoted 33 lines
drivers/vdpa/pds/vdpa_dev.c new file +33
-0+ } + +/* Pass vq setup info to DSC */ + err =pds_vdpa_cmd_init_vq (pdsv, qid, &hw->vqs[qid]); +if (err) { +pds_vdpa_release_irq (pdsv, qid); + ready = false; + } + }else { +pds_vdpa_release_irq (pdsv, qid); + (void )pds_vdpa_cmd_reset_vq (pdsv, qid); + } + + hw->vqs[qid].ready = ready; +} + +static bool +pds_vdpa_get_vq_ready (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->vqs[qid].ready; +} + +static int +pds_vdpa_set_vq_state (struct vdpa_device *vdpa_dev,u16 qid, +const struct vdpa_vq_state *state) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].used_idx = state->split.avail_index; + hw->vqs[qid].avail_idx = state->split.avail_index; +
drivers/vdpa/pds/vdpa_dev.c line 249
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+return 0 ; +} + +static int +pds_vdpa_get_vq_state (struct vdpa_device *vdpa_dev,u16 qid, +struct vdpa_vq_state *state) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + state->split.avail_index = hw->vqs[qid].avail_idx;
Who is in charge of reading avail_idx from the hardware?
Quoted 24 lines
drivers/vdpa/pds/vdpa_dev.c new file +24
-0+ +return 0 ; +} + +static struct vdpa_notification_area +pds_vdpa_get_vq_notification (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct virtio_pci_modern_device *vd_mdev; +struct vdpa_notification_area area; + + area.addr = hw->vqs[qid].notify_pa; + + vd_mdev = &pdsv->vdpa_aux->vdpa_vf->vd_mdev; +if (!vd_mdev->notify_offset_multiplier) + area.size =PAGE_SIZE ; +else + area.size = vd_mdev->notify_offset_multiplier; + +return area; +} + +static int
drivers/vdpa/pds/vdpa_dev.c line 283
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+pds_vdpa_get_vq_irq(struct vdpa_device *vdpa_dev, u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +int irq =VIRTIO_MSI_NO_VECTOR ; +int index; + +if (pdsv->vdpa_aux->vdpa_vf->intrs) { + index = hw->vqs[qid].intr_index; + irq = pdsv->vdpa_aux->vdpa_vf->intrs[index].irq;
The notification area mapping might only work well when each vq has it's own irq. Otherwise guest may see spurious interrupt which may degrade the performance.
Quoted 30 lines
drivers/vdpa/pds/vdpa_dev.c new file +30
-0+ } + +return irq; +} + +static u32 +pds_vdpa_get_vq_align (struct vdpa_device *vdpa_dev) +{ + +return PAGE_SIZE ; +} + +static u32 +pds_vdpa_get_vq_group (struct vdpa_device *vdpa_dev,u16 idx) +{ +return 0 ; +} + +static u64 +pds_vdpa_get_device_features (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return le64_to_cpu (pdsv->vdpa_aux->ident.hw_features); +} + +static int +pds_vdpa_set_driver_features (struct vdpa_device *vdpa_dev,u64 features) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev);
drivers/vdpa/pds/vdpa_dev.c line 323
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +u64 nego_features; +u64 set_features; +u64 missing; +int err; + +if (!(features &BIT_ULL (VIRTIO_F_ACCESS_PLATFORM )) && features) { +dev_err (dev,"VIRTIO_F_ACCESS_PLATFORM is not negotiated\n" ); +return -EOPNOTSUPP ;
Should we fail the FEATURE_OK in this case and all the other below error conditions?
Quoted 12 lines
drivers/vdpa/pds/vdpa_dev.c new file +12
-0+ } + + hw->req_features = features; + +/* Check for valid feature bits */ + nego_features = features &le64_to_cpu (pdsv->vdpa_aux->ident.hw_features); + missing = hw->req_features & ~nego_features; +if (missing) { +dev_err (dev,"Can't support all requested features in %#llx, missing %#llx features\n" , + hw->req_features, missing); +return -EOPNOTSUPP ; + }
drivers/vdpa/pds/vdpa_dev.c line 345
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ +dev_dbg (dev,"%s: %#llx => %#llx\n" , + __func__, hw->actual_features, nego_features); + +if (hw->actual_features == nego_features) +return 0 ; + +/* Update hw feature configuration, strip MAC bit if locally set */ +if (pdsv->vdpa_aux->local_mac_bit) + set_features = nego_features & ~BIT_ULL (VIRTIO_NET_F_MAC );
Need some document to explain how local_mac_bit work.
Quoted 24 lines
drivers/vdpa/pds/vdpa_dev.c new file +24
-0+else + set_features = nego_features; + err =pds_vdpa_cmd_set_features (pdsv, set_features); +if (!err) + hw->actual_features = nego_features; + +return err; +} + +static u64 +pds_vdpa_get_driver_features (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->actual_features; +} + +static void +pds_vdpa_set_config_cb (struct vdpa_device *vdpa_dev,struct vdpa_callback *cb) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->config_cb.callback = cb->callback; + hw->config_cb.private = cb->private;
drivers/vdpa/pds/vdpa_dev.c line 379
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+} + +static u16 +pds_vdpa_get_vq_num_max (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +u32 max_qlen; + + max_qlen =min_t (u32,PDS_VDPA_MAX_QLEN , +1 <<le16_to_cpu (pdsv->vdpa_aux->ident.max_qlen));
Assuming we can fetch the max_qlen from the device, any reason have another layer like PDS_VDPA_MAX_QLEN?
Quoted 12 lines
drivers/vdpa/pds/vdpa_dev.c new file +12
-0+ +return (u16 )max_qlen; +} + +static u32 +pds_vdpa_get_device_id (struct vdpa_device *vdpa_dev) +{ +return VIRTIO_ID_NET ; +} + +static u32 +pds_vdpa_get_vendor_id (struct vdpa_device *vdpa_dev)
drivers/vdpa/pds/vdpa_dev.c line 401
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+{ +return PCI_VENDOR_ID_PENSANDO ; +} + +static u8 +pds_vdpa_get_status (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->status;
How is this synchronized with the device or it is fully emulated by this driver?
Quoted 37 lines
drivers/vdpa/pds/vdpa_dev.c new file +37
-0+} + +static void +pds_vdpa_set_status (struct vdpa_device *vdpa_dev,u8 status) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +int err; + +if (hw->status == status) +return ; + +/* If the DRIVER_OK bit turns on, time to start the queues */ +if ((status ^ hw->status) &VIRTIO_CONFIG_S_DRIVER_OK ) { +if (status &VIRTIO_CONFIG_S_DRIVER_OK ) { + err =pds_vdpa_setup_driver (pdsv); +if (err) { +dev_err (dev,"failed to setup driver: %pe\n" ,ERR_PTR (err)); + status = hw->status |VIRTIO_CONFIG_S_FAILED ; + } + }else { +dev_warn (dev,"did not expect DRIVER_OK to be cleared\n" ); + } + } + + err =pds_vdpa_cmd_set_status (pdsv, status); +if (!err) + hw->status = status; +} + +static int +pds_vdpa_reset (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +int i;
drivers/vdpa/pds/vdpa_dev.c line 448
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ +if (hw->status ==0 ) +return 0 ; + +if (hw->status &VIRTIO_CONFIG_S_DRIVER_OK ) { + +/* Reset the vqs */ +for (i =0 ; i < hw->num_vqs; i++) { +pds_vdpa_release_irq (pdsv, i); + (void )pds_vdpa_cmd_reset_vq (pdsv, i);
(void) is unnecessary.
Quoted 35 lines
drivers/vdpa/pds/vdpa_dev.c new file +35
-0+ +memset (&pdsv->hw.vqs[i],0 ,sizeof (pdsv->hw.vqs[0 ])); + pdsv->hw.vqs[i].ready = false; + } + } + + hw->status =0 ; + (void )pds_vdpa_cmd_set_status (pdsv,0 ); + +return 0 ; +} + +static size_t +pds_vdpa_get_config_size (struct vdpa_device *vdpa_dev) +{ +return sizeof (struct virtio_net_config ); +} + +static void +pds_vdpa_get_config (struct vdpa_device *vdpa_dev, +unsigned int offset, +void *buf,unsigned int len) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +if (offset + len <=sizeof (struct virtio_net_config )) +memcpy (buf, (u8 *)&pdsv->vn_config + offset, len); +} + +static void +pds_vdpa_set_config (struct vdpa_device *vdpa_dev, +unsigned int offset,const void *buf, +unsigned int len) +{ +/* In the virtio_net context, this callback seems to only be
drivers/vdpa/pds/vdpa_dev.c line 493
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+* called in drivers supporting the older non-VERSION_1 API, +* so we can leave this an empty function, but we need to +* define the function in case it does get called, as there +* are currently no checks for existence before calling in +* that path. +* +* The implementation would be something like: +* if (offset + len <= sizeof(struct virtio_net_config)) +* memcpy((u8 *)&pdsv->vn_config + offset, buf, len); +*/
And we need to notify the hardware that config has been changed.
Quoted 77 lines
drivers/vdpa/pds/vdpa_dev.c new file +77
-0+} + +static const struct vdpa_config_ops pds_vdpa_ops = { + .set_vq_address = pds_vdpa_set_vq_address, + .set_vq_num = pds_vdpa_set_vq_num, + .kick_vq = pds_vdpa_kick_vq, + .set_vq_cb = pds_vdpa_set_vq_cb, + .set_vq_ready = pds_vdpa_set_vq_ready, + .get_vq_ready = pds_vdpa_get_vq_ready, + .set_vq_state = pds_vdpa_set_vq_state, + .get_vq_state = pds_vdpa_get_vq_state, + .get_vq_notification = pds_vdpa_get_vq_notification, + .get_vq_irq = pds_vdpa_get_vq_irq, + .get_vq_align = pds_vdpa_get_vq_align, + .get_vq_group = pds_vdpa_get_vq_group, + + .get_device_features = pds_vdpa_get_device_features, + .set_driver_features = pds_vdpa_set_driver_features, + .get_driver_features = pds_vdpa_get_driver_features, + .set_config_cb = pds_vdpa_set_config_cb, + .get_vq_num_max = pds_vdpa_get_vq_num_max, +/* .get_vq_num_min (optional) */ + .get_device_id = pds_vdpa_get_device_id, + .get_vendor_id = pds_vdpa_get_vendor_id, + .get_status = pds_vdpa_get_status, + .set_status = pds_vdpa_set_status, + .reset = pds_vdpa_reset, + .get_config_size = pds_vdpa_get_config_size, + .get_config = pds_vdpa_get_config, + .set_config = pds_vdpa_set_config, + +/* .get_generation (optional) */ +/* .get_iova_range (optional) */ +/* .set_group_asid */ +/* .set_map (optional) */ +/* .dma_map (optional) */ +/* .dma_unmap (optional) */ +/* .free (optional) */ +}; +static struct virtio_device_id pds_vdpa_id_table[] = { + {VIRTIO_ID_NET ,VIRTIO_DEV_ANY_ID }, + {0 }, +}; + +static int +pds_vdpa_dev_add (struct vdpa_mgmt_dev *mdev,const char *name, +const struct vdpa_dev_set_config *add_config) +{ +struct pds_vdpa_aux *vdpa_aux; +struct pds_vdpa_device *pdsv; +struct vdpa_mgmt_dev *mgmt; +u16 fw_max_vqs, vq_pairs; +struct device *dma_dev; +struct pds_vdpa_hw *hw; +struct pci_dev *pdev; +struct device *dev; +u8 mac[ETH_ALEN ]; +int err; +int i; + + vdpa_aux = container_of(mdev, struct pds_vdpa_aux, vdpa_mdev); + dev = &vdpa_aux->padev->aux_dev.dev; + mgmt = &vdpa_aux->vdpa_mdev; + +if (vdpa_aux->pdsv) { +dev_warn (dev,"Multiple vDPA devices on a VF is not supported.\n" ); +return -EOPNOTSUPP ; + } + + pdsv = vdpa_alloc_device(struct pds_vdpa_device, vdpa_dev, + dev, &pds_vdpa_ops,1 ,1 , name, false); +if (IS_ERR (pdsv)) { +dev_err (dev,"Failed to allocate vDPA structure: %pe\n" , pdsv); +return PTR_ERR (pdsv); + } + + vdpa_aux->pdsv = pdsv;
drivers/vdpa/pds/vdpa_dev.c line 580
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ pdsv->vdpa_aux = vdpa_aux; + pdsv->vdpa_aux->padev->priv = pdsv; + + pdev = vdpa_aux->vdpa_vf->pdev; + pdsv->vdpa_dev.dma_dev = &pdev->dev; + dma_dev = pdsv->vdpa_dev.dma_dev; + hw = &pdsv->hw; + + pdsv->vn_config_pa =dma_map_single (dma_dev, &pdsv->vn_config, +sizeof (pdsv->vn_config),DMA_FROM_DEVICE );
I think we should use coherent mapping instead of streaming mapping otherwise we may end up with coherency issues when accessing the device configuration space.
Quoted 27 lines
drivers/vdpa/pds/vdpa_dev.c new file +27
-0+if (dma_mapping_error (dma_dev, pdsv->vn_config_pa)) { +dev_err (dma_dev,"Failed to map vn_config space\n" ); + pdsv->vn_config_pa =0 ; + err = -ENOMEM ; + goto err_out; + } + + err =pds_vdpa_init_hw (pdsv); +if (err) { +dev_err (dev,"Failed to init hw: %pe\n" ,ERR_PTR (err)); + goto err_unmap; + } + + fw_max_vqs =le16_to_cpu (pdsv->vdpa_aux->ident.max_vqs); + vq_pairs = fw_max_vqs /2 ; + +/* Make sure we have the queues being requested */ +if (add_config->mask & (1 <<VDPA_ATTR_DEV_NET_CFG_MAX_VQP )) + vq_pairs = add_config->net.max_vq_pairs; + + hw->num_vqs =2 * vq_pairs; +if (mgmt->supported_features &BIT_ULL (VIRTIO_NET_F_CTRL_VQ )) + hw->num_vqs++; + +if (hw->num_vqs > fw_max_vqs) { +dev_err (dev,"%s: queue count requested %u greater than max %u\n" , + __func__, hw->num_vqs, fw_max_vqs);
drivers/vdpa/pds/vdpa_dev.c line 617
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ err = -ENOSPC ; + goto err_unmap; + } + +if (hw->num_vqs != fw_max_vqs) { + err =pds_vdpa_cmd_set_max_vq_pairs (pdsv, vq_pairs); +if (err == -ERANGE ) { + hw->num_vqs = fw_max_vqs; +dev_warn (dev,"Known FW issue - overriding to use max_vq_pairs %d\n" , + hw->num_vqs /2 );
Should we fail here? Since the device has a different max_vqp that expected.
Quoted 12 lines
drivers/vdpa/pds/vdpa_dev.c new file +12
-0+ }else if (err) { +dev_err (dev,"Failed to update max_vq_pairs: %pe\n" , +ERR_PTR (err)); + goto err_unmap; + } + } + +/* Set a mac, either from the user config if provided +* or set a random mac if default is 00:..:00 +*/ +if (add_config->mask & (1 <<VDPA_ATTR_DEV_NET_CFG_MACADDR )) { +ether_addr_copy (mac, add_config->net.mac);
drivers/vdpa/pds/vdpa_dev.c line 639
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+pds_vdpa_cmd_set_mac (pdsv, mac); + }else if (is_zero_ether_addr (pdsv->vn_config.mac)) { +eth_random_addr (mac); +pds_vdpa_cmd_set_mac (pdsv, mac); + } + +for (i =0 ; i < hw->num_vqs; i++) { + hw->vqs[i].qid = i; + hw->vqs[i].pdsv = pdsv; + hw->vqs[i].intr_index =VIRTIO_MSI_NO_VECTOR ;
Let's rename this as msix_vector to be aligned with the virtio spec.
Quoted 9 lines
drivers/vdpa/pds/vdpa_dev.c new file +9
-0+ hw->vqs[i].notify =vp_modern_map_vq_notify (&pdsv->vdpa_aux->vdpa_vf->vd_mdev, + i, &hw->vqs[i].notify_pa); + } + + pdsv->vdpa_dev.mdev = &vdpa_aux->vdpa_mdev; + +/* We use the _vdpa_register_device() call rather than the +* vdpa_register_device() to avoid a deadlock because this +* dev_add() is called with the vdpa_dev_lock already set
drivers/vdpa/pds/vdpa_dev.c line 658
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+* by vdpa_nl_cmd_dev_add_set_doit() +*/ + err =_vdpa_register_device (&pdsv->vdpa_dev, hw->num_vqs); +if (err) { +dev_err (dev,"Failed to register to vDPA bus: %pe\n" ,ERR_PTR (err)); + goto err_unmap; + } + +pds_vdpa_debugfs_add_vdpadev (pdsv); +dev_info (&pdsv->vdpa_dev.dev,"Added with mac %pM\n" , pdsv->vn_config.mac);
dev_dbg?
Quoted 54 lines
drivers/vdpa/pds/vdpa_dev.c new file +54
-0+ +return 0 ; + +err_unmap: +dma_unmap_single (dma_dev, pdsv->vn_config_pa, +sizeof (pdsv->vn_config),DMA_FROM_DEVICE ); +err_out: +put_device (&pdsv->vdpa_dev.dev); + vdpa_aux->pdsv =NULL ; +return err; +} + +static void +pds_vdpa_dev_del (struct vdpa_mgmt_dev *mdev,struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_aux *vdpa_aux; + +dev_info (&vdpa_dev->dev,"Removed\n" ); + + vdpa_aux = container_of(mdev, struct pds_vdpa_aux, vdpa_mdev); +_vdpa_unregister_device (vdpa_dev); +pds_vdpa_debugfs_del_vdpadev (pdsv); + +if (vdpa_aux->pdsv->vn_config_pa) +dma_unmap_single (vdpa_dev->dma_dev, vdpa_aux->pdsv->vn_config_pa, +sizeof (vdpa_aux->pdsv->vn_config),DMA_FROM_DEVICE ); + + vdpa_aux->pdsv =NULL ; +} + +static const struct vdpa_mgmtdev_ops pds_vdpa_mgmt_dev_ops = { + .dev_add = pds_vdpa_dev_add, + .dev_del = pds_vdpa_dev_del +}; + +int +pds_vdpa_get_mgmt_info (struct pds_vdpa_aux *vdpa_aux) +{ +struct pds_vdpa_pci_device *vdpa_pdev; +struct pds_vdpa_ident_cmd ident_cmd = { + .opcode =PDS_VDPA_CMD_IDENT , + .vf_id =cpu_to_le16 (vdpa_aux->vdpa_vf->vf_id), + }; +struct pds_vdpa_comp ident_comp = {0 }; +struct vdpa_mgmt_dev *mgmt; +struct device *dma_dev; +dma_addr_t ident_pa; +struct pci_dev *pdev; +struct device *dev; +__le64 mac_bit; +u16 max_vqs; +int err; +int i;
drivers/vdpa/pds/vdpa_dev.c line 722
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ + vdpa_pdev = vdpa_aux->vdpa_vf; + pdev = vdpa_pdev->pdev; + dev = &vdpa_aux->padev->aux_dev.dev; + mgmt = &vdpa_aux->vdpa_mdev; + +/* Get resource info from the device */ + dma_dev = &pdev->dev; + ident_pa =dma_map_single (dma_dev, &vdpa_aux->ident, +sizeof (vdpa_aux->ident),DMA_FROM_DEVICE );
I wonder how this work. The ident_pa is mapped through VF, but the command is sent to PF adminq if I understand correctly. If yes, this might work but looks tricky. We'd better explain this is safe since vDPA is not yet created so no userspace can use that. Or I wonder if we can just piggyback the ident via the adminq response so we don't need to worry the security implications.
Thanks
Quoted 75 lines
drivers/vdpa/pds/vdpa_dev.c new file +74
-1+if (dma_mapping_error (dma_dev, ident_pa)) { +dev_err (dma_dev,"Failed to map ident space\n" ); +return -ENOMEM ; + } + + ident_cmd.ident_pa =cpu_to_le64 (ident_pa); + ident_cmd.len =cpu_to_le32 (sizeof (vdpa_aux->ident)); + err = vdpa_aux->padev->ops->adminq_cmd (vdpa_aux->padev, + (union pds_core_adminq_cmd *)&ident_cmd, +sizeof (ident_cmd), + (union pds_core_adminq_comp *)&ident_comp, +0 ); +dma_unmap_single (dma_dev, ident_pa, +sizeof (vdpa_aux->ident),DMA_FROM_DEVICE ); +if (err) { +dev_err (dev,"Failed to ident hw, status %d: %pe\n" , + ident_comp.status,ERR_PTR (err)); +return err; + } + +/* The driver adds a default mac address if the device doesn't, +* so we need to sure we advertise VIRTIO_NET_F_MAC +*/ + mac_bit =cpu_to_le64 (BIT_ULL (VIRTIO_NET_F_MAC )); +if (!(vdpa_aux->ident.hw_features & mac_bit)) { + vdpa_aux->ident.hw_features |= mac_bit; + vdpa_aux->local_mac_bit = true; + } + + max_vqs =le16_to_cpu (vdpa_aux->ident.max_vqs); + mgmt->max_supported_vqs =min_t (u16,PDS_VDPA_MAX_QUEUES , max_vqs); +if (max_vqs >PDS_VDPA_MAX_QUEUES ) +dev_info (dev,"FYI - Device supports more vqs (%d) than driver (%d)\n" , + max_vqs,PDS_VDPA_MAX_QUEUES ); + + mgmt->ops = &pds_vdpa_mgmt_dev_ops; + mgmt->id_table = pds_vdpa_id_table; + mgmt->device = dev; + mgmt->supported_features =le64_to_cpu (vdpa_aux->ident.hw_features); + mgmt->config_attr_mask =BIT_ULL (VDPA_ATTR_DEV_NET_CFG_MACADDR ); + mgmt->config_attr_mask |=BIT_ULL (VDPA_ATTR_DEV_NET_CFG_MAX_VQP ); + +/* Set up interrupts now that we know how many we might want +* TX and RX pairs will share interrupts, so halve the vq count +* Add another for a control queue if supported +*/ + vdpa_pdev->nintrs = mgmt->max_supported_vqs /2 ; +if (mgmt->supported_features &BIT_ULL (VIRTIO_NET_F_CTRL_VQ )) + vdpa_pdev->nintrs++; + + err =pci_alloc_irq_vectors (pdev, vdpa_pdev->nintrs, vdpa_pdev->nintrs, +PCI_IRQ_MSIX ); +if (err <0 ) { +dev_err (dma_dev,"Couldn't get %d msix vectors: %pe\n" , + vdpa_pdev->nintrs,ERR_PTR (err)); +return err; + } + vdpa_pdev->nintrs = err; + err =0 ; + + vdpa_pdev->intrs =devm_kcalloc (&pdev->dev, vdpa_pdev->nintrs, +sizeof (*vdpa_pdev->intrs), +GFP_KERNEL ); +if (!vdpa_pdev->intrs) { + vdpa_pdev->nintrs =0 ; +pci_free_irq_vectors (pdev); +return -ENOMEM ; + } + +for (i =0 ; i < vdpa_pdev->nintrs; i++) + vdpa_pdev->intrs[i].irq =VIRTIO_MSI_NO_VECTOR ; + +return 0 ; +} --
Quoted 2 lines
2.17.1
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 18/19] pds_vdpa: add support for vdpa and vdpamgmt interfaces
Quoted 484 lines · Jason Wang
Quoted 483 lines · Shannon Nelson
This is the vDPA device support, where we advertise that we can support the virtio queues and deal with the configuration work through the pds_core's adminq.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/aux_drv.c | 33 ++ drivers/vdpa/pds/debugfs.c | 167 ++++++++ drivers/vdpa/pds/debugfs.h | 4 + drivers/vdpa/pds/vdpa_dev.c | 796 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1002 insertions(+), 1 deletion(-)create mode 100644
drivers/vdpa/pds/vdpa_dev.cdrivers/vdpa/pds/Makefile +2
-1@@ -7,4 +7,5 @@ pds_vdpa-y := aux_drv.o \ cmds.o \ pci_drv.o \ debugfs.o \ - virtio_pci.o + virtio_pci.o \ + vdpa_dev.odrivers/vdpa/pds/aux_drv.c +33
-0@@ -12,6 +12,7 @@#include <linux/pds/pds_vdpa.h> #include "aux_drv.h" +#include "vdpa_dev.h" #include "pci_drv.h" #include "debugfs.h" @@ -25,10 +26,25 @@ static void pds_vdpa_aux_notify_handler(struct pds_auxiliary_dev *padev, union pds_core_notifyq_comp *event) { + struct pds_vdpa_device *pdsv = padev->priv;struct device *dev = &padev->aux_dev.dev;u16 ecode =le16_to_cpu (event->ecode);dev_info (dev,"%s: event code %d\n" , __func__, ecode); + +/* Give the upper layers a hint that something interesting +* may have happened. It seems that the only thing this +* triggers in the virtio-net drivers above us is a check +* of link status. +* +* We don't set the NEEDS_RESET flag for EVENT_RESET +* because we're likely going through a recovery or +* fw_update and will be back up and running soon. +*/ +if (ecode ==PDS_EVENT_RESET || ecode ==PDS_EVENT_LINK_CHANGE ) { +if (pdsv->hw.config_cb.callback) + pdsv->hw.config_cb.callback (pdsv->hw.config_cb.private); + } } static int@@ -80,10 +96,25 @@ pds_vdpa_aux_probe(struct auxiliary_device *aux_dev, goto err_register_client; } +/* Get device ident info and set up the vdpa_mgmt_dev */ + err =pds_vdpa_get_mgmt_info (vdpa_aux); +if (err) + goto err_register_client; + +/* Let vdpa know that we can provide devices */ + err =vdpa_mgmtdev_register (&vdpa_aux->vdpa_mdev); +if (err) { +dev_err (dev,"%s: Failed to initialize vdpa_mgmt interface: %pe\n" , + __func__,ERR_PTR (err)); + goto err_mgmt_reg; + } +pds_vdpa_debugfs_add_ident (vdpa_aux);return 0 ; +err_mgmt_reg: + padev->ops->unregister_client (padev); err_register_client:auxiliary_set_drvdata (aux_dev,NULL ); err_invalid_driver:@@ -98,6 +129,8 @@ pds_vdpa_aux_remove(struct auxiliary_device *aux_dev)struct pds_vdpa_aux *vdpa_aux =auxiliary_get_drvdata (aux_dev);struct device *dev = &aux_dev->dev; +vdpa_mgmtdev_unregister (&vdpa_aux->vdpa_mdev); + vdpa_aux->padev->ops->unregister_client (vdpa_aux->padev);if (vdpa_aux->vdpa_vf)pci_dev_put (vdpa_aux->vdpa_vf->pdev);drivers/vdpa/pds/debugfs.c +167
-0@@ -11,6 +11,7 @@#include <linux/pds/pds_auxbus.h> #include <linux/pds/pds_vdpa.h> +#include "vdpa_dev.h" #include "aux_drv.h" #include "pci_drv.h" #include "debugfs.h" @@ -19,6 +20,72 @@static struct dentry *dbfs_dir; +#define PRINT_SBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & __name) \ + seq_printf(__seq, " %s", &#__name[16]); \ + } while (0) + +static void +print_status_bits (struct seq_file *seq,u16 status) +{ +seq_puts (seq,"status:" ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_ACKNOWLEDGE ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FEATURES_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_NEEDS_RESET ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FAILED ); +seq_puts (seq,"\n" ); +} + +#define PRINT_FBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & BIT_ULL(__name)) \ + seq_printf(__seq, " %s", #__name); \ + } while (0) + +static void +print_feature_bits (struct seq_file *seq,u64 features) +{ +seq_puts (seq,"features:" ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_GUEST_OFFLOADS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MTU ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MAC ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MRG_RXBUF ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STATUS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VLAN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX_EXTRA ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ANNOUNCE ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_MAC_ADDR ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HASH_REPORT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSC_EXT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STANDBY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_SPEED_DUPLEX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_NOTIFY_ON_EMPTY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ANY_LAYOUT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_VERSION_1 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ACCESS_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_RING_PACKED ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ORDER_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_SR_IOV ); +seq_puts (seq,"\n" ); +} +void pds_vdpa_debugfs_create (void ) {@@ -49,10 +116,18 @@ static int identity_show(struct seq_file *seq, void *v) { struct pds_vdpa_aux *vdpa_aux = seq->private; +struct vdpa_mgmt_dev *mgmt;seq_printf (seq,"aux_dev: %s\n" ,dev_name (&vdpa_aux->padev->aux_dev.dev)); + mgmt = &vdpa_aux->vdpa_mdev; +seq_printf (seq,"max_vqs: %d\n" , mgmt->max_supported_vqs); +seq_printf (seq,"config_attr_mask: %#llx\n" , mgmt->config_attr_mask); +seq_printf (seq,"supported_features: %#llx\n" , mgmt->supported_features); +print_feature_bits (seq, mgmt->supported_features); +seq_printf (seq,"local_mac_bit: %d\n" , vdpa_aux->local_mac_bit); +return 0 ; }DEFINE_SHOW_ATTRIBUTE (identity);@@ -64,4 +139,96 @@ pds_vdpa_debugfs_add_ident(struct pds_vdpa_aux *vdpa_aux) vdpa_aux, &identity_fops); } +static int +config_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_device *pdsv = seq->private; +struct virtio_net_config *vc = &pdsv->vn_config; + +seq_printf (seq,"mac: %pM\n" , vc->mac); +seq_printf (seq,"max_virtqueue_pairs: %d\n" , +__virtio16_to_cpu (true, vc->max_virtqueue_pairs)); +seq_printf (seq,"mtu: %d\n" ,__virtio16_to_cpu (true, vc->mtu)); +seq_printf (seq,"speed: %d\n" ,le32_to_cpu (vc->speed)); +seq_printf (seq,"duplex: %d\n" , vc->duplex); +seq_printf (seq,"rss_max_key_size: %d\n" , vc->rss_max_key_size); +seq_printf (seq,"rss_max_indirection_table_length: %d\n" , +le16_to_cpu (vc->rss_max_indirection_table_length)); +seq_printf (seq,"supported_hash_types: %#x\n" , +le32_to_cpu (vc->supported_hash_types)); +seq_printf (seq,"vn_status: %#x\n" , +__virtio16_to_cpu (true, vc->status)); +print_status_bits (seq,__virtio16_to_cpu (true, vc->status)); + +seq_printf (seq,"hw_status: %#x\n" , pdsv->hw.status); +print_status_bits (seq, pdsv->hw.status); +seq_printf (seq,"req_features: %#llx\n" , pdsv->hw.req_features); +print_feature_bits (seq, pdsv->hw.req_features); +seq_printf (seq,"actual_features: %#llx\n" , pdsv->hw.actual_features); +print_feature_bits (seq, pdsv->hw.actual_features); +seq_printf (seq,"vdpa_index: %d\n" , pdsv->hw.vdpa_index); +seq_printf (seq,"num_vqs: %d\n" , pdsv->hw.num_vqs); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (config); + +static int +vq_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_vq_info *vq = seq->private; +struct pds_vdpa_intr_info *intrs; + +seq_printf (seq,"ready: %d\n" , vq->ready); +seq_printf (seq,"desc_addr: %#llx\n" , vq->desc_addr); +seq_printf (seq,"avail_addr: %#llx\n" , vq->avail_addr); +seq_printf (seq,"used_addr: %#llx\n" , vq->used_addr); +seq_printf (seq,"q_len: %d\n" , vq->q_len); +seq_printf (seq,"qid: %d\n" , vq->qid); + +seq_printf (seq,"doorbell: %#llx\n" , vq->doorbell); +seq_printf (seq,"avail_idx: %d\n" , vq->avail_idx); +seq_printf (seq,"used_idx: %d\n" , vq->used_idx); +seq_printf (seq,"intr_index: %d\n" , vq->intr_index); + + intrs = vq->pdsv->vdpa_aux->vdpa_vf->intrs; +seq_printf (seq,"irq: %d\n" , intrs[vq->intr_index].irq); +seq_printf (seq,"irq-name: %s\n" , intrs[vq->intr_index].name); + +seq_printf (seq,"hw_qtype: %d\n" , vq->hw_qtype); +seq_printf (seq,"hw_qindex: %d\n" , vq->hw_qindex); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (vq); + +void +pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) +{ +struct dentry *dentry; +const char *name; +int i; + + dentry = pdsv->vdpa_aux->vdpa_vf->dentry; + name =dev_name (&pdsv->vdpa_dev.dev); + + pdsv->dentry =debugfs_create_dir (name, dentry); + +debugfs_create_file ("config" ,0400 , pdsv->dentry, pdsv, &config_fops); + +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +char name[8 ]; + +snprintf (name,sizeof (name),"vq%02d" , i); +debugfs_create_file (name,0400 , pdsv->dentry, &pdsv->hw.vqs[i], &vq_fops); + } +} + +void +pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) +{ +debugfs_remove_recursive (pdsv->dentry); + pdsv->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */ drivers/vdpa/pds/debugfs.h +4
-0@@ -13,12 +13,16 @@ void pds_vdpa_debugfs_destroy(void);void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux); +void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv); +void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv);#else static inline void pds_vdpa_debugfs_create (void ) { }static inline void pds_vdpa_debugfs_destroy (void ) { }static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) { } +static inline void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) { } +static inline void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) { }#endif #endif /* _PDS_VDPA_DEBUGFS_H_ */ drivers/vdpa/pds/vdpa_dev.c new file +156
-0@@ -0,0 +1,796 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/sysfs.h> +#include <linux/types.h> +#include <linux/vdpa.h> +#include <uapi/linux/virtio_pci.h> +#include <uapi/linux/vdpa.h> + +#include <linux/pds/pds_intr.h> +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> +#include <linux/pds/pds_vdpa.h> + +#include "vdpa_dev.h" +#include "pci_drv.h" +#include "aux_drv.h" +#include "pci_drv.h" +#include "cmds.h" +#include "debugfs.h" + +static int +pds_vdpa_setup_driver (struct pds_vdpa_device *pdsv) +{ +struct device *dev = &pdsv->vdpa_dev.dev; +int err =0 ; +int i; + +/* Verify all vqs[] are in ready state */ +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +if (!pdsv->hw.vqs[i].ready) { +dev_warn (dev,"%s: qid %d not ready\n" , __func__, i); + err = -ENOENT ; + } + } + +return err; +} + +static struct pds_vdpa_device * +vdpa_to_pdsv (struct vdpa_device *vdpa_dev) +{ + return container_of(vdpa_dev, struct pds_vdpa_device, vdpa_dev); +} + +static struct pds_vdpa_hw * +vdpa_to_hw (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return &pdsv->hw; +} + +static int +pds_vdpa_set_vq_address (struct vdpa_device *vdpa_dev,u16 qid, +u64 desc_addr,u64 driver_addr,u64 device_addr) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].desc_addr = desc_addr; + hw->vqs[qid].avail_addr = driver_addr; + hw->vqs[qid].used_addr = device_addr; + +return 0 ; +} + +static void +pds_vdpa_set_vq_num (struct vdpa_device *vdpa_dev,u16 qid,u32 num) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].q_len = num; +} + +static void +pds_vdpa_kick_vq (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +iowrite16 (qid, pdsv->hw.vqs[qid].notify); +} + +static void +pds_vdpa_set_vq_cb (struct vdpa_device *vdpa_dev,u16 qid, +struct vdpa_callback *cb) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].event_cb = *cb; +} + +static irqreturn_t +pds_vdpa_isr (int irq,void *data) +{ +struct pds_core_intr __iomem *intr_ctrl; +struct pds_vdpa_device *pdsv; +struct pds_vdpa_vq_info *vq; + + vq = data; + pdsv = vq->pdsv; + +if (vq->event_cb.callback) + vq->event_cb.callback (vq->event_cb.private); + +/* Since we don't actually know how many vq descriptors are +* covered in this interrupt cycle, we simply clean all the +* credits and re-enable the interrupt. +*/ + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; +pds_core_intr_clean_flags (&intr_ctrl[vq->intr_index], +PDS_CORE_INTR_CRED_REARM ); + +return IRQ_HANDLED ; +} + +static void +pds_vdpa_release_irq (struct pds_vdpa_device *pdsv,int qid) +{ +struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_core_intr __iomem *intr_ctrl; +int index; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; + index = pdsv->hw.vqs[qid].intr_index; +if (index ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (intrs[index].irq ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (qid &0x1 ) { + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + }else { +pds_core_intr_mask (&intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); +devm_free_irq (&pdev->dev, intrs[index].irq, &pdsv->hw.vqs[qid]); + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + intrs[index].irq =VIRTIO_MSI_NO_VECTOR ; + } +} + +static void +pds_vdpa_set_vq_ready (struct vdpa_device *vdpa_dev,u16 qid,bool ready) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +struct pds_core_intr __iomem *intr_ctrl; +int err; + + dev_dbg(dev, "%s: qid %d ready %d => %d\n",
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 161
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+ __func__, qid, hw->vqs[qid].ready, ready); +if (ready == hw->vqs[qid].ready) +return ; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr;It looks to me pds has a different layout/semantic for isr than virtio spec. I'd suggest to not reuse spec isr here to avoid confusion.
Hmm, yes, that needs some straightening out.
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 162
Quoted 2 lines
drivers/vdpa/pds/vdpa_dev.c new file +1
-0+if (ready) {Spec said no interrupt before
DRIVER_OK, it looks more simple if we move the interrupt setup toset_status():E.g we can know if we have sufficient vectors and use different mapping policies in advance.
I'll look at that.
Quoted 92 lines
Quoted 26 lines
drivers/vdpa/pds/vdpa_dev.c new file +25
-0+struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +int index =VIRTIO_MSI_NO_VECTOR ; +int i; + +/* Tx and Rx queues share interrupts, and they start with +* even numbers, so only find an interrupt for the even numbered +* qid, and let the odd number use what the previous queue got. +*/ +if (qid &0x1 ) { +int even = qid & ~0x1 ; + + index = hw->vqs[even].intr_index; + }else { +for (i =0 ; i < pdsv->vdpa_aux->vdpa_vf->nintrs; i++) { +if (intrs[i].irq ==VIRTIO_MSI_NO_VECTOR ) { + index = i; +break ; + } + } + } + +if (qid &0x1 ) { + hw->vqs[qid].intr_index = index; + }else if (index !=VIRTIO_MSI_NO_VECTOR ) { +int irq;drivers/vdpa/pds/vdpa_dev.c line 197
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ + irq =pci_irq_vector (pdev, index); +snprintf (intrs[index].name,sizeof (intrs[index].name), +"vdpa-%s-%d" ,dev_name (dev), index); + + err =devm_request_irq (&pdev->dev, irq, pds_vdpa_isr,0 , + intrs[index].name, &hw->vqs[qid]); +if (err) { +dev_info (dev,"%s: no irq for qid %d: %pe\n" , + __func__, qid,ERR_PTR (err));Should we fail?
drivers/vdpa/pds/vdpa_dev.c line 202
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+ }else { + intrs[index].irq = irq; + hw->vqs[qid].intr_index = index; +pds_core_intr_mask (&intr_ctrl[index], +PDS_CORE_INTR_MASK_CLEAR );I guess the reason that you don't simply use VF MSI-X is the DPU can support vDPA subdevice in the future?
drivers/vdpa/pds/vdpa_dev.c line 206
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ } + }else { +dev_info (dev,"%s: no intr slot for qid %d\n" , + __func__, qid);Do we need to fail here?
Quoted 37 lines
drivers/vdpa/pds/vdpa_dev.c new file +37
-0+ } + +/* Pass vq setup info to DSC */ + err =pds_vdpa_cmd_init_vq (pdsv, qid, &hw->vqs[qid]); +if (err) { +pds_vdpa_release_irq (pdsv, qid); + ready = false; + } + }else { +pds_vdpa_release_irq (pdsv, qid); + (void )pds_vdpa_cmd_reset_vq (pdsv, qid); + } + + hw->vqs[qid].ready = ready; +} + +static bool +pds_vdpa_get_vq_ready (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->vqs[qid].ready; +} + +static int +pds_vdpa_set_vq_state (struct vdpa_device *vdpa_dev,u16 qid, +const struct vdpa_vq_state *state) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].used_idx = state->split.avail_index; + hw->vqs[qid].avail_idx = state->split.avail_index; + +return 0 ; +} + +static int
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 249
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+pds_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid, + struct vdpa_vq_state *state) +{ + struct pds_vdpa_hw *hw = vdpa_to_hw(vdpa_dev); + + state->split.avail_index = hw->vqs[qid].avail_idx;Who is in charge of reading
avail_idxfrom the hardware?
We didn't have that available in the early FW, so it isn't here yet. Work in progerss.
Quoted 31 lines
Quoted 31 lines
drivers/vdpa/pds/vdpa_dev.c new file +30
-0+ +return 0 ; +} + +static struct vdpa_notification_area +pds_vdpa_get_vq_notification (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct virtio_pci_modern_device *vd_mdev; +struct vdpa_notification_area area; + + area.addr = hw->vqs[qid].notify_pa; + + vd_mdev = &pdsv->vdpa_aux->vdpa_vf->vd_mdev; +if (!vd_mdev->notify_offset_multiplier) + area.size =PAGE_SIZE ; +else + area.size = vd_mdev->notify_offset_multiplier; + +return area; +} + +static int +pds_vdpa_get_vq_irq (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +int irq =VIRTIO_MSI_NO_VECTOR ; +int index;
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 283
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ +if (pdsv->vdpa_aux->vdpa_vf->intrs) { + index = hw->vqs[qid].intr_index; + irq = pdsv->vdpa_aux->vdpa_vf->intrs[index].irq;The notification area mapping might only work well when each vq has it's own irq. Otherwise guest may see spurious interrupt which may degrade the performance.
We haven't been expecting to use shared interrupts - are we being overly optimistic?
Quoted 36 lines
Quoted 36 lines
drivers/vdpa/pds/vdpa_dev.c new file +35
-0+ } + +return irq; +} + +static u32 +pds_vdpa_get_vq_align (struct vdpa_device *vdpa_dev) +{ + +return PAGE_SIZE ; +} + +static u32 +pds_vdpa_get_vq_group (struct vdpa_device *vdpa_dev,u16 idx) +{ +return 0 ; +} + +static u64 +pds_vdpa_get_device_features (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return le64_to_cpu (pdsv->vdpa_aux->ident.hw_features); +} + +static int +pds_vdpa_set_driver_features (struct vdpa_device *vdpa_dev,u64 features) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +u64 nego_features; +u64 set_features; +u64 missing;
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 323
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+int err; + +if (!(features &BIT_ULL (VIRTIO_F_ACCESS_PLATFORM )) && features) { +dev_err (dev,"VIRTIO_F_ACCESS_PLATFORM is not negotiated\n" ); +return -EOPNOTSUPP ;Should we fail the
FEATURE_OKin this case and all the other below error conditions?
Perhaps I'm missing a nuance in the inteface... isn't that what we're doing by returning a non-zero status?
Quoted 17 lines
Quoted 17 lines
drivers/vdpa/pds/vdpa_dev.c new file +16
-0+ } + + hw->req_features = features; + +/* Check for valid feature bits */ + nego_features = features &le64_to_cpu (pdsv->vdpa_aux->ident.hw_features); + missing = hw->req_features & ~nego_features; +if (missing) { +dev_err (dev,"Can't support all requested features in %#llx, missing %#llx features\n" , + hw->req_features, missing); +return -EOPNOTSUPP ; + } + +dev_dbg (dev,"%s: %#llx => %#llx\n" , + __func__, hw->actual_features, nego_features); +
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 345
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+if (hw->actual_features == nego_features) +return 0 ; + +/* Update hw feature configuration, strip MAC bit if locally set */ +if (pdsv->vdpa_aux->local_mac_bit) + set_features = nego_features & ~BIT_ULL (VIRTIO_NET_F_MAC );Need some document to explain how
local_mac_bitwork.
I'll look at expanding the comment in pds_vdpa_get_mgmt_info() where this would get set to true. Basically, it is tracking whether or not the driver is faking the VIRTIO_NET_F_MAC capability for the device.
Quoted 30 lines
Quoted 30 lines
drivers/vdpa/pds/vdpa_dev.c new file +29
-0+else + set_features = nego_features; + err =pds_vdpa_cmd_set_features (pdsv, set_features); +if (!err) + hw->actual_features = nego_features; + +return err; +} + +static u64 +pds_vdpa_get_driver_features (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->actual_features; +} + +static void +pds_vdpa_set_config_cb (struct vdpa_device *vdpa_dev,struct vdpa_callback *cb) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->config_cb.callback = cb->callback; + hw->config_cb.private = cb->private; +} + +static u16 +pds_vdpa_get_vq_num_max (struct vdpa_device *vdpa_dev) +{
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 379
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +u32 max_qlen; + + max_qlen =min_t (u32,PDS_VDPA_MAX_QLEN , +1 <<le16_to_cpu (pdsv->vdpa_aux->ident.max_qlen));Assuming we can fetch the
max_qlenfrom the device, any reason have another layer likePDS_VDPA_MAX_QLEN?
Yes, I think this can be simplified to just the read.
Quoted 17 lines
Quoted 17 lines
drivers/vdpa/pds/vdpa_dev.c new file +16
-0+ +return (u16 )max_qlen; +} + +static u32 +pds_vdpa_get_device_id (struct vdpa_device *vdpa_dev) +{ +return VIRTIO_ID_NET ; +} + +static u32 +pds_vdpa_get_vendor_id (struct vdpa_device *vdpa_dev) +{ +return PCI_VENDOR_ID_PENSANDO ; +} +
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 401
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+static u8 +pds_vdpa_get_status (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->status;How is this synchronized with the device or it is fully emulated by this driver?
For now has been fronted by the driver, but probably should be coming out of the virtio_net_config block.
Quoted 42 lines
Quoted 42 lines
drivers/vdpa/pds/vdpa_dev.c new file +41
-0+} + +static void +pds_vdpa_set_status (struct vdpa_device *vdpa_dev,u8 status) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +int err; + +if (hw->status == status) +return ; + +/* If the DRIVER_OK bit turns on, time to start the queues */ +if ((status ^ hw->status) &VIRTIO_CONFIG_S_DRIVER_OK ) { +if (status &VIRTIO_CONFIG_S_DRIVER_OK ) { + err =pds_vdpa_setup_driver (pdsv); +if (err) { +dev_err (dev,"failed to setup driver: %pe\n" ,ERR_PTR (err)); + status = hw->status |VIRTIO_CONFIG_S_FAILED ; + } + }else { +dev_warn (dev,"did not expect DRIVER_OK to be cleared\n" ); + } + } + + err =pds_vdpa_cmd_set_status (pdsv, status); +if (!err) + hw->status = status; +} + +static int +pds_vdpa_reset (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +int i; + +if (hw->status ==0 ) +return 0 ; +
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 448
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+if (hw->status &VIRTIO_CONFIG_S_DRIVER_OK ) { + +/* Reset the vqs */ +for (i =0 ; i < hw->num_vqs; i++) { +pds_vdpa_release_irq (pdsv, i); + (void )pds_vdpa_cmd_reset_vq (pdsv, i);(void) is unnecessary.
yep
Quoted 40 lines
Quoted 40 lines
drivers/vdpa/pds/vdpa_dev.c new file +39
-0+ +memset (&pdsv->hw.vqs[i],0 ,sizeof (pdsv->hw.vqs[0 ])); + pdsv->hw.vqs[i].ready = false; + } + } + + hw->status =0 ; + (void )pds_vdpa_cmd_set_status (pdsv,0 ); + +return 0 ; +} + +static size_t +pds_vdpa_get_config_size (struct vdpa_device *vdpa_dev) +{ +return sizeof (struct virtio_net_config ); +} + +static void +pds_vdpa_get_config (struct vdpa_device *vdpa_dev, +unsigned int offset, +void *buf,unsigned int len) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +if (offset + len <=sizeof (struct virtio_net_config )) +memcpy (buf, (u8 *)&pdsv->vn_config + offset, len); +} + +static void +pds_vdpa_set_config (struct vdpa_device *vdpa_dev, +unsigned int offset,const void *buf, +unsigned int len) +{ +/* In the virtio_net context, this callback seems to only be +* called in drivers supporting the older non-VERSION_1 API, +* so we can leave this an empty function, but we need to +* define the function in case it does get called, as there +* are currently no checks for existence before calling in
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 493
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+* that path. +* +* The implementation would be something like: +* if (offset + len <= sizeof(struct virtio_net_config)) +* memcpy((u8 *)&pdsv->vn_config + offset, buf, len); +*/ And we need to notify the hardware that config has been changed.
Sure
Quoted 84 lines
Quoted 84 lines
drivers/vdpa/pds/vdpa_dev.c new file +83
-0+} + +static const struct vdpa_config_ops pds_vdpa_ops = { + .set_vq_address = pds_vdpa_set_vq_address, + .set_vq_num = pds_vdpa_set_vq_num, + .kick_vq = pds_vdpa_kick_vq, + .set_vq_cb = pds_vdpa_set_vq_cb, + .set_vq_ready = pds_vdpa_set_vq_ready, + .get_vq_ready = pds_vdpa_get_vq_ready, + .set_vq_state = pds_vdpa_set_vq_state, + .get_vq_state = pds_vdpa_get_vq_state, + .get_vq_notification = pds_vdpa_get_vq_notification, + .get_vq_irq = pds_vdpa_get_vq_irq, + .get_vq_align = pds_vdpa_get_vq_align, + .get_vq_group = pds_vdpa_get_vq_group, + + .get_device_features = pds_vdpa_get_device_features, + .set_driver_features = pds_vdpa_set_driver_features, + .get_driver_features = pds_vdpa_get_driver_features, + .set_config_cb = pds_vdpa_set_config_cb, + .get_vq_num_max = pds_vdpa_get_vq_num_max, +/* .get_vq_num_min (optional) */ + .get_device_id = pds_vdpa_get_device_id, + .get_vendor_id = pds_vdpa_get_vendor_id, + .get_status = pds_vdpa_get_status, + .set_status = pds_vdpa_set_status, + .reset = pds_vdpa_reset, + .get_config_size = pds_vdpa_get_config_size, + .get_config = pds_vdpa_get_config, + .set_config = pds_vdpa_set_config, + +/* .get_generation (optional) */ +/* .get_iova_range (optional) */ +/* .set_group_asid */ +/* .set_map (optional) */ +/* .dma_map (optional) */ +/* .dma_unmap (optional) */ +/* .free (optional) */ +}; +static struct virtio_device_id pds_vdpa_id_table[] = { + {VIRTIO_ID_NET ,VIRTIO_DEV_ANY_ID }, + {0 }, +}; + +static int +pds_vdpa_dev_add (struct vdpa_mgmt_dev *mdev,const char *name, +const struct vdpa_dev_set_config *add_config) +{ +struct pds_vdpa_aux *vdpa_aux; +struct pds_vdpa_device *pdsv; +struct vdpa_mgmt_dev *mgmt; +u16 fw_max_vqs, vq_pairs; +struct device *dma_dev; +struct pds_vdpa_hw *hw; +struct pci_dev *pdev; +struct device *dev; +u8 mac[ETH_ALEN ]; +int err; +int i; + + vdpa_aux = container_of(mdev, struct pds_vdpa_aux, vdpa_mdev); + dev = &vdpa_aux->padev->aux_dev.dev; + mgmt = &vdpa_aux->vdpa_mdev; + +if (vdpa_aux->pdsv) { +dev_warn (dev,"Multiple vDPA devices on a VF is not supported.\n" ); +return -EOPNOTSUPP ; + } + + pdsv = vdpa_alloc_device(struct pds_vdpa_device, vdpa_dev, + dev, &pds_vdpa_ops,1 ,1 , name, false); +if (IS_ERR (pdsv)) { +dev_err (dev,"Failed to allocate vDPA structure: %pe\n" , pdsv); +return PTR_ERR (pdsv); + } + + vdpa_aux->pdsv = pdsv; + pdsv->vdpa_aux = vdpa_aux; + pdsv->vdpa_aux->padev->priv = pdsv; + + pdev = vdpa_aux->vdpa_vf->pdev; + pdsv->vdpa_dev.dma_dev = &pdev->dev; + dma_dev = pdsv->vdpa_dev.dma_dev;
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 580
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ hw = &pdsv->hw; + + pdsv->vn_config_pa =dma_map_single (dma_dev, &pdsv->vn_config, +sizeof (pdsv->vn_config),DMA_FROM_DEVICE );I think we should use coherent mapping instead of streaming mapping otherwise we may end up with coherency issues when accessing the device configuration space.
Makes sense
Quoted 32 lines
Quoted 32 lines
drivers/vdpa/pds/vdpa_dev.c new file +31
-0+if (dma_mapping_error (dma_dev, pdsv->vn_config_pa)) { +dev_err (dma_dev,"Failed to map vn_config space\n" ); + pdsv->vn_config_pa =0 ; + err = -ENOMEM ; + goto err_out; + } + + err =pds_vdpa_init_hw (pdsv); +if (err) { +dev_err (dev,"Failed to init hw: %pe\n" ,ERR_PTR (err)); + goto err_unmap; + } + + fw_max_vqs =le16_to_cpu (pdsv->vdpa_aux->ident.max_vqs); + vq_pairs = fw_max_vqs /2 ; + +/* Make sure we have the queues being requested */ +if (add_config->mask & (1 <<VDPA_ATTR_DEV_NET_CFG_MAX_VQP )) + vq_pairs = add_config->net.max_vq_pairs; + + hw->num_vqs =2 * vq_pairs; +if (mgmt->supported_features &BIT_ULL (VIRTIO_NET_F_CTRL_VQ )) + hw->num_vqs++; + +if (hw->num_vqs > fw_max_vqs) { +dev_err (dev,"%s: queue count requested %u greater than max %u\n" , + __func__, hw->num_vqs, fw_max_vqs); + err = -ENOSPC ; + goto err_unmap; + } +
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 617
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+if (hw->num_vqs != fw_max_vqs) { + err =pds_vdpa_cmd_set_max_vq_pairs (pdsv, vq_pairs); +if (err == -ERANGE ) { + hw->num_vqs = fw_max_vqs; +dev_warn (dev,"Known FW issue - overriding to use max_vq_pairs %d\n" , + hw->num_vqs /2 );Should we fail here? Since the device has a different
max_vqpthat expected.
Wasn't sure if we should annoy users with a fail here, or try to adjust and continue on with something that should work.
Quoted 17 lines
Quoted 17 lines
drivers/vdpa/pds/vdpa_dev.c new file +16
-0+ }else if (err) { +dev_err (dev,"Failed to update max_vq_pairs: %pe\n" , +ERR_PTR (err)); + goto err_unmap; + } + } + +/* Set a mac, either from the user config if provided +* or set a random mac if default is 00:..:00 +*/ +if (add_config->mask & (1 <<VDPA_ATTR_DEV_NET_CFG_MACADDR )) { +ether_addr_copy (mac, add_config->net.mac); +pds_vdpa_cmd_set_mac (pdsv, mac); + }else if (is_zero_ether_addr (pdsv->vn_config.mac)) { +eth_random_addr (mac); +pds_vdpa_cmd_set_mac (pdsv, mac);
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 639
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+ } + +for (i =0 ; i < hw->num_vqs; i++) { + hw->vqs[i].qid = i; + hw->vqs[i].pdsv = pdsv; + hw->vqs[i].intr_index =VIRTIO_MSI_NO_VECTOR ;Let's rename this as
msix_vectorto be aligned with the virtio spec.
Sure.
Quoted 14 lines
Quoted 14 lines
drivers/vdpa/pds/vdpa_dev.c new file +13
-0+ hw->vqs[i].notify =vp_modern_map_vq_notify (&pdsv->vdpa_aux->vdpa_vf->vd_mdev, + i, &hw->vqs[i].notify_pa); + } + + pdsv->vdpa_dev.mdev = &vdpa_aux->vdpa_mdev; + +/* We use the _vdpa_register_device() call rather than the +* vdpa_register_device() to avoid a deadlock because this +* dev_add() is called with the vdpa_dev_lock already set +* by vdpa_nl_cmd_dev_add_set_doit() +*/ + err =_vdpa_register_device (&pdsv->vdpa_dev, hw->num_vqs); +if (err) {
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 658
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+dev_err (dev,"Failed to register to vDPA bus: %pe\n" ,ERR_PTR (err)); + goto err_unmap; + } + +pds_vdpa_debugfs_add_vdpadev (pdsv); +dev_info (&pdsv->vdpa_dev.dev,"Added with mac %pM\n" , pdsv->vn_config.mac);
dev_dbg?
That was the eventual idea.
Quoted 64 lines
Quoted 64 lines
drivers/vdpa/pds/vdpa_dev.c new file +63
-0+ +return 0 ; + +err_unmap: +dma_unmap_single (dma_dev, pdsv->vn_config_pa, +sizeof (pdsv->vn_config),DMA_FROM_DEVICE ); +err_out: +put_device (&pdsv->vdpa_dev.dev); + vdpa_aux->pdsv =NULL ; +return err; +} + +static void +pds_vdpa_dev_del (struct vdpa_mgmt_dev *mdev,struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_aux *vdpa_aux; + +dev_info (&vdpa_dev->dev,"Removed\n" ); + + vdpa_aux = container_of(mdev, struct pds_vdpa_aux, vdpa_mdev); +_vdpa_unregister_device (vdpa_dev); +pds_vdpa_debugfs_del_vdpadev (pdsv); + +if (vdpa_aux->pdsv->vn_config_pa) +dma_unmap_single (vdpa_dev->dma_dev, vdpa_aux->pdsv->vn_config_pa, +sizeof (vdpa_aux->pdsv->vn_config),DMA_FROM_DEVICE ); + + vdpa_aux->pdsv =NULL ; +} + +static const struct vdpa_mgmtdev_ops pds_vdpa_mgmt_dev_ops = { + .dev_add = pds_vdpa_dev_add, + .dev_del = pds_vdpa_dev_del +}; + +int +pds_vdpa_get_mgmt_info (struct pds_vdpa_aux *vdpa_aux) +{ +struct pds_vdpa_pci_device *vdpa_pdev; +struct pds_vdpa_ident_cmd ident_cmd = { + .opcode =PDS_VDPA_CMD_IDENT , + .vf_id =cpu_to_le16 (vdpa_aux->vdpa_vf->vf_id), + }; +struct pds_vdpa_comp ident_comp = {0 }; +struct vdpa_mgmt_dev *mgmt; +struct device *dma_dev; +dma_addr_t ident_pa; +struct pci_dev *pdev; +struct device *dev; +__le64 mac_bit; +u16 max_vqs; +int err; +int i; + + vdpa_pdev = vdpa_aux->vdpa_vf; + pdev = vdpa_pdev->pdev; + dev = &vdpa_aux->padev->aux_dev.dev; + mgmt = &vdpa_aux->vdpa_mdev; + +/* Get resource info from the device */ + dma_dev = &pdev->dev; + ident_pa = dma_map_single(dma_dev, &vdpa_aux->ident,
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 722
Quoted 1 line
drivers/vdpa/pds/vdpa_dev.c new file +1
-0+ sizeof(vdpa_aux->ident), DMA_FROM_DEVICE);I wonder how this work. The
ident_pais mapped through VF, but the command is sent to PF adminq if I understand correctly. If yes, this might work but looks tricky. We'd better explain this is safe since vDPA is not yet created so no userspace can use that. Or I wonder if we can just piggyback the ident via the adminq response so we don't need to worry the security implications.
I'll work with our FW folks on this.
Thanks for all the comments,
sln
Quoted 81 lines
Thanks
Quoted 77 lines
drivers/vdpa/pds/vdpa_dev.c new file +74
-1+if (dma_mapping_error (dma_dev, ident_pa)) { +dev_err (dma_dev,"Failed to map ident space\n" ); +return -ENOMEM ; + } + + ident_cmd.ident_pa =cpu_to_le64 (ident_pa); + ident_cmd.len =cpu_to_le32 (sizeof (vdpa_aux->ident)); + err = vdpa_aux->padev->ops->adminq_cmd (vdpa_aux->padev, + (union pds_core_adminq_cmd *)&ident_cmd, +sizeof (ident_cmd), + (union pds_core_adminq_comp *)&ident_comp, +0 ); +dma_unmap_single (dma_dev, ident_pa, +sizeof (vdpa_aux->ident),DMA_FROM_DEVICE ); +if (err) { +dev_err (dev,"Failed to ident hw, status %d: %pe\n" , + ident_comp.status,ERR_PTR (err)); +return err; + } + +/* The driver adds a default mac address if the device doesn't, +* so we need to sure we advertise VIRTIO_NET_F_MAC +*/ + mac_bit =cpu_to_le64 (BIT_ULL (VIRTIO_NET_F_MAC )); +if (!(vdpa_aux->ident.hw_features & mac_bit)) { + vdpa_aux->ident.hw_features |= mac_bit; + vdpa_aux->local_mac_bit = true; + } + + max_vqs =le16_to_cpu (vdpa_aux->ident.max_vqs); + mgmt->max_supported_vqs =min_t (u16,PDS_VDPA_MAX_QUEUES , max_vqs); +if (max_vqs >PDS_VDPA_MAX_QUEUES ) +dev_info (dev,"FYI - Device supports more vqs (%d) than driver (%d)\n" , + max_vqs,PDS_VDPA_MAX_QUEUES ); + + mgmt->ops = &pds_vdpa_mgmt_dev_ops; + mgmt->id_table = pds_vdpa_id_table; + mgmt->device = dev; + mgmt->supported_features =le64_to_cpu (vdpa_aux->ident.hw_features); + mgmt->config_attr_mask =BIT_ULL (VDPA_ATTR_DEV_NET_CFG_MACADDR ); + mgmt->config_attr_mask |=BIT_ULL (VDPA_ATTR_DEV_NET_CFG_MAX_VQP ); + +/* Set up interrupts now that we know how many we might want +* TX and RX pairs will share interrupts, so halve the vq count +* Add another for a control queue if supported +*/ + vdpa_pdev->nintrs = mgmt->max_supported_vqs /2 ; +if (mgmt->supported_features &BIT_ULL (VIRTIO_NET_F_CTRL_VQ )) + vdpa_pdev->nintrs++; + + err =pci_alloc_irq_vectors (pdev, vdpa_pdev->nintrs, vdpa_pdev->nintrs, +PCI_IRQ_MSIX ); +if (err <0 ) { +dev_err (dma_dev,"Couldn't get %d msix vectors: %pe\n" , + vdpa_pdev->nintrs,ERR_PTR (err)); +return err; + } + vdpa_pdev->nintrs = err; + err =0 ; + + vdpa_pdev->intrs =devm_kcalloc (&pdev->dev, vdpa_pdev->nintrs, +sizeof (*vdpa_pdev->intrs), +GFP_KERNEL ); +if (!vdpa_pdev->intrs) { + vdpa_pdev->nintrs =0 ; +pci_free_irq_vectors (pdev); +return -ENOMEM ; + } + +for (i =0 ; i < vdpa_pdev->nintrs; i++) + vdpa_pdev->intrs[i].irq =VIRTIO_MSI_NO_VECTOR ; + +return 0 ; +} --2.17.1
Re: [RFC PATCH net-next 18/19] pds_vdpa: add support for vdpa and vdpamgmt interfaces
Quoted 652 lines · Shannon Nelson
Quoted 484 lines · Jason Wang
Quoted 483 lines · Shannon Nelson
This is the vDPA device support, where we advertise that we can support the virtio queues and deal with the configuration work through the pds_core's adminq.
Signed-off-by Shannon Nelson <[email protected]>--- drivers/vdpa/pds/Makefile | 3 +- drivers/vdpa/pds/aux_drv.c | 33 ++ drivers/vdpa/pds/debugfs.c | 167 ++++++++ drivers/vdpa/pds/debugfs.h | 4 + drivers/vdpa/pds/vdpa_dev.c | 796 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1002 insertions(+), 1 deletion(-)create mode 100644
drivers/vdpa/pds/vdpa_dev.cdrivers/vdpa/pds/Makefile +2
-1@@ -7,4 +7,5 @@ pds_vdpa-y := aux_drv.o \ cmds.o \ pci_drv.o \ debugfs.o \ - virtio_pci.o + virtio_pci.o \ + vdpa_dev.odrivers/vdpa/pds/aux_drv.c +33
-0@@ -12,6 +12,7 @@#include <linux/pds/pds_vdpa.h> #include "aux_drv.h" +#include "vdpa_dev.h" #include "pci_drv.h" #include "debugfs.h" @@ -25,10 +26,25 @@ static void pds_vdpa_aux_notify_handler(struct pds_auxiliary_dev *padev, union pds_core_notifyq_comp *event) { + struct pds_vdpa_device *pdsv = padev->priv;struct device *dev = &padev->aux_dev.dev;u16 ecode =le16_to_cpu (event->ecode);dev_info (dev,"%s: event code %d\n" , __func__, ecode); + +/* Give the upper layers a hint that something interesting +* may have happened. It seems that the only thing this +* triggers in the virtio-net drivers above us is a check +* of link status. +* +* We don't set the NEEDS_RESET flag for EVENT_RESET +* because we're likely going through a recovery or +* fw_update and will be back up and running soon. +*/ +if (ecode ==PDS_EVENT_RESET || ecode ==PDS_EVENT_LINK_CHANGE ) { +if (pdsv->hw.config_cb.callback) + pdsv->hw.config_cb.callback (pdsv->hw.config_cb.private); + } } static int@@ -80,10 +96,25 @@ pds_vdpa_aux_probe(struct auxiliary_device *aux_dev, goto err_register_client; } +/* Get device ident info and set up the vdpa_mgmt_dev */ + err =pds_vdpa_get_mgmt_info (vdpa_aux); +if (err) + goto err_register_client; + +/* Let vdpa know that we can provide devices */ + err =vdpa_mgmtdev_register (&vdpa_aux->vdpa_mdev); +if (err) { +dev_err (dev,"%s: Failed to initialize vdpa_mgmt interface: %pe\n" , + __func__,ERR_PTR (err)); + goto err_mgmt_reg; + } +pds_vdpa_debugfs_add_ident (vdpa_aux);return 0 ; +err_mgmt_reg: + padev->ops->unregister_client (padev); err_register_client:auxiliary_set_drvdata (aux_dev,NULL ); err_invalid_driver:@@ -98,6 +129,8 @@ pds_vdpa_aux_remove(struct auxiliary_device *aux_dev)struct pds_vdpa_aux *vdpa_aux =auxiliary_get_drvdata (aux_dev);struct device *dev = &aux_dev->dev; +vdpa_mgmtdev_unregister (&vdpa_aux->vdpa_mdev); + vdpa_aux->padev->ops->unregister_client (vdpa_aux->padev);if (vdpa_aux->vdpa_vf)pci_dev_put (vdpa_aux->vdpa_vf->pdev);drivers/vdpa/pds/debugfs.c +167
-0@@ -11,6 +11,7 @@#include <linux/pds/pds_auxbus.h> #include <linux/pds/pds_vdpa.h> +#include "vdpa_dev.h" #include "aux_drv.h" #include "pci_drv.h" #include "debugfs.h" @@ -19,6 +20,72 @@static struct dentry *dbfs_dir; +#define PRINT_SBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & __name) \ + seq_printf(__seq, " %s", &#__name[16]); \ + } while (0) + +static void +print_status_bits (struct seq_file *seq,u16 status) +{ +seq_puts (seq,"status:" ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_ACKNOWLEDGE ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_DRIVER_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FEATURES_OK ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_NEEDS_RESET ); +PRINT_SBIT_NAME (seq, status,VIRTIO_CONFIG_S_FAILED ); +seq_puts (seq,"\n" ); +} + +#define PRINT_FBIT_NAME (__seq, __f, __name) \ + do { \ + if (__f & BIT_ULL(__name)) \ + seq_printf(__seq, " %s", #__name); \ + } while (0) + +static void +print_feature_bits (struct seq_file *seq,u64 features) +{ +seq_puts (seq,"features:" ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_CSUM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_GUEST_OFFLOADS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MTU ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MAC ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO4 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_TSO6 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_ECN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HOST_UFO ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MRG_RXBUF ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STATUS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_VLAN ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_RX_EXTRA ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_GUEST_ANNOUNCE ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_MQ ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_CTRL_MAC_ADDR ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_HASH_REPORT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSS ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_RSC_EXT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_STANDBY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_NET_F_SPEED_DUPLEX ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_NOTIFY_ON_EMPTY ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ANY_LAYOUT ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_VERSION_1 ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ACCESS_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_RING_PACKED ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_ORDER_PLATFORM ); +PRINT_FBIT_NAME (seq, features,VIRTIO_F_SR_IOV ); +seq_puts (seq,"\n" ); +} +void pds_vdpa_debugfs_create (void ) {@@ -49,10 +116,18 @@ static int identity_show(struct seq_file *seq, void *v) { struct pds_vdpa_aux *vdpa_aux = seq->private; +struct vdpa_mgmt_dev *mgmt;seq_printf (seq,"aux_dev: %s\n" ,dev_name (&vdpa_aux->padev->aux_dev.dev)); + mgmt = &vdpa_aux->vdpa_mdev; +seq_printf (seq,"max_vqs: %d\n" , mgmt->max_supported_vqs); +seq_printf (seq,"config_attr_mask: %#llx\n" , mgmt->config_attr_mask); +seq_printf (seq,"supported_features: %#llx\n" , mgmt->supported_features); +print_feature_bits (seq, mgmt->supported_features); +seq_printf (seq,"local_mac_bit: %d\n" , vdpa_aux->local_mac_bit); +return 0 ; }DEFINE_SHOW_ATTRIBUTE (identity);@@ -64,4 +139,96 @@ pds_vdpa_debugfs_add_ident(struct pds_vdpa_aux *vdpa_aux) vdpa_aux, &identity_fops); } +static int +config_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_device *pdsv = seq->private; +struct virtio_net_config *vc = &pdsv->vn_config; + +seq_printf (seq,"mac: %pM\n" , vc->mac); +seq_printf (seq,"max_virtqueue_pairs: %d\n" , +__virtio16_to_cpu (true, vc->max_virtqueue_pairs)); +seq_printf (seq,"mtu: %d\n" ,__virtio16_to_cpu (true, vc->mtu)); +seq_printf (seq,"speed: %d\n" ,le32_to_cpu (vc->speed)); +seq_printf (seq,"duplex: %d\n" , vc->duplex); +seq_printf (seq,"rss_max_key_size: %d\n" , vc->rss_max_key_size); +seq_printf (seq,"rss_max_indirection_table_length: %d\n" , +le16_to_cpu (vc->rss_max_indirection_table_length)); +seq_printf (seq,"supported_hash_types: %#x\n" , +le32_to_cpu (vc->supported_hash_types)); +seq_printf (seq,"vn_status: %#x\n" , +__virtio16_to_cpu (true, vc->status)); +print_status_bits (seq,__virtio16_to_cpu (true, vc->status)); + +seq_printf (seq,"hw_status: %#x\n" , pdsv->hw.status); +print_status_bits (seq, pdsv->hw.status); +seq_printf (seq,"req_features: %#llx\n" , pdsv->hw.req_features); +print_feature_bits (seq, pdsv->hw.req_features); +seq_printf (seq,"actual_features: %#llx\n" , pdsv->hw.actual_features); +print_feature_bits (seq, pdsv->hw.actual_features); +seq_printf (seq,"vdpa_index: %d\n" , pdsv->hw.vdpa_index); +seq_printf (seq,"num_vqs: %d\n" , pdsv->hw.num_vqs); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (config); + +static int +vq_show (struct seq_file *seq,void *v) +{ +struct pds_vdpa_vq_info *vq = seq->private; +struct pds_vdpa_intr_info *intrs; + +seq_printf (seq,"ready: %d\n" , vq->ready); +seq_printf (seq,"desc_addr: %#llx\n" , vq->desc_addr); +seq_printf (seq,"avail_addr: %#llx\n" , vq->avail_addr); +seq_printf (seq,"used_addr: %#llx\n" , vq->used_addr); +seq_printf (seq,"q_len: %d\n" , vq->q_len); +seq_printf (seq,"qid: %d\n" , vq->qid); + +seq_printf (seq,"doorbell: %#llx\n" , vq->doorbell); +seq_printf (seq,"avail_idx: %d\n" , vq->avail_idx); +seq_printf (seq,"used_idx: %d\n" , vq->used_idx); +seq_printf (seq,"intr_index: %d\n" , vq->intr_index); + + intrs = vq->pdsv->vdpa_aux->vdpa_vf->intrs; +seq_printf (seq,"irq: %d\n" , intrs[vq->intr_index].irq); +seq_printf (seq,"irq-name: %s\n" , intrs[vq->intr_index].name); + +seq_printf (seq,"hw_qtype: %d\n" , vq->hw_qtype); +seq_printf (seq,"hw_qindex: %d\n" , vq->hw_qindex); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (vq); + +void +pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) +{ +struct dentry *dentry; +const char *name; +int i; + + dentry = pdsv->vdpa_aux->vdpa_vf->dentry; + name =dev_name (&pdsv->vdpa_dev.dev); + + pdsv->dentry =debugfs_create_dir (name, dentry); + +debugfs_create_file ("config" ,0400 , pdsv->dentry, pdsv, &config_fops); + +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +char name[8 ]; + +snprintf (name,sizeof (name),"vq%02d" , i); +debugfs_create_file (name,0400 , pdsv->dentry, &pdsv->hw.vqs[i], &vq_fops); + } +} + +void +pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) +{ +debugfs_remove_recursive (pdsv->dentry); + pdsv->dentry =NULL ; +} +#endif /* CONFIG_DEBUG_FS */ drivers/vdpa/pds/debugfs.h +4
-0@@ -13,12 +13,16 @@ void pds_vdpa_debugfs_destroy(void);void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev);void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux); +void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv); +void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv);#else static inline void pds_vdpa_debugfs_create (void ) { }static inline void pds_vdpa_debugfs_destroy (void ) { }static inline void pds_vdpa_debugfs_add_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_del_pcidev (struct pds_vdpa_pci_device *vdpa_pdev) { }static inline void pds_vdpa_debugfs_add_ident (struct pds_vdpa_aux *vdpa_aux) { } +static inline void pds_vdpa_debugfs_add_vdpadev (struct pds_vdpa_device *pdsv) { } +static inline void pds_vdpa_debugfs_del_vdpadev (struct pds_vdpa_device *pdsv) { }#endif #endif /* _PDS_VDPA_DEBUGFS_H_ */ drivers/vdpa/pds/vdpa_dev.c new file +156
-0@@ -0,0 +1,796 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/sysfs.h> +#include <linux/types.h> +#include <linux/vdpa.h> +#include <uapi/linux/virtio_pci.h> +#include <uapi/linux/vdpa.h> + +#include <linux/pds/pds_intr.h> +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> +#include <linux/pds/pds_vdpa.h> + +#include "vdpa_dev.h" +#include "pci_drv.h" +#include "aux_drv.h" +#include "pci_drv.h" +#include "cmds.h" +#include "debugfs.h" + +static int +pds_vdpa_setup_driver (struct pds_vdpa_device *pdsv) +{ +struct device *dev = &pdsv->vdpa_dev.dev; +int err =0 ; +int i; + +/* Verify all vqs[] are in ready state */ +for (i =0 ; i < pdsv->hw.num_vqs; i++) { +if (!pdsv->hw.vqs[i].ready) { +dev_warn (dev,"%s: qid %d not ready\n" , __func__, i); + err = -ENOENT ; + } + } + +return err; +} + +static struct pds_vdpa_device * +vdpa_to_pdsv (struct vdpa_device *vdpa_dev) +{ + return container_of(vdpa_dev, struct pds_vdpa_device, vdpa_dev); +} + +static struct pds_vdpa_hw * +vdpa_to_hw (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return &pdsv->hw; +} + +static int +pds_vdpa_set_vq_address (struct vdpa_device *vdpa_dev,u16 qid, +u64 desc_addr,u64 driver_addr,u64 device_addr) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].desc_addr = desc_addr; + hw->vqs[qid].avail_addr = driver_addr; + hw->vqs[qid].used_addr = device_addr; + +return 0 ; +} + +static void +pds_vdpa_set_vq_num (struct vdpa_device *vdpa_dev,u16 qid,u32 num) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].q_len = num; +} + +static void +pds_vdpa_kick_vq (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +iowrite16 (qid, pdsv->hw.vqs[qid].notify); +} + +static void +pds_vdpa_set_vq_cb (struct vdpa_device *vdpa_dev,u16 qid, +struct vdpa_callback *cb) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].event_cb = *cb; +} + +static irqreturn_t +pds_vdpa_isr (int irq,void *data) +{ +struct pds_core_intr __iomem *intr_ctrl; +struct pds_vdpa_device *pdsv; +struct pds_vdpa_vq_info *vq; + + vq = data; + pdsv = vq->pdsv; + +if (vq->event_cb.callback) + vq->event_cb.callback (vq->event_cb.private); + +/* Since we don't actually know how many vq descriptors are +* covered in this interrupt cycle, we simply clean all the +* credits and re-enable the interrupt. +*/ + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; +pds_core_intr_clean_flags (&intr_ctrl[vq->intr_index], +PDS_CORE_INTR_CRED_REARM ); + +return IRQ_HANDLED ; +} + +static void +pds_vdpa_release_irq (struct pds_vdpa_device *pdsv,int qid) +{ +struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_core_intr __iomem *intr_ctrl; +int index; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr; + index = pdsv->hw.vqs[qid].intr_index; +if (index ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (intrs[index].irq ==VIRTIO_MSI_NO_VECTOR ) +return ; + +if (qid &0x1 ) { + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + }else { +pds_core_intr_mask (&intr_ctrl[index],PDS_CORE_INTR_MASK_SET ); +devm_free_irq (&pdev->dev, intrs[index].irq, &pdsv->hw.vqs[qid]); + pdsv->hw.vqs[qid].intr_index =VIRTIO_MSI_NO_VECTOR ; + intrs[index].irq =VIRTIO_MSI_NO_VECTOR ; + } +} + +static void +pds_vdpa_set_vq_ready (struct vdpa_device *vdpa_dev,u16 qid,bool ready) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pci_dev *pdev = pdsv->vdpa_aux->vdpa_vf->pdev; +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +struct pds_core_intr __iomem *intr_ctrl; +int err; + + dev_dbg(dev, "%s: qid %d ready %d => %d\n",Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 161
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+ __func__, qid, hw->vqs[qid].ready, ready); +if (ready == hw->vqs[qid].ready) +return ; + + intr_ctrl = (struct pds_core_intr __iomem *)pdsv->vdpa_aux->vdpa_vf->vd_mdev.isr;It looks to me pds has a different layout/semantic for isr than virtio spec. I'd suggest to not reuse spec isr here to avoid confusion.
Hmm, yes, that needs some straightening out.
Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 162
Quoted 2 lines
drivers/vdpa/pds/vdpa_dev.c new file +1
-0+if (ready) {Spec said no interrupt before
DRIVER_OK, it looks more simple if we move the interrupt setup toset_status():E.g we can know if we have sufficient vectors and use different mapping policies in advance.
I'll look at that.
Quoted 92 lines
Quoted 26 lines
drivers/vdpa/pds/vdpa_dev.c new file +25
-0+struct pds_vdpa_intr_info *intrs = pdsv->vdpa_aux->vdpa_vf->intrs; +int index =VIRTIO_MSI_NO_VECTOR ; +int i; + +/* Tx and Rx queues share interrupts, and they start with +* even numbers, so only find an interrupt for the even numbered +* qid, and let the odd number use what the previous queue got. +*/ +if (qid &0x1 ) { +int even = qid & ~0x1 ; + + index = hw->vqs[even].intr_index; + }else { +for (i =0 ; i < pdsv->vdpa_aux->vdpa_vf->nintrs; i++) { +if (intrs[i].irq ==VIRTIO_MSI_NO_VECTOR ) { + index = i; +break ; + } + } + } + +if (qid &0x1 ) { + hw->vqs[qid].intr_index = index; + }else if (index !=VIRTIO_MSI_NO_VECTOR ) { +int irq;drivers/vdpa/pds/vdpa_dev.c line 197
Quoted 10 lines
drivers/vdpa/pds/vdpa_dev.c new file +10
-0+ + irq =pci_irq_vector (pdev, index); +snprintf (intrs[index].name,sizeof (intrs[index].name), +"vdpa-%s-%d" ,dev_name (dev), index); + + err =devm_request_irq (&pdev->dev, irq, pds_vdpa_isr,0 , + intrs[index].name, &hw->vqs[qid]); +if (err) { +dev_info (dev,"%s: no irq for qid %d: %pe\n" , + __func__, qid,ERR_PTR (err));Should we fail?
drivers/vdpa/pds/vdpa_dev.c line 202
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+ }else { + intrs[index].irq = irq; + hw->vqs[qid].intr_index = index; +pds_core_intr_mask (&intr_ctrl[index], +PDS_CORE_INTR_MASK_CLEAR );I guess the reason that you don't simply use VF MSI-X is the DPU can support vDPA subdevice in the future?
drivers/vdpa/pds/vdpa_dev.c line 206
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ } + }else { +dev_info (dev,"%s: no intr slot for qid %d\n" , + __func__, qid);Do we need to fail here?
Quoted 37 lines
drivers/vdpa/pds/vdpa_dev.c new file +37
-0+ } + +/* Pass vq setup info to DSC */ + err =pds_vdpa_cmd_init_vq (pdsv, qid, &hw->vqs[qid]); +if (err) { +pds_vdpa_release_irq (pdsv, qid); + ready = false; + } + }else { +pds_vdpa_release_irq (pdsv, qid); + (void )pds_vdpa_cmd_reset_vq (pdsv, qid); + } + + hw->vqs[qid].ready = ready; +} + +static bool +pds_vdpa_get_vq_ready (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + +return hw->vqs[qid].ready; +} + +static int +pds_vdpa_set_vq_state (struct vdpa_device *vdpa_dev,u16 qid, +const struct vdpa_vq_state *state) +{ +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); + + hw->vqs[qid].used_idx = state->split.avail_index; + hw->vqs[qid].avail_idx = state->split.avail_index; + +return 0 ; +} + +static intQuoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 249
Quoted 6 lines
drivers/vdpa/pds/vdpa_dev.c new file +6
-0+pds_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid, + struct vdpa_vq_state *state) +{ + struct pds_vdpa_hw *hw = vdpa_to_hw(vdpa_dev); + + state->split.avail_index = hw->vqs[qid].avail_idx;Who is in charge of reading
avail_idxfrom the hardware?We didn't have that available in the early FW, so it isn't here yet. Work in progerss.
Quoted 31 lines
Quoted 31 lines
drivers/vdpa/pds/vdpa_dev.c new file +30
-0+ +return 0 ; +} + +static struct vdpa_notification_area +pds_vdpa_get_vq_notification (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct virtio_pci_modern_device *vd_mdev; +struct vdpa_notification_area area; + + area.addr = hw->vqs[qid].notify_pa; + + vd_mdev = &pdsv->vdpa_aux->vdpa_vf->vd_mdev; +if (!vd_mdev->notify_offset_multiplier) + area.size =PAGE_SIZE ; +else + area.size = vd_mdev->notify_offset_multiplier; + +return area; +} + +static int +pds_vdpa_get_vq_irq (struct vdpa_device *vdpa_dev,u16 qid) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +int irq =VIRTIO_MSI_NO_VECTOR ; +int index;Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 283
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c new file +4
-0+ +if (pdsv->vdpa_aux->vdpa_vf->intrs) { + index = hw->vqs[qid].intr_index; + irq = pdsv->vdpa_aux->vdpa_vf->intrs[index].irq;The notification area mapping might only work well when each vq has it's own irq. Otherwise guest may see spurious interrupt which may degrade the performance.
Quoted 2 lines
We haven't been expecting to use shared interrupts - are we being overly optimistic?
So at least from the codes above, I think we may end up with e.g two queues that are using the same irq? And the comment said:
/* Tx and Rx queues share interrupts, and they start with
* even numbers, so only find an interrupt for the
even numbered
* qid, and let the odd number use what the previous queue got.
*/
if (qid & 0x1) {
int even = qid & ~0x1;
index = hw->vqs[even].intr_index;It said TX and RX share interrupts.
Quoted 47 lines
Quoted 38 lines
Quoted 38 lines
drivers/vdpa/pds/vdpa_dev.c new file +35
-0+ } + +return irq; +} + +static u32 +pds_vdpa_get_vq_align (struct vdpa_device *vdpa_dev) +{ + +return PAGE_SIZE ; +} + +static u32 +pds_vdpa_get_vq_group (struct vdpa_device *vdpa_dev,u16 idx) +{ +return 0 ; +} + +static u64 +pds_vdpa_get_device_features (struct vdpa_device *vdpa_dev) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); + +return le64_to_cpu (pdsv->vdpa_aux->ident.hw_features); +} + +static int +pds_vdpa_set_driver_features (struct vdpa_device *vdpa_dev,u64 features) +{ +struct pds_vdpa_device *pdsv =vdpa_to_pdsv (vdpa_dev); +struct pds_vdpa_hw *hw =vdpa_to_hw (vdpa_dev); +struct device *dev = &pdsv->vdpa_dev.dev; +u64 nego_features; +u64 set_features; +u64 missing;Quoted 8 lines
drivers/vdpa/pds/vdpa_dev.c line 323
Quoted 5 lines
drivers/vdpa/pds/vdpa_dev.c new file +5
-0+int err; + +if (!(features &BIT_ULL (VIRTIO_F_ACCESS_PLATFORM )) && features) { +dev_err (dev,"VIRTIO_F_ACCESS_PLATFORM is not negotiated\n" ); +return -EOPNOTSUPP ;Should we fail the
FEATURE_OKin this case and all the other below error conditions?
Quoted 2 lines
Perhaps I'm missing a nuance in the inteface... isn't that what we're doing by returning a non-zero status?
Kind of, but to be compliant with the spec, the subsequent get_feature should return status without FEATURE_OK, I'm not sure this can be guaranteed:
static u8pds_vdpa_get_status(struct vdpa_device *vdpa_dev)
{
struct pds_vdpa_hw *hw = vdpa_to_hw(vdpa_dev);
return hw->status;
}Quoted 7 lines
Quoted 4 lines
drivers/vdpa/pds/vdpa_dev.c line 617
Quoted 2 lines
drivers/vdpa/pds/vdpa_dev.c new file +2
-0+dev_warn (dev,"Known FW issue - overriding to use max_vq_pairs %d\n" , + hw->num_vqs /2 );Should we fail here? Since the device has a different
max_vqpthat expected.Wasn't sure if we should annoy users with a fail here, or try to adjust and continue on with something that should work.
I think it's better to fail since it's the behaviour of other vDPA devices and software virtio devices.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RFC PATCH 19/19 pds_vdpa: add Kconfig entry and pds_vdpa.rst
In reply to Shannon Nelson--- .../ethernet/pensando/pds_vdpa.rst | 85 +++++++++++++++++++ MAINTAINERS | 1 + drivers/vdpa/Kconfig | 7 ++ 3 files changed, 93 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst
Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst new file +85 -0
@@ -0,0 +1,85 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +========================================================== +PCI vDPA driver for the Pensando(R) DSC adapter family +========================================================== + +Pensando vDPA VF Device Driver +Copyright(c) 2022 Pensando Systems, Inc + +Overview +======== + +The ``pds_vdpa`` driver is a PCI and auxiliary bus driver and supplies +a vDPA device for use by the virtio network stack. It is used with +the Pensando Virtual Function devices that offer vDPA and virtio queue +services. It depends on the ``pds_core`` driver and hardware for the PF +and for device configuration services. + +Using the device +================ + +The ``pds_vdpa`` device is enabled via multiple configuration steps and +depends on the ``pds_core`` driver to create and enable SR-IOV Virtual +Function devices. + +Shown below are the steps to bind the driver to a VF and also to the +associated auxiliary device created by the ``pds_core`` driver. This +example assumes the pds_core and pds_vdpa modules are already +loaded. + +.. code-block:: bash + + #!/bin/bash + + modprobe pds_core + modprobe pds_vdpa + + PF_BDF=`grep "vDPA.*1" /sys/kernel/debug/pds_core/*/viftypes | head -1 | awk -F / '{print $6}'` + + # Enable vDPA VF auxiliary device(s) in the PF + devlink dev param set pci/$PF_BDF name enable_vnet value true cmode runtime + + # Create a VF for vDPA use + echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs + + # Find the vDPA services/devices available + PDS_VDPA_MGMT=`vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1` + + # Create a vDPA device for use in virtio network configurations + vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55 + + # Set up an ethernet interface on the vdpa device + modprobe virtio_vdpa + + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> Pensando devices + -> Pensando Ethernet PDS_VDPA Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]
MAINTAINERS +1 -0
@@ -16152,6 +16152,7 @@ L: [email protected] S: Supported F: Documentation/networking/device_drivers/ethernet/pensando/ F: drivers/net/ethernet/pensando/ +F: drivers/vdpa/pds/ F: include/linux/pds/ PER-CPU MEMORY ALLOCATOR
drivers/vdpa/Kconfig +7 -0
@@ -86,4 +86,11 @@ config ALIBABA_ENI_VDPA VDPA driver for Alibaba ENI (Elastic Network Interface) which is built upon virtio 0.9.5 specification. +config PDS_VDPA + tristate "vDPA driver for Pensando DSC devices" + select VHOST_RING + depends on PDS_CORE + help + VDPA network driver for Pensando's PDS Core devices. + endif # VDPA
Signature · 1 line
2.17.1
Re: [RFC PATCH net-next 19/19] pds_vdpa: add Kconfig entry and pds_vdpa.rst
Quoted 117 lines · Shannon Nelson
Signed-off-by Shannon Nelson <[email protected]>--- .../ethernet/pensando/pds_vdpa.rst | 85 +++++++++++++++++++ MAINTAINERS | 1 + drivers/vdpa/Kconfig | 7 ++ 3 files changed, 93 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rstDocumentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst new file +85
-0@@ -0,0 +1,85 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +========================================================== +PCI vDPA driver for the Pensando(R) DSC adapter family +========================================================== + +Pensando vDPA VF Device Driver +Copyright(c) 2022 Pensando Systems, Inc + +Overview +======== + +The ``pds_vdpa`` driver is a PCI and auxiliary bus driver and supplies +a vDPA device for use by the virtio network stack. It is used with +the Pensando Virtual Function devices that offer vDPA and virtio queue +services. It depends on the ``pds_core`` driver and hardware for the PF +and for device configuration services. + +Using the device +================ + +The ``pds_vdpa`` device is enabled via multiple configuration steps and +depends on the ``pds_core`` driver to create and enable SR-IOV Virtual +Function devices. + +Shown below are the steps to bind the driver to a VF and also to the +associated auxiliary device created by the ``pds_core`` driver. This +example assumes the pds_core and pds_vdpa modules are already +loaded. + +.. code-block:: bash + + #!/bin/bash + + modprobe pds_core + modprobe pds_vdpa + + PF_BDF=`grep "vDPA.*1" /sys/kernel/debug/pds_core/*/viftypes | head -1 | awk -F / '{print $6}'` + + # Enable vDPA VF auxiliary device(s) in the PF + devlink dev param set pci/$PF_BDF name enable_vnet value true cmode runtime + + # Create a VF for vDPA use + echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs + + # Find the vDPA services/devices available + PDS_VDPA_MGMT=`vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1` + + # Create a vDPA device for use in virtio network configurations + vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55 + + # Set up an ethernet interface on the vdpa device + modprobe virtio_vdpa + + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> Pensando devices + -> Pensando Ethernet PDS_VDPA Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]MAINTAINERS +1
-0@@ -16152,6 +16152,7 @@ L: [email protected] S: Supported F: Documentation/networking/device_drivers/ethernet/pensando/ F: drivers/net/ethernet/pensando/ +F: drivers/vdpa/pds/ F: include/linux/pds/ PER-CPU MEMORY ALLOCATORdrivers/vdpa/Kconfig +0
-0
drivers/vdpa/Kconfig config ALIBABA_ENI_VDPA · line 91
Quoted 6 lines
drivers/vdpa/Kconfig +3
-0@@ -86,4 +86,11 @@ config ALIBABA_ENI_VDPA VDPA driver for Alibaba ENI (Elastic Network Interface) which is built upon virtio 0.9.5 specification. +config PDS_VDPA + tristate "vDPA driver for Pensando DSC devices" + select VHOST_RING
Any reason it needs to select on vringh?
Thanks
Quoted 8 lines
drivers/vdpa/Kconfig +4
-1+ depends on PDS_CORE + help + VDPA network driver for Pensando's PDS Core devices. + endif # VDPA --2.17.1
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH net-next 19/19] pds_vdpa: add Kconfig entry and pds_vdpa.rst
Quoted 122 lines · Jason Wang
Quoted 120 lines · Shannon Nelson
Signed-off-by Shannon Nelson <[email protected]>--- .../ethernet/pensando/pds_vdpa.rst | 85 +++++++++++++++++++ MAINTAINERS | 1 + drivers/vdpa/Kconfig | 7 ++ 3 files changed, 93 insertions(+)create mode 100644
Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rstDocumentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst new file +85
-0@@ -0,0 +1,85 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +========================================================== +PCI vDPA driver for the Pensando(R) DSC adapter family +========================================================== + +Pensando vDPA VF Device Driver +Copyright(c) 2022 Pensando Systems, Inc + +Overview +======== + +The ``pds_vdpa`` driver is a PCI and auxiliary bus driver and supplies +a vDPA device for use by the virtio network stack. It is used with +the Pensando Virtual Function devices that offer vDPA and virtio queue +services. It depends on the ``pds_core`` driver and hardware for the PF +and for device configuration services. + +Using the device +================ + +The ``pds_vdpa`` device is enabled via multiple configuration steps and +depends on the ``pds_core`` driver to create and enable SR-IOV Virtual +Function devices. + +Shown below are the steps to bind the driver to a VF and also to the +associated auxiliary device created by the ``pds_core`` driver. This +example assumes the pds_core and pds_vdpa modules are already +loaded. + +.. code-block:: bash + + #!/bin/bash + + modprobe pds_core + modprobe pds_vdpa + + PF_BDF=`grep "vDPA.*1" /sys/kernel/debug/pds_core/*/viftypes | head -1 | awk -F / '{print $6}'` + + # Enable vDPA VF auxiliary device(s) in the PF + devlink dev param set pci/$PF_BDF name enable_vnet value true cmode runtime + + # Create a VF for vDPA use + echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs + + # Find the vDPA services/devices available + PDS_VDPA_MGMT=`vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1` + + # Create a vDPA device for use in virtio network configurations + vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55 + + # Set up an ethernet interface on the vdpa device + modprobe virtio_vdpa + + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> Pensando devices + -> Pensando Ethernet PDS_VDPA Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]MAINTAINERS +1
-0@@ -16152,6 +16152,7 @@ L: [email protected] S: Supported F: Documentation/networking/device_drivers/ethernet/pensando/ F: drivers/net/ethernet/pensando/ +F: drivers/vdpa/pds/ F: include/linux/pds/ PER-CPU MEMORY ALLOCATORdrivers/vdpa/Kconfig +0
-0@@ -86,4 +86,11 @@ config ALIBABA_ENI_VDPA VDPA driver for Alibaba ENI (Elastic Network Interface) which is built upon virtio 0.9.5 specification.
Quoted 8 lines
drivers/vdpa/Kconfig config ALIBABA_ENI_VDPA · line 91
Quoted 3 lines
drivers/vdpa/Kconfig +3
-0+config PDS_VDPA + tristate "vDPA driver for Pensando DSC devices" + select VHOST_RINGAny reason it needs to select on vringh?
Thanks
Hi Jason,
Thanks for your comments, I appreciate the time. I'll be able to respond to them more fully next week when I'm back from the holidays.
sln
Re: [RFC PATCH net-next 19/19] pds_vdpa: add Kconfig entry and pds_vdpa.rst
In reply to Jason WangQuoted 121 lines · Jason Wang
Quoted 120 lines · Shannon Nelson
Signed-off-by Shannon Nelson <[email protected]>--- .../ethernet/pensando/pds_vdpa.rst | 85 +++++++++++++++++++ MAINTAINERS | 1 + drivers/vdpa/Kconfig | 7 ++ 3 files changed, 93 insertions(+)create mode 100644
Documentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rstDocumentation/networking/device_drivers/ethernet/pensando/pds_vdpa.rst new file +85
-0@@ -0,0 +1,85 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +========================================================== +PCI vDPA driver for the Pensando(R) DSC adapter family +========================================================== + +Pensando vDPA VF Device Driver +Copyright(c) 2022 Pensando Systems, Inc + +Overview +======== + +The ``pds_vdpa`` driver is a PCI and auxiliary bus driver and supplies +a vDPA device for use by the virtio network stack. It is used with +the Pensando Virtual Function devices that offer vDPA and virtio queue +services. It depends on the ``pds_core`` driver and hardware for the PF +and for device configuration services. + +Using the device +================ + +The ``pds_vdpa`` device is enabled via multiple configuration steps and +depends on the ``pds_core`` driver to create and enable SR-IOV Virtual +Function devices. + +Shown below are the steps to bind the driver to a VF and also to the +associated auxiliary device created by the ``pds_core`` driver. This +example assumes the pds_core and pds_vdpa modules are already +loaded. + +.. code-block:: bash + + #!/bin/bash + + modprobe pds_core + modprobe pds_vdpa + + PF_BDF=`grep "vDPA.*1" /sys/kernel/debug/pds_core/*/viftypes | head -1 | awk -F / '{print $6}'` + + # Enable vDPA VF auxiliary device(s) in the PF + devlink dev param set pci/$PF_BDF name enable_vnet value true cmode runtime + + # Create a VF for vDPA use + echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs + + # Find the vDPA services/devices available + PDS_VDPA_MGMT=`vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1` + + # Create a vDPA device for use in virtio network configurations + vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55 + + # Set up an ethernet interface on the vdpa device + modprobe virtio_vdpa + + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> Pensando devices + -> Pensando Ethernet PDS_VDPA Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]MAINTAINERS +1
-0@@ -16152,6 +16152,7 @@ L: [email protected] S: Supported F: Documentation/networking/device_drivers/ethernet/pensando/ F: drivers/net/ethernet/pensando/ +F: drivers/vdpa/pds/ F: include/linux/pds/ PER-CPU MEMORY ALLOCATORdrivers/vdpa/Kconfig +0
-0@@ -86,4 +86,11 @@ config ALIBABA_ENI_VDPA VDPA driver for Alibaba ENI (Elastic Network Interface) which is built upon virtio 0.9.5 specification.
Quoted 5 lines
drivers/vdpa/Kconfig config ALIBABA_ENI_VDPA · line 91
Quoted 3 lines
drivers/vdpa/Kconfig +3
-0+config PDS_VDPA + tristate "vDPA driver for Pensando DSC devices" + select VHOST_RINGAny reason it needs to select on vringh?
Copied from an example... maybe not needed.
sln
Quoted 11 lines
Thanks
Quoted 8 lines
drivers/vdpa/Kconfig +4
-1+ depends on PDS_CORE + help + VDPA network driver for Pensando's PDS Core devices. + endif # VDPA --2.17.1