Skip to contentSkip to conversation

pds_vdpa driver

PATCH Superseded Shannon Nelson 4 participants 25 messages Apr 25, 2023 – May 3, 2023

Subsystems (6)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover pds_vdpa driver

This patchset implements a new module for the AMD/Pensando DSC that supports vDPA services on PDS Core VF devices. This code is based on and depends on include files from the pds_core driver described here[0]. The pds_core driver creates the auxiliary_bus devices that this module connects to, and this creates vdpa devices for use by the vdpa module.

The first version of this driver was a part of the original pds_core RFC
[1] but has since been reworked to pull out the PCI driver and to make better use of the virtio and virtio_net configuration spaces made available by the DSC's PCI configuration. As the device development has progressed, the ability to rely on the virtio config spaces has grown.

This patchset includes a modification to the existing vp_modern_probe() which implements overrides for the PCI device id check and the DMA mask. These are intended to be used with vendor vDPA devices that implement enough of the virtio config space to be used directly, but don't use the virtio device id.

To use this module, enable the VFs and turn on the vDPA services in the pds_core PF, then use the 'vdpa' utility to create devices for use by virtio_vdpa or vhost_vdpa:
echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs devlink dev param set pci/$PF_BDF name enable_vnet value true cmode runtime PDS_VDPA_MGMT=vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1 vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55

[0] Link: https://lore.kernel.org/netdev/[email protected]/
[1] Link: https://lore.kernel.org/netdev/[email protected]/

Changes:
v4:
- rename device_id_check_override() to device_id_check()
- make device_id_check() return the device_id found and checked
- removed pds_vdpa.h, put its adminq changes into pds_adminq.h
- added a patch to separate out the adminq changes
- added a patch to move an adminq enum from pds_common.h to pds_adminq.h
- moved adminq calls for get/set_vq_state into cmds.c
- limit max_vqs by number of msix available
- don't increment nintrs for CVQ, it should already be covered from max_vqs
- pds_core API related rework following pds_core inclusion to net-next
- use non-debugfs method to find PF pci address in pds_vdpa.rst instructions

v3:

- added a patch to modify vp_modern_probe() such that specific device id and DMA mask overrides can be used
- add pds_vdpa.rst into index file
- dev_dbg instead of dev_err on most of the adminq commands
- rework use of pds_vdpa_cmd_reset() and pds_vdpa_init_hw() for better firmware setup in start-stop-start scenarios
- removed unused pds_vdpa_cmd_set_features(), we can rely on vp_modern_set_features()
- remove unused hw_qtype and hw_qindex from pds_vdpa_vq_info
- reworked debugfs print_feature_bits to also print unknown bits
- changed use of PAGE_SIZE to local PDS_PAGE_SIZE to keep with FW layout needs without regard to kernel PAGE_SIZE configuration

v2:
https://lore.kernel.org/netdev/[email protected]/
- removed PCI driver code
- replaced home-grown event listener with notifier
- replaced many adminq uses with direct virtio_net config access
- reworked irqs to follow virtio layout
- removed local_mac_bit logic
- replaced uses of devm_ interfaces as suggested in pds_core reviews
- updated copyright strings to reflect the new owner

Shannon Nelson (10):
virtio: allow caller to override device id and DMA mask
pds_vdpa: Add new vDPA driver for AMD/Pensando DSC
pds_vdpa: move enum from common to adminq header
pds_vdpa: new adminq entries
pds_vdpa: get vdpa management info
pds_vdpa: virtio bar setup for vdpa
pds_vdpa: add vdpa config client commands
pds_vdpa: add support for vdpa and vdpamgmt interfaces
pds_vdpa: subscribe to the pds_core events
pds_vdpa: pds_vdps.rst and Kconfig

 .../device_drivers/ethernet/amd/pds_vdpa.rst  |  85 +++
 .../device_drivers/ethernet/index.rst         |   1 +
 MAINTAINERS                                   |   4 +
 drivers/vdpa/Kconfig                          |   8 +
 drivers/vdpa/Makefile                         |   1 +
 drivers/vdpa/pds/Makefile                     |  10 +
 drivers/vdpa/pds/aux_drv.c                    | 140 ++++
 drivers/vdpa/pds/aux_drv.h                    |  26 +
 drivers/vdpa/pds/cmds.c                       | 207 +++++
 drivers/vdpa/pds/cmds.h                       |  20 +
 drivers/vdpa/pds/debugfs.c                    | 287 +++++++
 drivers/vdpa/pds/debugfs.h                    |  17 +
 drivers/vdpa/pds/vdpa_dev.c                   | 704 ++++++++++++++++++
 drivers/vdpa/pds/vdpa_dev.h                   |  47 ++
 drivers/virtio/virtio_pci_modern_dev.c        |  37 +-
 include/linux/pds/pds_adminq.h                | 287 +++++++
 include/linux/pds/pds_common.h                |  21 +-
 include/linux/virtio_pci_modern.h             |   6 +
 18 files changed, 1876 insertions(+), 32 deletions(-)
 create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vdpa.rst
 create mode 100644 drivers/vdpa/pds/Makefile
 create mode 100644 drivers/vdpa/pds/aux_drv.c
 create mode 100644 drivers/vdpa/pds/aux_drv.h
 create mode 100644 drivers/vdpa/pds/cmds.c
 create mode 100644 drivers/vdpa/pds/cmds.h
 create mode 100644 drivers/vdpa/pds/debugfs.c
 create mode 100644 drivers/vdpa/pds/debugfs.h
 create mode 100644 drivers/vdpa/pds/vdpa_dev.c
 create mode 100644 drivers/vdpa/pds/vdpa_dev.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v4 1/10 virtio: allow caller to override device id and DMA mask

To add a bit of flexibility with various virtio based devices, allow the caller to specify a different device id and DMA mask. This adds fields to struct virtio_pci_modern_device to specify an override device id check and a DMA mask.

int (*device_id_check)(struct pci_dev *pdev);
If defined by the driver, this function will be called to check that the PCI device is the vendor's expected device, and will return the found device id to be stored in mdev->id.device. This allows vendors with alternative vendor device ids to use this library on their own device BAR.

u64 dma_mask;
If defined by the driver, this mask will be used in a call to dma_set_mask_and_coherent() instead of the traditional DMA_BIT_MASK(64). This allows limiting the DMA space on vendor devices with address limitations.

---
 drivers/virtio/virtio_pci_modern_dev.c | 37 +++++++++++++++++---------
 include/linux/virtio_pci_modern.h      |  6 +++++
 2 files changed, 31 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_modern_dev.c +25 -12
index 869cb46bef96..1f2db76e8f91 100644
@@ -218,21 +218,29 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 	int err, common, isr, notify, device;
 	u32 notify_length;
 	u32 notify_offset;
+	int devid;
 
 	check_offsets();
 
-	/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
-	if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
-		return -ENODEV;
-
-	if (pci_dev->device < 0x1040) {
-		/* Transitional devices: use the PCI subsystem device id as
-		 * virtio device id, same as legacy driver always did.
-		 */
-		mdev->id.device = pci_dev->subsystem_device;
+	if (mdev->device_id_check) {
+		devid = mdev->device_id_check(pci_dev);
+		if (devid < 0)
+			return devid;
+		mdev->id.device = devid;
 	} else {
-		/* Modern devices: simply use PCI device id, but start from 0x1040. */
-		mdev->id.device = pci_dev->device - 0x1040;
+		/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+		if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+			return -ENODEV;
+
+		if (pci_dev->device < 0x1040) {
+			/* Transitional devices: use the PCI subsystem device id as
+			 * virtio device id, same as legacy driver always did.
+			 */
+			mdev->id.device = pci_dev->subsystem_device;
+		} else {
+			/* Modern devices: simply use PCI device id, but start from 0x1040. */
+			mdev->id.device = pci_dev->device - 0x1040;
+		}
 	}
 	mdev->id.vendor = pci_dev->subsystem_vendor;
 
@@ -260,7 +268,12 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 		return -EINVAL;
 	}
 
-	err = dma_set_mask_and_coherent(&pci_dev->dev, DMA_BIT_MASK(64));
+	if (mdev->dma_mask)
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						mdev->dma_mask);
+	else
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						DMA_BIT_MASK(64));
 	if (err)
 		err = dma_set_mask_and_coherent(&pci_dev->dev,
 						DMA_BIT_MASK(32));
include/linux/virtio_pci_modern.h +6 -0
index c4eeb79b0139..067ac1d789bc 100644
@@ -38,6 +38,12 @@ struct virtio_pci_modern_device {
 	int modern_bars;
 
 	struct virtio_device_id id;
+
+	/* optional check for vendor virtio device, returns dev_id or -ERRNO */
+	int (*device_id_check)(struct pci_dev *pdev);
+
+	/* optional mask for devices with limited DMA space */
+	u64 dma_mask;
 };
 
 /*
Signature · 1 line
2.17.1
reviewer
#2 lore raw

Re: [PATCH v4 virtio 01/10] virtio: allow caller to override device id and DMA mask

Quoted 39 lines · Shannon Nelson

To add a bit of flexibility with various virtio based devices, allow the caller to specify a different device id and DMA mask. This adds fields to struct virtio_pci_modern_device to specify an override device id check and a DMA mask.

int (*device_id_check)(struct pci_dev *pdev);
If defined by the driver, this function will be called to check that the PCI device is the vendor's expected device, and will return the found device id to be stored in mdev->id.device. This allows vendors with alternative vendor device ids to use this library on their own device BAR.

u64 dma_mask;
If defined by the driver, this mask will be used in a call to dma_set_mask_and_coherent() instead of the traditional DMA_BIT_MASK(64). This allows limiting the DMA space on vendor devices with address limitations.

---
 drivers/virtio/virtio_pci_modern_dev.c | 37 +++++++++++++++++---------
 include/linux/virtio_pci_modern.h      |  6 +++++
 2 files changed, 31 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_modern_dev.c +1 -3
index 869cb46bef96..1f2db76e8f91 100644
@@ -218,21 +218,29 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 	int err, common, isr, notify, device;
 	u32 notify_length;
 	u32 notify_offset;
+	int devid;
 
 	check_offsets();
 
-	/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
-	if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
-		return -ENODEV;

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 228

Quoted 10 lines
drivers/virtio/virtio_pci_modern_dev.c +4 -6
-
-	if (pci_dev->device < 0x1040) {
-		/* Transitional devices: use the PCI subsystem device id as
-		 * virtio device id, same as legacy driver always did.
-		 */
-		mdev->id.device = pci_dev->subsystem_device;
+	if (mdev->device_id_check) {
+		devid = mdev->device_id_check(pci_dev);
+		if (devid < 0)
+			return devid;

I would want to know is there any other reason to return the errno? How about return -ENODEV directly?

Thanks.

Quoted 54 lines
drivers/virtio/virtio_pci_modern_dev.c +20 -3
+		mdev->id.device = devid;
 	} else {
-		/* Modern devices: simply use PCI device id, but start from 0x1040. */
-		mdev->id.device = pci_dev->device - 0x1040;
+		/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+		if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+			return -ENODEV;
+
+		if (pci_dev->device < 0x1040) {
+			/* Transitional devices: use the PCI subsystem device id as
+			 * virtio device id, same as legacy driver always did.
+			 */
+			mdev->id.device = pci_dev->subsystem_device;
+		} else {
+			/* Modern devices: simply use PCI device id, but start from 0x1040. */
+			mdev->id.device = pci_dev->device - 0x1040;
+		}
 	}
 	mdev->id.vendor = pci_dev->subsystem_vendor;
 
@@ -260,7 +268,12 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 		return -EINVAL;
 	}
 
-	err = dma_set_mask_and_coherent(&pci_dev->dev, DMA_BIT_MASK(64));
+	if (mdev->dma_mask)
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						mdev->dma_mask);
+	else
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						DMA_BIT_MASK(64));
 	if (err)
 		err = dma_set_mask_and_coherent(&pci_dev->dev,
 						DMA_BIT_MASK(32));
include/linux/virtio_pci_modern.h +6 -1
index c4eeb79b0139..067ac1d789bc 100644
@@ -38,6 +38,12 @@ struct virtio_pci_modern_device {
 	int modern_bars;
 
 	struct virtio_device_id id;
+
+	/* optional check for vendor virtio device, returns dev_id or -ERRNO */
+	int (*device_id_check)(struct pci_dev *pdev);
+
+	/* optional mask for devices with limited DMA space */
+	u64 dma_mask;
 };
 
 /*
--

2.17.1

#3 lore raw

Re: [PATCH v4 virtio 01/10] virtio: allow caller to override device id and DMA mask

Quoted 38 lines · Xuan Zhuo
Quoted 36 lines · Shannon Nelson

To add a bit of flexibility with various virtio based devices, allow the caller to specify a different device id and DMA mask. This adds fields to struct virtio_pci_modern_device to specify an override device id check and a DMA mask.

int (*device_id_check)(struct pci_dev *pdev);
If defined by the driver, this function will be called to check that the PCI device is the vendor's expected device, and will return the found device id to be stored in mdev->id.device. This allows vendors with alternative vendor device ids to use this library on their own device BAR.

u64 dma_mask;
If defined by the driver, this mask will be used in a call to dma_set_mask_and_coherent() instead of the traditional DMA_BIT_MASK(64). This allows limiting the DMA space on vendor devices with address limitations.

---
  drivers/virtio/virtio_pci_modern_dev.c | 37 +++++++++++++++++---------
  include/linux/virtio_pci_modern.h      |  6 +++++
  2 files changed, 31 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_modern_dev.c +1 -0
index 869cb46bef96..1f2db76e8f91 100644
@@ -218,21 +218,29 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
       int err, common, isr, notify, device;
       u32 notify_length;
       u32 notify_offset;
+     int devid;
 
       check_offsets();
 
Quoted 16 lines

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 228

Quoted 13 lines
drivers/virtio/virtio_pci_modern_dev.c +4 -9
-     /* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
-     if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
-             return -ENODEV;
-
-     if (pci_dev->device < 0x1040) {
-             /* Transitional devices: use the PCI subsystem device id as
-              * virtio device id, same as legacy driver always did.
-              */
-             mdev->id.device = pci_dev->subsystem_device;
+     if (mdev->device_id_check) {
+             devid = mdev->device_id_check(pci_dev);
+             if (devid < 0)
+                     return devid;

I would want to know is there any other reason to return the errno? How about return -ENODEV directly?

Because if device_id_check() is returning an errno, it is trying to communicate some information about what went wrong, and I really get annoyed when an intermediate layer stomps on the value and makes that potentially useful information disappear.

sln

Quoted 58 lines

Thanks.

Quoted 54 lines
drivers/virtio/virtio_pci_modern_dev.c +20 -3
+             mdev->id.device = devid;
       } else {
-             /* Modern devices: simply use PCI device id, but start from 0x1040. */
-             mdev->id.device = pci_dev->device - 0x1040;
+             /* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+             if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+                     return -ENODEV;
+
+             if (pci_dev->device < 0x1040) {
+                     /* Transitional devices: use the PCI subsystem device id as
+                      * virtio device id, same as legacy driver always did.
+                      */
+                     mdev->id.device = pci_dev->subsystem_device;
+             } else {
+                     /* Modern devices: simply use PCI device id, but start from 0x1040. */
+                     mdev->id.device = pci_dev->device - 0x1040;
+             }
       }
       mdev->id.vendor = pci_dev->subsystem_vendor;
 
@@ -260,7 +268,12 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
               return -EINVAL;
       }
 
-     err = dma_set_mask_and_coherent(&pci_dev->dev, DMA_BIT_MASK(64));
+     if (mdev->dma_mask)
+             err = dma_set_mask_and_coherent(&pci_dev->dev,
+                                             mdev->dma_mask);
+     else
+             err = dma_set_mask_and_coherent(&pci_dev->dev,
+                                             DMA_BIT_MASK(64));
       if (err)
               err = dma_set_mask_and_coherent(&pci_dev->dev,
                                               DMA_BIT_MASK(32));
include/linux/virtio_pci_modern.h +6 -1
index c4eeb79b0139..067ac1d789bc 100644
@@ -38,6 +38,12 @@ struct virtio_pci_modern_device {
       int modern_bars;
 
       struct virtio_device_id id;
+
+     /* optional check for vendor virtio device, returns dev_id or -ERRNO */
+     int (*device_id_check)(struct pci_dev *pdev);
+
+     /* optional mask for devices with limited DMA space */
+     u64 dma_mask;
  };
 
  /*
--

2.17.1

#4 lore raw

Re: [PATCH v4 virtio 01/10] virtio: allow caller to override device id and DMA mask

In reply to Shannon Nelson
Quoted 12 lines · Shannon Nelson

To add a bit of flexibility with various virtio based devices, allow the caller to specify a different device id and DMA mask. This adds fields to struct virtio_pci_modern_device to specify an override device id check and a DMA mask.

int (*device_id_check)(struct pci_dev *pdev);
If defined by the driver, this function will be called to check that the PCI device is the vendor's expected device, and will return the found device id to be stored in mdev->id.device. This allows vendors with alternative vendor device ids to use this library on their own device BAR.

Quoted 5 lines

u64 dma_mask;
If defined by the driver, this mask will be used in a call to dma_set_mask_and_coherent() instead of the traditional DMA_BIT_MASK(64). This allows limiting the DMA space on vendor devices with address limitations.

Hi Shannon,

I don't feel strongly about this, but as there are two new features, perhaps it would appropriate to have two patches.

Quoted 41 lines
---
 drivers/virtio/virtio_pci_modern_dev.c | 37 +++++++++++++++++---------
 include/linux/virtio_pci_modern.h      |  6 +++++
 2 files changed, 31 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_modern_dev.c +12 -11
index 869cb46bef96..1f2db76e8f91 100644
@@ -218,21 +218,29 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 	int err, common, isr, notify, device;
 	u32 notify_length;
 	u32 notify_offset;
+	int devid;
 
 	check_offsets();
 
-	/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
-	if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
-		return -ENODEV;
-
-	if (pci_dev->device < 0x1040) {
-		/* Transitional devices: use the PCI subsystem device id as
-		 * virtio device id, same as legacy driver always did.
-		 */
-		mdev->id.device = pci_dev->subsystem_device;
+	if (mdev->device_id_check) {
+		devid = mdev->device_id_check(pci_dev);
+		if (devid < 0)
+			return devid;
+		mdev->id.device = devid;
 	} else {
-		/* Modern devices: simply use PCI device id, but start from 0x1040. */
-		mdev->id.device = pci_dev->device - 0x1040;
+		/* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+		if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+			return -ENODEV;
+
+		if (pci_dev->device < 0x1040) {
+			/* Transitional devices: use the PCI subsystem device id as

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 246

Quoted 10 lines
drivers/virtio/virtio_pci_modern_dev.c +7 -0
+			 * virtio device id, same as legacy driver always did.
+			 */
+			mdev->id.device = pci_dev->subsystem_device;
+		} else {
+			/* Modern devices: simply use PCI device id, but start from 0x1040. */
+			mdev->id.device = pci_dev->device - 0x1040;
+		}
 	}
 	mdev->id.vendor = pci_dev->subsystem_vendor;
 

The diff above is verbose, but looks good to me :)

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 276

Quoted 11 lines
drivers/virtio/virtio_pci_modern_dev.c +6 -1
@@ -260,7 +268,12 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 		return -EINVAL;
 	}
 
-	err = dma_set_mask_and_coherent(&pci_dev->dev, DMA_BIT_MASK(64));
+	if (mdev->dma_mask)
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						mdev->dma_mask);
+	else
+		err = dma_set_mask_and_coherent(&pci_dev->dev,
+						DMA_BIT_MASK(64));

Maybe it is nicer to avoid duplicating the function call, something like this:

	u64 dma_mask;
	...

	dma_mask = mdev->dma_mask ? : DMA_BIT_MASK(64);
	err = dma_set_mask_and_coherent(&pci_dev->dev, dma_mask);

or, without a local variable.

	err = dma_set_mask_and_coherent(&pci_dev->dev,
					mdev->dma_mask ? : DMA_BIT_MASK(64));

include/linux/virtio_pci_modern.h

Quoted 7 lines
drivers/virtio/virtio_pci_modern_dev.c +0 -0
 	if (err)
 		err = dma_set_mask_and_coherent(&pci_dev->dev,
 						DMA_BIT_MASK(32));
include/linux/virtio_pci_modern.h +0 -0
index c4eeb79b0139..067ac1d789bc 100644

Maybe it would be good to add kdoc for struct virtio_pci_modern_device at some point.

Quoted 16 lines
include/linux/virtio_pci_modern.h +6 -0
@@ -38,6 +38,12 @@ struct virtio_pci_modern_device {
 	int modern_bars;
 
 	struct virtio_device_id id;
+
+	/* optional check for vendor virtio device, returns dev_id or -ERRNO */
+	int (*device_id_check)(struct pci_dev *pdev);
+
+	/* optional mask for devices with limited DMA space */
+	u64 dma_mask;
 };
 
 /*
Signature · 1 line
2.17.1
#5 lore raw

Re: [PATCH v4 virtio 01/10] virtio: allow caller to override device id and DMA mask

Quoted 21 lines · Simon Horman
Quoted 12 lines · Shannon Nelson

To add a bit of flexibility with various virtio based devices, allow the caller to specify a different device id and DMA mask. This adds fields to struct virtio_pci_modern_device to specify an override device id check and a DMA mask.

int (*device_id_check)(struct pci_dev *pdev);
If defined by the driver, this function will be called to check that the PCI device is the vendor's expected device, and will return the found device id to be stored in mdev->id.device. This allows vendors with alternative vendor device ids to use this library on their own device BAR.

Quoted 5 lines

u64 dma_mask;
If defined by the driver, this mask will be used in a call to dma_set_mask_and_coherent() instead of the traditional DMA_BIT_MASK(64). This allows limiting the DMA space on vendor devices with address limitations.

Quoted 4 lines

Hi Shannon,

I don't feel strongly about this, but as there are two new features, perhaps it would appropriate to have two patches.

Sure, I can respin and split these out to separate patches, and I'll keep your verbosity notes below in mind :-). Yes, the kdoc would be a good thing, but I'd like to keep the mission-creep to a minimum and come back to that one separately.

sln

Quoted 104 lines
Quoted 42 lines
---
  drivers/virtio/virtio_pci_modern_dev.c | 37 +++++++++++++++++---------
  include/linux/virtio_pci_modern.h      |  6 +++++
  2 files changed, 31 insertions(+), 12 deletions(-)
drivers/virtio/virtio_pci_modern_dev.c +12 -11
index 869cb46bef96..1f2db76e8f91 100644
@@ -218,21 +218,29 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
       int err, common, isr, notify, device;
       u32 notify_length;
       u32 notify_offset;
+     int devid;
 
       check_offsets();
 
-     /* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
-     if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
-             return -ENODEV;
-
-     if (pci_dev->device < 0x1040) {
-             /* Transitional devices: use the PCI subsystem device id as
-              * virtio device id, same as legacy driver always did.
-              */
-             mdev->id.device = pci_dev->subsystem_device;
+     if (mdev->device_id_check) {
+             devid = mdev->device_id_check(pci_dev);
+             if (devid < 0)
+                     return devid;
+             mdev->id.device = devid;
       } else {
-             /* Modern devices: simply use PCI device id, but start from 0x1040. */
-             mdev->id.device = pci_dev->device - 0x1040;
+             /* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+             if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+                     return -ENODEV;
+
+             if (pci_dev->device < 0x1040) {
+                     /* Transitional devices: use the PCI subsystem device id as

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 246

Quoted 10 lines
drivers/virtio/virtio_pci_modern_dev.c +7 -0
+                      * virtio device id, same as legacy driver always did.
+                      */
+                     mdev->id.device = pci_dev->subsystem_device;
+             } else {
+                     /* Modern devices: simply use PCI device id, but start from 0x1040. */
+                     mdev->id.device = pci_dev->device - 0x1040;
+             }
       }
       mdev->id.vendor = pci_dev->subsystem_vendor;
 

The diff above is verbose, but looks good to me :)

drivers/virtio/virtio_pci_modern_dev.c int vp_modern_probe(struct virtio_pci_modern_device *mdev) · line 276

Quoted 11 lines
drivers/virtio/virtio_pci_modern_dev.c +6 -1
@@ -260,7 +268,12 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
               return -EINVAL;
       }
 
-     err = dma_set_mask_and_coherent(&pci_dev->dev, DMA_BIT_MASK(64));
+     if (mdev->dma_mask)
+             err = dma_set_mask_and_coherent(&pci_dev->dev,
+                                             mdev->dma_mask);
+     else
+             err = dma_set_mask_and_coherent(&pci_dev->dev,
+                                             DMA_BIT_MASK(64));

Maybe it is nicer to avoid duplicating the function call, something like this:

         u64 dma_mask;
         ...

         dma_mask = mdev->dma_mask ? : DMA_BIT_MASK(64);
         err = dma_set_mask_and_coherent(&pci_dev->dev, dma_mask);

or, without a local variable.

         err = dma_set_mask_and_coherent(&pci_dev->dev,
                                         mdev->dma_mask ? : DMA_BIT_MASK(64));

include/linux/virtio_pci_modern.h

Quoted 7 lines
drivers/virtio/virtio_pci_modern_dev.c +0 -0
       if (err)
               err = dma_set_mask_and_coherent(&pci_dev->dev,
                                               DMA_BIT_MASK(32));
include/linux/virtio_pci_modern.h +0 -0
index c4eeb79b0139..067ac1d789bc 100644

Maybe it would be good to add kdoc for struct virtio_pci_modern_device at some point.

Quoted 12 lines
include/linux/virtio_pci_modern.h +6 -0
@@ -38,6 +38,12 @@ struct virtio_pci_modern_device {
       int modern_bars;
 
       struct virtio_device_id id;
+
+     /* optional check for vendor virtio device, returns dev_id or -ERRNO */
+     int (*device_id_check)(struct pci_dev *pdev);
+
+     /* optional mask for devices with limited DMA space */
+     u64 dma_mask;
  };
 
Quoted 4 lines
Quoted 4 lines
include/linux/virtio_pci_modern.h +0 -1
  /*
--

2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#6 lore raw

PATCH v4 2/10 pds_vdpa: Add new vDPA driver for AMD/Pensando DSC

In reply to Shannon Nelson

This is the initial auxiliary driver framework for a new vDPA device driver, an auxiliary_bus client of the pds_core driver. The pds_core driver supplies the PCI services for the VF device and for accessing the adminq in the PF device.

This patch adds the very basics of registering for the auxiliary device and setting up debugfs entries.

---
 drivers/vdpa/Makefile          |  1 +
 drivers/vdpa/pds/Makefile      |  8 ++++
 drivers/vdpa/pds/aux_drv.c     | 83 ++++++++++++++++++++++++++++++++++
 drivers/vdpa/pds/aux_drv.h     | 15 ++++++
 drivers/vdpa/pds/debugfs.c     | 25 ++++++++++
 drivers/vdpa/pds/debugfs.h     | 12 +++++
 include/linux/pds/pds_common.h |  2 +
 7 files changed, 146 insertions(+)
 create mode 100644 drivers/vdpa/pds/Makefile
 create mode 100644 drivers/vdpa/pds/aux_drv.c
 create mode 100644 drivers/vdpa/pds/aux_drv.h
 create mode 100644 drivers/vdpa/pds/debugfs.c
 create mode 100644 drivers/vdpa/pds/debugfs.h
drivers/vdpa/Makefile +1 -0
index 59396ff2a318..8f53c6f3cca7 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_MLX5_VDPA) += mlx5/
 obj-$(CONFIG_VP_VDPA)    += virtio_pci/
 obj-$(CONFIG_ALIBABA_ENI_VDPA) += alibaba/
 obj-$(CONFIG_SNET_VDPA) += solidrun/
+obj-$(CONFIG_PDS_VDPA) += pds/
drivers/vdpa/pds/Makefile new file +8 -0
new file mode 100644
index 000000000000..a9cd2f450ae1
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright(c) 2023 Advanced Micro Devices, Inc
+
+obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o
+
+pds_vdpa-y := aux_drv.o
+
+pds_vdpa-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/vdpa/pds/aux_drv.c new file +83 -0

83 lines not on this page · raw message

drivers/vdpa/pds/aux_drv.h new file +15 -0
new file mode 100644
index 000000000000..f1e99359424e
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _AUX_DRV_H_
+#define _AUX_DRV_H_
+
+#define PDS_VDPA_DRV_DESCRIPTION    "AMD/Pensando vDPA VF Device Driver"
+#define PDS_VDPA_DRV_NAME           KBUILD_MODNAME
+
+struct pds_vdpa_aux {
+	struct pds_auxiliary_dev *padev;
+
+	struct dentry *dentry;
+};
+#endif /* _AUX_DRV_H_ */
drivers/vdpa/pds/debugfs.c new file +25 -0
new file mode 100644
index 000000000000..5be22fb7a76a
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include <linux/pci.h>
+
+#include <linux/pds/pds_common.h>
+#include <linux/pds/pds_core_if.h>
+#include <linux/pds/pds_adminq.h>
+#include <linux/pds/pds_auxbus.h>
+
+#include "aux_drv.h"
+#include "debugfs.h"
+
+static struct dentry *dbfs_dir;
+
+void pds_vdpa_debugfs_create(void)
+{
+	dbfs_dir = debugfs_create_dir(PDS_VDPA_DRV_NAME, NULL);
+}
+
+void pds_vdpa_debugfs_destroy(void)
+{
+	debugfs_remove_recursive(dbfs_dir);
+	dbfs_dir = NULL;
+}
drivers/vdpa/pds/debugfs.h new file +12 -0
new file mode 100644
index 000000000000..658849591a99
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _PDS_VDPA_DEBUGFS_H_
+#define _PDS_VDPA_DEBUGFS_H_
+
+#include <linux/debugfs.h>
+
+void pds_vdpa_debugfs_create(void);
+void pds_vdpa_debugfs_destroy(void);
+
+#endif /* _PDS_VDPA_DEBUGFS_H_ */
include/linux/pds/pds_common.h +2 -0
index 060331486d50..2a0d1669cfd0 100644
@@ -39,6 +39,8 @@ enum pds_core_vif_types {
 #define PDS_DEV_TYPE_RDMA_STR	"RDMA"
 #define PDS_DEV_TYPE_LM_STR	"LM"
 
+#define PDS_VDPA_DEV_NAME	PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_VDPA_STR
+
 #define PDS_CORE_IFNAMSIZ		16
 
 /**
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Reviewed-by on v4 2/10
#7 lore raw

Re: [PATCH v4 virtio 02/10] pds_vdpa: Add new vDPA driver for AMD/Pensando DSC

Quoted 10 lines · Shannon Nelson

This is the initial auxiliary driver framework for a new vDPA device driver, an auxiliary_bus client of the pds_core driver. The pds_core driver supplies the PCI services for the VF device and for accessing the adminq in the PF device.

This patch adds the very basics of registering for the auxiliary device and setting up debugfs entries.

#8 lore raw

PATCH v4 3/10 pds_vdpa: move enum from common to adminq header

In reply to Shannon Nelson

The pds_core_logical_qtype enum and IFNAMSIZ are not needed in the common PDS header, only needed when working with the adminq, so move them to the adminq header.

Note: This patch might conflict with pds_vfio patches that are
in review, depending on which patchset gets pulled first.

---
 include/linux/pds/pds_adminq.h | 21 +++++++++++++++++++++
 include/linux/pds/pds_common.h | 21 ---------------------
 2 files changed, 21 insertions(+), 21 deletions(-)
include/linux/pds/pds_adminq.h +21 -0
index 98a60ce87b92..61b0a8634e1a 100644
@@ -222,6 +222,27 @@ enum pds_core_lif_type {
 	PDS_CORE_LIF_TYPE_DEFAULT = 0,
 };
 
+#define PDS_CORE_IFNAMSIZ		16
+
+/**
+ * enum pds_core_logical_qtype - Logical Queue Types
+ * @PDS_CORE_QTYPE_ADMINQ:    Administrative Queue
+ * @PDS_CORE_QTYPE_NOTIFYQ:   Notify Queue
+ * @PDS_CORE_QTYPE_RXQ:       Receive Queue
+ * @PDS_CORE_QTYPE_TXQ:       Transmit Queue
+ * @PDS_CORE_QTYPE_EQ:        Event Queue
+ * @PDS_CORE_QTYPE_MAX:       Max queue type supported
+ */
+enum pds_core_logical_qtype {
+	PDS_CORE_QTYPE_ADMINQ  = 0,
+	PDS_CORE_QTYPE_NOTIFYQ = 1,
+	PDS_CORE_QTYPE_RXQ     = 2,
+	PDS_CORE_QTYPE_TXQ     = 3,
+	PDS_CORE_QTYPE_EQ      = 4,
+
+	PDS_CORE_QTYPE_MAX     = 16   /* don't change - used in struct size */
+};
+
 /**
  * union pds_core_lif_config - LIF configuration
  * @state:	    LIF state (enum pds_core_lif_state)
include/linux/pds/pds_common.h +0 -21
index 2a0d1669cfd0..435c8e8161c2 100644
@@ -41,27 +41,6 @@ enum pds_core_vif_types {
 
 #define PDS_VDPA_DEV_NAME	PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_VDPA_STR
 
-#define PDS_CORE_IFNAMSIZ		16
-
-/**
- * enum pds_core_logical_qtype - Logical Queue Types
- * @PDS_CORE_QTYPE_ADMINQ:    Administrative Queue
- * @PDS_CORE_QTYPE_NOTIFYQ:   Notify Queue
- * @PDS_CORE_QTYPE_RXQ:       Receive Queue
- * @PDS_CORE_QTYPE_TXQ:       Transmit Queue
- * @PDS_CORE_QTYPE_EQ:        Event Queue
- * @PDS_CORE_QTYPE_MAX:       Max queue type supported
- */
-enum pds_core_logical_qtype {
-	PDS_CORE_QTYPE_ADMINQ  = 0,
-	PDS_CORE_QTYPE_NOTIFYQ = 1,
-	PDS_CORE_QTYPE_RXQ     = 2,
-	PDS_CORE_QTYPE_TXQ     = 3,
-	PDS_CORE_QTYPE_EQ      = 4,
-
-	PDS_CORE_QTYPE_MAX     = 16   /* don't change - used in struct size */
-};
-
 int pdsc_register_notify(struct notifier_block *nb);
 void pdsc_unregister_notify(struct notifier_block *nb);
 void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
Signature · 1 line
2.17.1
reviewer Reviewed-by on v4 3/10
#9 lore raw

Re: [PATCH v4 virtio 03/10] pds_vdpa: move enum from common to adminq header

Quoted 8 lines · Shannon Nelson

The pds_core_logical_qtype enum and IFNAMSIZ are not needed in the common PDS header, only needed when working with the adminq, so move them to the adminq header.

Note: This patch might conflict with pds_vfio patches that are
in review, depending on which patchset gets pulled first.

Reviewed-by on v4 3/10
#10 lore raw

Re: [PATCH v4 virtio 03/10] pds_vdpa: move enum from common to adminq header

In reply to Shannon Nelson
Quoted 8 lines · Shannon Nelson

The pds_core_logical_qtype enum and IFNAMSIZ are not needed in the common PDS header, only needed when working with the adminq, so move them to the adminq header.

Note: This patch might conflict with pds_vfio patches that are
in review, depending on which patchset gets pulled first.

#11 lore raw

PATCH v4 4/10 pds_vdpa: new adminq entries

In reply to Shannon Nelson

Add new adminq definitions in support for vDPA operations.

---
 include/linux/pds/pds_adminq.h | 266 +++++++++++++++++++++++++++++++++
 1 file changed, 266 insertions(+)
include/linux/pds/pds_adminq.h +266 -0
index 61b0a8634e1a..c66ead725434 100644
@@ -605,6 +605,257 @@ struct pds_core_q_init_comp {
 	u8     color;
 };
 
+/*
+ * enum pds_vdpa_cmd_opcode - vDPA Device commands
+ */
+enum pds_vdpa_cmd_opcode {
+	PDS_VDPA_CMD_INIT		= 48,
+	PDS_VDPA_CMD_IDENT		= 49,
+	PDS_VDPA_CMD_RESET		= 51,
+	PDS_VDPA_CMD_VQ_RESET		= 52,
+	PDS_VDPA_CMD_VQ_INIT		= 53,
+	PDS_VDPA_CMD_STATUS_UPDATE	= 54,
+	PDS_VDPA_CMD_SET_FEATURES	= 55,
+	PDS_VDPA_CMD_SET_ATTR		= 56,
+	PDS_VDPA_CMD_VQ_SET_STATE	= 57,
+	PDS_VDPA_CMD_VQ_GET_STATE	= 58,
+};
+
+/**
+ * struct pds_vdpa_cmd - generic command
+ * @opcode:	Opcode
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ */
+struct pds_vdpa_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+};
+
+/**
+ * struct pds_vdpa_init_cmd - INIT command
+ * @opcode:	Opcode PDS_VDPA_CMD_INIT
+ * @vdpa_index: Index for vdpa subdevice
+ * @vf_id:	VF id
+ */
+struct pds_vdpa_init_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+};
+
+/**
+ * struct pds_vdpa_ident - vDPA identification data
+ * @hw_features:	vDPA features supported by device
+ * @max_vqs:		max queues available (2 queues for a single queuepair)
+ * @max_qlen:		log(2) of maximum number of descriptors
+ * @min_qlen:		log(2) of minimum number of descriptors
+ *
+ * This struct is used in a DMA block that is set up for the PDS_VDPA_CMD_IDENT
+ * transaction.  Set up the DMA block and send the address in the IDENT cmd
+ * data, the DSC will write the ident information, then we can remove the DMA
+ * block after reading the answer.  If the completion status is 0, then there
+ * is valid information, else there was an error and the data should be invalid.
+ */
+struct pds_vdpa_ident {
+	__le64 hw_features;
+	__le16 max_vqs;
+	__le16 max_qlen;
+	__le16 min_qlen;
+};
+
+/**
+ * struct pds_vdpa_ident_cmd - IDENT command
+ * @opcode:	Opcode PDS_VDPA_CMD_IDENT
+ * @rsvd:       Word boundary padding
+ * @vf_id:	VF id
+ * @len:	length of ident info DMA space
+ * @ident_pa:	address for DMA of ident info (struct pds_vdpa_ident)
+ *			only used for this transaction, then forgotten by DSC
+ */
+struct pds_vdpa_ident_cmd {
+	u8     opcode;
+	u8     rsvd;
+	__le16 vf_id;
+	__le32 len;
+	__le64 ident_pa;
+};
+
+/**
+ * struct pds_vdpa_status_cmd - STATUS_UPDATE command
+ * @opcode:	Opcode PDS_VDPA_CMD_STATUS_UPDATE
+ * @vdpa_index: Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @status:	new status bits
+ */
+struct pds_vdpa_status_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	u8     status;
+};
+
+/**
+ * enum pds_vdpa_attr - List of VDPA device attributes
+ * @PDS_VDPA_ATTR_MAC:          MAC address
+ * @PDS_VDPA_ATTR_MAX_VQ_PAIRS: Max virtqueue pairs
+ */
+enum pds_vdpa_attr {
+	PDS_VDPA_ATTR_MAC          = 1,
+	PDS_VDPA_ATTR_MAX_VQ_PAIRS = 2,
+};
+
+/**
+ * struct pds_vdpa_setattr_cmd - SET_ATTR command
+ * @opcode:		Opcode PDS_VDPA_CMD_SET_ATTR
+ * @vdpa_index:		Index for vdpa subdevice
+ * @vf_id:		VF id
+ * @attr:		attribute to be changed (enum pds_vdpa_attr)
+ * @pad:		Word boundary padding
+ * @mac:		new mac address to be assigned as vdpa device address
+ * @max_vq_pairs:	new limit of virtqueue pairs
+ */
+struct pds_vdpa_setattr_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	u8     attr;
+	u8     pad[3];
+	union {
+		u8 mac[6];
+		__le16 max_vq_pairs;
+	} __packed;
+};
+
+/**
+ * struct pds_vdpa_vq_init_cmd - queue init command
+ * @opcode: Opcode PDS_VDPA_CMD_VQ_INIT
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @qid:	Queue id (bit0 clear = rx, bit0 set = tx, qid=N is ctrlq)
+ * @len:	log(2) of max descriptor count
+ * @desc_addr:	DMA address of descriptor area
+ * @avail_addr:	DMA address of available descriptors (aka driver area)
+ * @used_addr:	DMA address of used descriptors (aka device area)
+ * @intr_index:	interrupt index
+ */
+struct pds_vdpa_vq_init_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	__le16 qid;
+	__le16 len;
+	__le64 desc_addr;
+	__le64 avail_addr;
+	__le64 used_addr;
+	__le16 intr_index;
+};
+
+/**
+ * struct pds_vdpa_vq_init_comp - queue init completion
+ * @status:	Status of the command (enum pds_core_status_code)
+ * @hw_qtype:	HW queue type, used in doorbell selection
+ * @hw_qindex:	HW queue index, used in doorbell selection
+ * @rsvd:	Word boundary padding
+ * @color:	Color bit
+ */
+struct pds_vdpa_vq_init_comp {
+	u8     status;
+	u8     hw_qtype;
+	__le16 hw_qindex;
+	u8     rsvd[11];
+	u8     color;
+};
+
+/**
+ * struct pds_vdpa_vq_reset_cmd - queue reset command
+ * @opcode:	Opcode PDS_VDPA_CMD_VQ_RESET
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @qid:	Queue id
+ */
+struct pds_vdpa_vq_reset_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	__le16 qid;
+};
+
+/**
+ * struct pds_vdpa_set_features_cmd - set hw features
+ * @opcode: Opcode PDS_VDPA_CMD_SET_FEATURES
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @rsvd:       Word boundary padding
+ * @features:	Feature bit mask
+ */
+struct pds_vdpa_set_features_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	__le32 rsvd;
+	__le64 features;
+};
+
+/**
+ * struct pds_vdpa_vq_set_state_cmd - set vq state
+ * @opcode:	Opcode PDS_VDPA_CMD_VQ_SET_STATE
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @qid:	Queue id
+ * @avail:	Device avail index.
+ * @used:	Device used index.
+ *
+ * If the virtqueue uses packed descriptor format, then the avail and used
+ * index must have a wrap count.  The bits should be arranged like the upper
+ * 16 bits in the device available notification data: 15 bit index, 1 bit wrap.
+ */
+struct pds_vdpa_vq_set_state_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	__le16 qid;
+	__le16 avail;
+	__le16 used;
+};
+
+/**
+ * struct pds_vdpa_vq_get_state_cmd - get vq state
+ * @opcode:	Opcode PDS_VDPA_CMD_VQ_GET_STATE
+ * @vdpa_index:	Index for vdpa subdevice
+ * @vf_id:	VF id
+ * @qid:	Queue id
+ */
+struct pds_vdpa_vq_get_state_cmd {
+	u8     opcode;
+	u8     vdpa_index;
+	__le16 vf_id;
+	__le16 qid;
+};
+
+/**
+ * struct pds_vdpa_vq_get_state_comp - get vq state completion
+ * @status:	Status of the command (enum pds_core_status_code)
+ * @rsvd0:      Word boundary padding
+ * @avail:	Device avail index.
+ * @used:	Device used index.
+ * @rsvd:       Word boundary padding
+ * @color:	Color bit
+ *
+ * If the virtqueue uses packed descriptor format, then the avail and used
+ * index will have a wrap count.  The bits will be arranged like the "next"
+ * part of device available notification data: 15 bit index, 1 bit wrap.
+ */
+struct pds_vdpa_vq_get_state_comp {
+	u8     status;
+	u8     rsvd0;
+	__le16 avail;
+	__le16 used;
+	u8     rsvd[9];
+	u8     color;
+};
+
 union pds_core_adminq_cmd {
 	u8     opcode;
 	u8     bytes[64];
@@ -621,6 +872,18 @@ union pds_core_adminq_cmd {
 
 	struct pds_core_q_identify_cmd    q_ident;
 	struct pds_core_q_init_cmd        q_init;
+
+	struct pds_vdpa_cmd		  vdpa;
+	struct pds_vdpa_init_cmd	  vdpa_init;
+	struct pds_vdpa_ident_cmd	  vdpa_ident;
+	struct pds_vdpa_status_cmd	  vdpa_status;
+	struct pds_vdpa_setattr_cmd	  vdpa_setattr;
+	struct pds_vdpa_set_features_cmd  vdpa_set_features;
+	struct pds_vdpa_vq_init_cmd	  vdpa_vq_init;
+	struct pds_vdpa_vq_reset_cmd	  vdpa_vq_reset;
+	struct pds_vdpa_vq_set_state_cmd  vdpa_vq_set_state;
+	struct pds_vdpa_vq_get_state_cmd  vdpa_vq_get_state;
+
 };
 
 union pds_core_adminq_comp {
@@ -642,6 +905,9 @@ union pds_core_adminq_comp {
 
 	struct pds_core_q_identify_comp   q_ident;
 	struct pds_core_q_init_comp       q_init;
+
+	struct pds_vdpa_vq_init_comp	  vdpa_vq_init;
+	struct pds_vdpa_vq_get_state_comp vdpa_vq_get_state;
 };
 
 #ifndef __CHECKER__
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#13 lore raw

PATCH v4 5/10 pds_vdpa: get vdpa management info

In reply to Shannon Nelson

Find the vDPA management information from the DSC in order to advertise it to the vdpa subsystem.

---
 drivers/vdpa/pds/Makefile   |   3 +-
 drivers/vdpa/pds/aux_drv.c  |  17 ++++++
 drivers/vdpa/pds/aux_drv.h  |   7 +++
 drivers/vdpa/pds/debugfs.c  |   1 +
 drivers/vdpa/pds/vdpa_dev.c | 108 ++++++++++++++++++++++++++++++++++++
 drivers/vdpa/pds/vdpa_dev.h |  15 +++++
 6 files changed, 150 insertions(+), 1 deletion(-)
 create mode 100644 drivers/vdpa/pds/vdpa_dev.c
 create mode 100644 drivers/vdpa/pds/vdpa_dev.h
drivers/vdpa/pds/Makefile +2 -1
index a9cd2f450ae1..13b50394ec64 100644
@@ -3,6 +3,7 @@
 
 obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o
 
-pds_vdpa-y := aux_drv.o
+pds_vdpa-y := aux_drv.o \
+	      vdpa_dev.o
 
 pds_vdpa-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/vdpa/pds/aux_drv.c +17 -0
index e4a0ad61ea22..aa748cf55d2b 100644
@@ -3,6 +3,7 @@
 
 #include <linux/auxiliary_bus.h>
 #include <linux/pci.h>
+#include <linux/vdpa.h>
 
 #include <linux/pds/pds_common.h>
 #include <linux/pds/pds_core_if.h>
@@ -11,6 +12,7 @@
 
 #include "aux_drv.h"
 #include "debugfs.h"
+#include "vdpa_dev.h"
 
 static const struct auxiliary_device_id pds_vdpa_id_table[] = {
 	{ .name = PDS_VDPA_DEV_NAME, },
@@ -24,15 +26,28 @@ static int pds_vdpa_probe(struct auxiliary_device *aux_dev,
 	struct pds_auxiliary_dev *padev =
 		container_of(aux_dev, struct pds_auxiliary_dev, aux_dev);
 	struct pds_vdpa_aux *vdpa_aux;
+	int err;
 
 	vdpa_aux = kzalloc(sizeof(*vdpa_aux), GFP_KERNEL);
 	if (!vdpa_aux)
 		return -ENOMEM;
 
 	vdpa_aux->padev = padev;
+	vdpa_aux->vf_id = pci_iov_vf_id(padev->vf_pdev);
 	auxiliary_set_drvdata(aux_dev, vdpa_aux);
 
+	/* Get device ident info and set up the vdpa_mgmt_dev */
+	err = pds_vdpa_get_mgmt_info(vdpa_aux);
+	if (err)
+		goto err_free_mem;
+
 	return 0;
+
+err_free_mem:
+	kfree(vdpa_aux);
+	auxiliary_set_drvdata(aux_dev, NULL);
+
+	return err;
 }
 
 static void pds_vdpa_remove(struct auxiliary_device *aux_dev)
@@ -40,6 +55,8 @@ static void pds_vdpa_remove(struct auxiliary_device *aux_dev)
 	struct pds_vdpa_aux *vdpa_aux = auxiliary_get_drvdata(aux_dev);
 	struct device *dev = &aux_dev->dev;
 
+	pci_free_irq_vectors(vdpa_aux->padev->vf_pdev);
+
 	kfree(vdpa_aux);
 	auxiliary_set_drvdata(aux_dev, NULL);
 
drivers/vdpa/pds/aux_drv.h +7 -0
index f1e99359424e..dcec782e79eb 100644
@@ -10,6 +10,13 @@
 struct pds_vdpa_aux {
 	struct pds_auxiliary_dev *padev;
 
+	struct vdpa_mgmt_dev vdpa_mdev;
+
+	struct pds_vdpa_ident ident;
+
+	int vf_id;
 	struct dentry *dentry;
+
+	int nintrs;
 };
 #endif /* _AUX_DRV_H_ */
drivers/vdpa/pds/debugfs.c +1 -0
index 5be22fb7a76a..d91dceb07380 100644
@@ -2,6 +2,7 @@
 /* Copyright(c) 2023 Advanced Micro Devices, Inc */
 
 #include <linux/pci.h>
+#include <linux/vdpa.h>
 
 #include <linux/pds/pds_common.h>
 #include <linux/pds/pds_core_if.h>
drivers/vdpa/pds/vdpa_dev.c new file +108 -0

108 lines not on this page · raw message

drivers/vdpa/pds/vdpa_dev.h new file +15 -0
new file mode 100644
index 000000000000..97fab833a0aa
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _VDPA_DEV_H_
+#define _VDPA_DEV_H_
+
+#define PDS_VDPA_MAX_QUEUES	65
+
+struct pds_vdpa_device {
+	struct vdpa_device vdpa_dev;
+	struct pds_vdpa_aux *vdpa_aux;
+};
+
+int pds_vdpa_get_mgmt_info(struct pds_vdpa_aux *vdpa_aux);
+#endif /* _VDPA_DEV_H_ */
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#15 lore raw

PATCH v4 6/10 pds_vdpa: virtio bar setup for vdpa

In reply to Shannon Nelson

Prep and use the "modern" virtio bar utilities to get our virtio config space ready.

---
 drivers/vdpa/pds/aux_drv.c | 25 +++++++++++++++++++++++++
 drivers/vdpa/pds/aux_drv.h |  3 +++
 2 files changed, 28 insertions(+)
drivers/vdpa/pds/aux_drv.c +25 -0
index aa748cf55d2b..0c4a135b1484 100644
@@ -4,6 +4,7 @@
 #include <linux/auxiliary_bus.h>
 #include <linux/pci.h>
 #include <linux/vdpa.h>
+#include <linux/virtio_pci_modern.h>
 
 #include <linux/pds/pds_common.h>
 #include <linux/pds/pds_core_if.h>
@@ -19,12 +20,22 @@ static const struct auxiliary_device_id pds_vdpa_id_table[] = {
 	{},
 };
 
+static int pds_vdpa_device_id_check(struct pci_dev *pdev)
+{
+	if (pdev->device != PCI_DEVICE_ID_PENSANDO_VDPA_VF ||
+	    pdev->vendor != PCI_VENDOR_ID_PENSANDO)
+		return -ENODEV;
+
+	return PCI_DEVICE_ID_PENSANDO_VDPA_VF;
+}
+
 static int pds_vdpa_probe(struct auxiliary_device *aux_dev,
 			  const struct auxiliary_device_id *id)
 
 {
 	struct pds_auxiliary_dev *padev =
 		container_of(aux_dev, struct pds_auxiliary_dev, aux_dev);
+	struct device *dev = &aux_dev->dev;
 	struct pds_vdpa_aux *vdpa_aux;
 	int err;
 
@@ -41,8 +52,21 @@ static int pds_vdpa_probe(struct auxiliary_device *aux_dev,
 	if (err)
 		goto err_free_mem;
 
+	/* Find the virtio configuration */
+	vdpa_aux->vd_mdev.pci_dev = padev->vf_pdev;
+	vdpa_aux->vd_mdev.device_id_check = pds_vdpa_device_id_check;
+	vdpa_aux->vd_mdev.dma_mask = DMA_BIT_MASK(PDS_CORE_ADDR_LEN);
+	err = vp_modern_probe(&vdpa_aux->vd_mdev);
+	if (err) {
+		dev_err(dev, "Unable to probe for virtio configuration: %pe\n",
+			ERR_PTR(err));
+		goto err_free_mgmt_info;
+	}
+
 	return 0;
 
+err_free_mgmt_info:
+	pci_free_irq_vectors(padev->vf_pdev);
 err_free_mem:
 	kfree(vdpa_aux);
 	auxiliary_set_drvdata(aux_dev, NULL);
@@ -55,6 +79,7 @@ static void pds_vdpa_remove(struct auxiliary_device *aux_dev)
 	struct pds_vdpa_aux *vdpa_aux = auxiliary_get_drvdata(aux_dev);
 	struct device *dev = &aux_dev->dev;
 
+	vp_modern_remove(&vdpa_aux->vd_mdev);
 	pci_free_irq_vectors(vdpa_aux->padev->vf_pdev);
 
 	kfree(vdpa_aux);
drivers/vdpa/pds/aux_drv.h +3 -0
index dcec782e79eb..99e0ff340bfa 100644
@@ -4,6 +4,8 @@
 #ifndef _AUX_DRV_H_
 #define _AUX_DRV_H_
 
+#include <linux/virtio_pci_modern.h>
+
 #define PDS_VDPA_DRV_DESCRIPTION    "AMD/Pensando vDPA VF Device Driver"
 #define PDS_VDPA_DRV_NAME           KBUILD_MODNAME
 
@@ -16,6 +18,7 @@ struct pds_vdpa_aux {
 
 	int vf_id;
 	struct dentry *dentry;
+	struct virtio_pci_modern_device vd_mdev;
 
 	int nintrs;
 };
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#17 lore raw

PATCH v4 7/10 pds_vdpa: add vdpa config client commands

In reply to Shannon Nelson

These are the adminq commands that will be needed for setting up and using the vDPA device. There are a number of commands defined in the FW's API, but by making use of the FW's virtio BAR we only need a few of these commands for vDPA support.

---

Note: the previous version was Acked-by Jason Wang, but this has gone
through some rework of the adminq command API usage and added two new commands. It is still essentially the same code, but I've dropped the Acked-by for now.

 drivers/vdpa/pds/Makefile   |   1 +
 drivers/vdpa/pds/cmds.c     | 207 ++++++++++++++++++++++++++++++++++++
 drivers/vdpa/pds/cmds.h     |  20 ++++
 drivers/vdpa/pds/vdpa_dev.h |  33 +++++-
 4 files changed, 260 insertions(+), 1 deletion(-)
 create mode 100644 drivers/vdpa/pds/cmds.c
 create mode 100644 drivers/vdpa/pds/cmds.h
drivers/vdpa/pds/Makefile +1 -0
index 13b50394ec64..2e22418e3ab3 100644
@@ -4,6 +4,7 @@
 obj-$(CONFIG_PDS_VDPA) := pds_vdpa.o
 
 pds_vdpa-y := aux_drv.o \
+	      cmds.o \
 	      vdpa_dev.o
 
 pds_vdpa-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/vdpa/pds/cmds.c new file +207 -0

207 lines not on this page · raw message

drivers/vdpa/pds/cmds.h new file +20 -0
new file mode 100644
index 000000000000..cf4f8764e73c
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _VDPA_CMDS_H_
+#define _VDPA_CMDS_H_
+
+int pds_vdpa_init_hw(struct pds_vdpa_device *pdsv);
+
+int pds_vdpa_cmd_reset(struct pds_vdpa_device *pdsv);
+int pds_vdpa_cmd_set_mac(struct pds_vdpa_device *pdsv, u8 *mac);
+int pds_vdpa_cmd_set_max_vq_pairs(struct pds_vdpa_device *pdsv, u16 max_vqp);
+int pds_vdpa_cmd_init_vq(struct pds_vdpa_device *pdsv, u16 qid,
+			 struct pds_vdpa_vq_info *vq_info);
+int pds_vdpa_cmd_reset_vq(struct pds_vdpa_device *pdsv, u16 qid);
+int pds_vdpa_cmd_set_features(struct pds_vdpa_device *pdsv, u64 features);
+int pds_vdpa_cmd_set_vq_state(struct pds_vdpa_device *pdsv,
+			      u16 qid, u16 avail, u16 used);
+int pds_vdpa_cmd_get_vq_state(struct pds_vdpa_device *pdsv,
+			      u16 qid, u16 *avail, u16 *used);
+#endif /* _VDPA_CMDS_H_ */
drivers/vdpa/pds/vdpa_dev.h +32 -1
index 97fab833a0aa..a21596f438c1 100644
@@ -4,11 +4,42 @@
 #ifndef _VDPA_DEV_H_
 #define _VDPA_DEV_H_
 
-#define PDS_VDPA_MAX_QUEUES	65
+#include <linux/pci.h>
+#include <linux/vdpa.h>
+
+struct pds_vdpa_vq_info {
+	bool ready;
+	u64 desc_addr;
+	u64 avail_addr;
+	u64 used_addr;
+	u32 q_len;
+	u16 qid;
+	int irq;
+	char irq_name[32];
+
+	void __iomem *notify;
+	dma_addr_t notify_pa;
+
+	u64 doorbell;
+	u16 avail_idx;
+	u16 used_idx;
 
+	struct vdpa_callback event_cb;
+	struct pds_vdpa_device *pdsv;
+};
+
+#define PDS_VDPA_MAX_QUEUES	65
+#define PDS_VDPA_MAX_QLEN	32768
 struct pds_vdpa_device {
 	struct vdpa_device vdpa_dev;
 	struct pds_vdpa_aux *vdpa_aux;
+
+	struct pds_vdpa_vq_info vqs[PDS_VDPA_MAX_QUEUES];
+	u64 req_features;		/* features requested by vdpa */
+	u64 actual_features;		/* features negotiated and in use */
+	u8 vdpa_index;			/* rsvd for future subdevice use */
+	u8 num_vqs;			/* num vqs in use */
+	struct vdpa_callback config_cb;
 };
 
 int pds_vdpa_get_mgmt_info(struct pds_vdpa_aux *vdpa_aux);
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Reviewed-by on v4 7/10
#18 lore raw

Re: [PATCH v4 virtio 07/10] pds_vdpa: add vdpa config client commands

Quoted 7 lines · Shannon Nelson

These are the adminq commands that will be needed for setting up and using the vDPA device. There are a number of commands defined in the FW's API, but by making use of the FW's virtio BAR we only need a few of these commands for vDPA support.

#19 lore raw

PATCH v4 8/10 pds_vdpa: add support for vdpa and vdpamgmt interfaces

In reply to Shannon Nelson

This is the vDPA device support, where we advertise that we can support the virtio queues and deal with the configuration work through the pds_core's adminq.

---
 drivers/vdpa/pds/aux_drv.c  |  15 +
 drivers/vdpa/pds/aux_drv.h  |   1 +
 drivers/vdpa/pds/debugfs.c  | 261 ++++++++++++++++++
 drivers/vdpa/pds/debugfs.h  |   5 +
 drivers/vdpa/pds/vdpa_dev.c | 532 +++++++++++++++++++++++++++++++++++-
 5 files changed, 813 insertions(+), 1 deletion(-)
drivers/vdpa/pds/aux_drv.c +15 -0
index 0c4a135b1484..186e9ee22eb1 100644
@@ -63,8 +63,21 @@ static int pds_vdpa_probe(struct auxiliary_device *aux_dev,
 		goto err_free_mgmt_info;
 	}
 
+	/* Let vdpa know that we can provide devices */
+	err = vdpa_mgmtdev_register(&vdpa_aux->vdpa_mdev);
+	if (err) {
+		dev_err(dev, "%s: Failed to initialize vdpa_mgmt interface: %pe\n",
+			__func__, ERR_PTR(err));
+		goto err_free_virtio;
+	}
+
+	pds_vdpa_debugfs_add_pcidev(vdpa_aux);
+	pds_vdpa_debugfs_add_ident(vdpa_aux);
+
 	return 0;
 
+err_free_virtio:
+	vp_modern_remove(&vdpa_aux->vd_mdev);
 err_free_mgmt_info:
 	pci_free_irq_vectors(padev->vf_pdev);
 err_free_mem:
@@ -79,9 +92,11 @@ static void pds_vdpa_remove(struct auxiliary_device *aux_dev)
 	struct pds_vdpa_aux *vdpa_aux = auxiliary_get_drvdata(aux_dev);
 	struct device *dev = &aux_dev->dev;
 
+	vdpa_mgmtdev_unregister(&vdpa_aux->vdpa_mdev);
 	vp_modern_remove(&vdpa_aux->vd_mdev);
 	pci_free_irq_vectors(vdpa_aux->padev->vf_pdev);
 
+	pds_vdpa_debugfs_del_vdpadev(vdpa_aux);
 	kfree(vdpa_aux);
 	auxiliary_set_drvdata(aux_dev, NULL);
 
drivers/vdpa/pds/aux_drv.h +1 -0
index 99e0ff340bfa..26b75344156e 100644
@@ -13,6 +13,7 @@ struct pds_vdpa_aux {
 	struct pds_auxiliary_dev *padev;
 
 	struct vdpa_mgmt_dev vdpa_mdev;
+	struct pds_vdpa_device *pdsv;
 
 	struct pds_vdpa_ident ident;
 
drivers/vdpa/pds/debugfs.c +261 -0

270 lines not on this page · raw message

drivers/vdpa/pds/debugfs.h +5 -0
index 658849591a99..c088a4e8f1e9 100644
@@ -8,5 +8,10 @@
 
 void pds_vdpa_debugfs_create(void);
 void pds_vdpa_debugfs_destroy(void);
+void pds_vdpa_debugfs_add_pcidev(struct pds_vdpa_aux *vdpa_aux);
+void pds_vdpa_debugfs_add_ident(struct pds_vdpa_aux *vdpa_aux);
+void pds_vdpa_debugfs_add_vdpadev(struct pds_vdpa_aux *vdpa_aux);
+void pds_vdpa_debugfs_del_vdpadev(struct pds_vdpa_aux *vdpa_aux);
+void pds_vdpa_debugfs_reset_vdpadev(struct pds_vdpa_aux *vdpa_aux);
 
 #endif /* _PDS_VDPA_DEBUGFS_H_ */
drivers/vdpa/pds/vdpa_dev.c +531 -1

556 lines not on this page · raw message

Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#21 lore raw

PATCH v4 9/10 pds_vdpa: subscribe to the pds_core events

In reply to Shannon Nelson

Register for the pds_core's notification events, primarily to find out when the FW has been reset so we can pass this on back up the chain.

---
 drivers/vdpa/pds/vdpa_dev.c | 68 ++++++++++++++++++++++++++++++++++++-
 drivers/vdpa/pds/vdpa_dev.h |  1 +
 2 files changed, 68 insertions(+), 1 deletion(-)
drivers/vdpa/pds/vdpa_dev.c +67 -1
index c3316f0faa0c..93b12f73423f 100644
@@ -21,6 +21,61 @@ static struct pds_vdpa_device *vdpa_to_pdsv(struct vdpa_device *vdpa_dev)
 	return container_of(vdpa_dev, struct pds_vdpa_device, vdpa_dev);
 }
 
+static int pds_vdpa_notify_handler(struct notifier_block *nb,
+				   unsigned long ecode,
+				   void *data)
+{
+	struct pds_vdpa_device *pdsv = container_of(nb, struct pds_vdpa_device, nb);
+	struct device *dev = &pdsv->vdpa_aux->padev->aux_dev.dev;
+
+	dev_dbg(dev, "%s: event code %lu\n", __func__, ecode);
+
+	/* Give the upper layers a hint that something interesting
+	 * may have happened.  It seems that the only thing this
+	 * triggers in the virtio-net drivers above us is a check
+	 * of link status.
+	 *
+	 * We don't set the NEEDS_RESET flag for EVENT_RESET
+	 * because we're likely going through a recovery or
+	 * fw_update and will be back up and running soon.
+	 */
+	if (ecode == PDS_EVENT_RESET || ecode == PDS_EVENT_LINK_CHANGE) {
+		if (pdsv->config_cb.callback)
+			pdsv->config_cb.callback(pdsv->config_cb.private);
+	}
+
+	return 0;
+}
+
+static int pds_vdpa_register_event_handler(struct pds_vdpa_device *pdsv)
+{
+	struct device *dev = &pdsv->vdpa_aux->padev->aux_dev.dev;
+	struct notifier_block *nb = &pdsv->nb;
+	int err;
+
+	if (!nb->notifier_call) {
+		nb->notifier_call = pds_vdpa_notify_handler;
+		err = pdsc_register_notify(nb);
+		if (err) {
+			nb->notifier_call = NULL;
+			dev_err(dev, "failed to register pds event handler: %ps\n",
+				ERR_PTR(err));
+			return -EINVAL;
+		}
+		dev_dbg(dev, "pds event handler registered\n");
+	}
+
+	return 0;
+}
+
+static void pds_vdpa_unregister_event_handler(struct pds_vdpa_device *pdsv)
+{
+	if (pdsv->nb.notifier_call) {
+		pdsc_unregister_notify(&pdsv->nb);
+		pdsv->nb.notifier_call = NULL;
+	}
+}
+
 static int pds_vdpa_set_vq_address(struct vdpa_device *vdpa_dev, u16 qid,
 				   u64 desc_addr, u64 driver_addr, u64 device_addr)
 {
@@ -522,6 +577,12 @@ static int pds_vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 
 	pdsv->vdpa_dev.mdev = &vdpa_aux->vdpa_mdev;
 
+	err = pds_vdpa_register_event_handler(pdsv);
+	if (err) {
+		dev_err(dev, "Failed to register for PDS events: %pe\n", ERR_PTR(err));
+		goto err_unmap;
+	}
+
 	/* We use the _vdpa_register_device() call rather than the
 	 * vdpa_register_device() to avoid a deadlock because our
 	 * dev_add() is called with the vdpa_dev_lock already set
@@ -530,13 +591,15 @@ static int pds_vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 	err = _vdpa_register_device(&pdsv->vdpa_dev, pdsv->num_vqs);
 	if (err) {
 		dev_err(dev, "Failed to register to vDPA bus: %pe\n", ERR_PTR(err));
-		goto err_unmap;
+		goto err_unevent;
 	}
 
 	pds_vdpa_debugfs_add_vdpadev(vdpa_aux);
 
 	return 0;
 
+err_unevent:
+	pds_vdpa_unregister_event_handler(pdsv);
 err_unmap:
 	put_device(&pdsv->vdpa_dev.dev);
 	vdpa_aux->pdsv = NULL;
@@ -546,8 +609,11 @@ static int pds_vdpa_dev_add(struct vdpa_mgmt_dev *mdev, const char *name,
 static void pds_vdpa_dev_del(struct vdpa_mgmt_dev *mdev,
 			     struct vdpa_device *vdpa_dev)
 {
+	struct pds_vdpa_device *pdsv = vdpa_to_pdsv(vdpa_dev);
 	struct pds_vdpa_aux *vdpa_aux;
 
+	pds_vdpa_unregister_event_handler(pdsv);
+
 	vdpa_aux = container_of(mdev, struct pds_vdpa_aux, vdpa_mdev);
 	_vdpa_unregister_device(vdpa_dev);
 
drivers/vdpa/pds/vdpa_dev.h +1 -0
index a21596f438c1..1650a2b08845 100644
@@ -40,6 +40,7 @@ struct pds_vdpa_device {
 	u8 vdpa_index;			/* rsvd for future subdevice use */
 	u8 num_vqs;			/* num vqs in use */
 	struct vdpa_callback config_cb;
+	struct notifier_block nb;
 };
 
 int pds_vdpa_get_mgmt_info(struct pds_vdpa_aux *vdpa_aux);
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#23 lore raw

PATCH v4 10/10 pds_vdpa: pds_vdps.rst and Kconfig

In reply to Shannon Nelson

Add the documentation and Kconfig entry for pds_vdpa driver.

---
 .../device_drivers/ethernet/amd/pds_vdpa.rst  | 85 +++++++++++++++++++
 .../device_drivers/ethernet/index.rst         |  1 +
 MAINTAINERS                                   |  4 +
 drivers/vdpa/Kconfig                          |  8 ++
 4 files changed, 98 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vdpa.rst
Documentation/networking/device_drivers/ethernet/amd/pds_vdpa.rst new file +85 -0
new file mode 100644
index 000000000000..587927d3de92
@@ -0,0 +1,85 @@
+.. SPDX-License-Identifier: GPL-2.0+
+.. note: can be edited and viewed with /usr/bin/formiko-vim
+
+==========================================================
+PCI vDPA driver for the AMD/Pensando(R) DSC adapter family
+==========================================================
+
+AMD/Pensando vDPA VF Device Driver
+
+Copyright(c) 2023 Advanced Micro Devices, Inc
+
+Overview
+========
+
+The ``pds_vdpa`` driver is an auxiliary bus driver that supplies
+a vDPA device for use by the virtio network stack.  It is used with
+the Pensando Virtual Function devices that offer vDPA and virtio queue
+services.  It depends on the ``pds_core`` driver and hardware for the PF
+and VF PCI handling as well as for device configuration services.
+
+Using the device
+================
+
+The ``pds_vdpa`` device is enabled via multiple configuration steps and
+depends on the ``pds_core`` driver to create and enable SR-IOV Virtual
+Function devices.  After the VFs are enabled, we enable the vDPA service
+in the ``pds_core`` device to create the auxiliary devices used by pds_vdpa.
+
+Example steps:
+
+.. code-block:: bash
+
+  #!/bin/bash
+
+  modprobe pds_core
+  modprobe vdpa
+  modprobe pds_vdpa
+
+  PF_BDF=`ls /sys/module/pds_core/drivers/pci\:pds_core/*/sriov_numvfs | awk -F / '{print $7}'`
+
+  # Enable vDPA VF auxiliary device(s) in the PF
+  devlink dev param set pci/$PF_BDF name enable_vnet cmode runtime value true
+
+  # Create a VF for vDPA use
+  echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs
+
+  # Find the vDPA services/devices available
+  PDS_VDPA_MGMT=`vdpa mgmtdev show | grep vDPA | head -1 | cut -d: -f1`
+
+  # Create a vDPA device for use in virtio network configurations
+  vdpa dev add name vdpa1 mgmtdev $PDS_VDPA_MGMT mac 00:11:22:33:44:55
+
+  # Set up an ethernet interface on the vdpa device
+  modprobe virtio_vdpa
+
+
+
+Enabling the driver
+===================
+
+The driver is enabled via the standard kernel configuration system,
+using the make command::
+
+  make oldconfig/menuconfig/etc.
+
+The driver is located in the menu structure at:
+
+  -> Device Drivers
+    -> Network device support (NETDEVICES [=y])
+      -> Ethernet driver support
+        -> Pensando devices
+          -> Pensando Ethernet PDS_VDPA Support
+
+Support
+=======
+
+For general Linux networking support, please use the netdev mailing
+list, which is monitored by Pensando personnel::
+
+  [email protected]
+
+For more specific support needs, please use the Pensando driver support
+email::
+
+  [email protected]
Documentation/networking/device_drivers/ethernet/index.rst +1 -0
index 417ca514a4d0..94ecb67c0885 100644
@@ -15,6 +15,7 @@ Contents:
    amazon/ena
    altera/altera_tse
    amd/pds_core
+   amd/pds_vdpa
    aquantia/atlantic
    chelsio/cxgb
    cirrus/cs89x0
MAINTAINERS +4 -0
index 6ac562e0381e..93210a8ac74f 100644
@@ -22148,6 +22148,10 @@ SNET DPU VIRTIO DATA PATH ACCELERATOR
 R:	Alvaro Karsz <[email protected]>
 F:	drivers/vdpa/solidrun/
 
+PDS DSC VIRTIO DATA PATH ACCELERATOR
+R:	Shannon Nelson <[email protected]>
+F:	drivers/vdpa/pds/
+
 VIRTIO BALLOON
 M:	"Michael S. Tsirkin" <[email protected]>
 M:	David Hildenbrand <[email protected]>
drivers/vdpa/Kconfig +8 -0
index cd6ad92f3f05..2ee1b288691d 100644
@@ -116,4 +116,12 @@ config ALIBABA_ENI_VDPA
 	  This driver includes a HW monitor device that
 	  reads health values from the DPU.
 
+config PDS_VDPA
+	tristate "vDPA driver for AMD/Pensando DSC devices"
+	depends on PDS_CORE
+	help
+	  vDPA network driver for AMD/Pensando's PDS Core devices.
+	  With this driver, the VirtIO dataplane can be
+	  offloaded to an AMD/Pensando DSC device.
+
 endif # VDPA
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization