RDMA/mana_ib: Enable RDMA devices on the CDX bus
PATCH v2 1/1 RDMA/mana_ib: Enable RDMA devices on the CDX bus
From: Zhiyue Qiu <[email protected]>
Enable MANA RDMA on CDX by depending on the shared GDMA core and matching the CDX auxiliary devices.
Map non-PCI userspace doorbells with pgprot_device() to match the CDX transport, and report the Microsoft vendor ID directly.
Allocate missing DMA parameters before GDMA setup using managed memory. Register a cleanup action to clear the device pointer before the memory is freed.
---
Changes in v2:
- Allocate missing CDX DMA parameters before GDMA setup.
drivers/infiniband/hw/mana/Kconfig | 10 +++---- drivers/infiniband/hw/mana/device.c | 2 ++ drivers/infiniband/hw/mana/main.c | 9 ++++-- .../net/ethernet/microsoft/mana/gdma_cdx.c | 30 +++++++++++++++---- 4 files changed, 38 insertions(+), 13 deletions(-)
drivers/infiniband/hw/mana/Kconfig +5 -5
@@ -1,10 +1,10 @@ # SPDX-License-Identifier: GPL-2.0-only config MANA_INFINIBAND tristate "Microsoft Azure Network Adapter support" - depends on NETDEVICES && ETHERNET && PCI && MICROSOFT_MANA + depends on NETDEVICES && ETHERNET && MICROSOFT_GDMA_CORE help This driver provides low-level RDMA support for Microsoft Azure - Network Adapter (MANA). MANA supports RDMA features that can be used - for workloads (e.g. DPDK, MPI etc) that uses RDMA verbs to directly - access hardware from user-mode processes in Microsoft Azure cloud - environment. + Network Adapter (MANA) on the PCI or CDX bus. MANA supports RDMA + features that workloads such as DPDK and MPI can use to directly + access hardware from user-mode processes in the Microsoft Azure + cloud environment.
drivers/infiniband/hw/mana/device.c +2 -0
@@ -257,6 +257,8 @@ static void mana_ib_remove(struct auxiliary_device *adev)static const struct auxiliary_device_id mana_id_table[] = { { .name ="mana.rdma" , }, { .name ="mana.eth" , }, + { .name ="mana_cdx.rdma" , }, + { .name ="mana_cdx.eth" , }, {}, };
drivers/infiniband/hw/mana/main.c +6 -3
@@ -572,7 +572,11 @@ int mana_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma) pfn = (gc->phys_db_page_base + gc->db_page_size * mana_ucontext->doorbell) >>PAGE_SHIFT ; - prot =pgprot_writecombine (vma->vm_page_prot); +/* Match the Device attributes used by the CDX transport. */ +if (dev_is_pci (gc->dev)) + prot =pgprot_writecombine (vma->vm_page_prot); +else + prot =pgprot_device (vma->vm_page_prot); ret =rdma_user_mmap_io (ibcontext, vma, pfn,PAGE_SIZE , prot,NULL );
@@ -623,14 +627,13 @@ int mana_ib_query_device(struct ib_device *ibdev, struct ib_device_attr *props, struct ib_udata *uhw) { struct mana_ib_dev *dev = container_of(ibdev, struct mana_ib_dev, ib_dev); -struct pci_dev *pdev =to_pci_dev (mdev_to_gc (dev)->dev);int err; err =ib_no_udata_io (uhw);if (err)return err; - props->vendor_id = pdev->vendor; + props->vendor_id =PCI_VENDOR_ID_MICROSOFT ; props->vendor_part_id = dev->gdma_dev->dev_id.type; props->max_mr_size =MANA_IB_MAX_MR_SIZE ; props->page_size_cap = dev->adapter_caps.page_size_cap;
drivers/net/ethernet/microsoft/mana/gdma_cdx.c +25 -5
@@ -3,6 +3,7 @@#include <linux/module.h> #include <linux/bitmap.h> #include <linux/cdx/cdx_bus.h> +#include <linux/device.h> #include <linux/msi.h> #include <linux/xarray.h>
@@ -61,11 +62,8 @@ static int mana_cdx_setup_hwc_irqs(struct gdma_context *gc)return err; } -/* The PCI path also calls dma_set_max_seg_size(), but that is -* deliberately omitted here: the CDX core does not attach a -* dma_parms to the device, so the helper would only WARN, and MANA -* never builds scatter-gather lists, so the limit is never read. -*/ +dma_set_max_seg_size (gc->dev,UINT_MAX ); +dma_set_seg_boundary (gc->dev,DMA_BIT_MASK (32 )); err =cdx_enable_msi (cdx_dev);if (err) {
@@ -196,6 +194,13 @@ static const struct gdma_bus_ops mana_cdx_bus_ops = { .dev_reset = mana_cdx_dev_reset, }; +static void mana_cdx_clear_dma_parms (void *data) +{ +struct device *dev = data; + + dev->dma_parms =NULL ; +} +static int mana_cdx_gd_probe (struct cdx_device *cdx_dev) {struct gdma_context *gc;
@@ -241,6 +246,21 @@ static int mana_cdx_gd_probe(struct cdx_device *cdx_dev) gc->bar0_va = bar0_va; gc->dev = &cdx_dev->dev; gc->bus_ops = &mana_cdx_bus_ops; +if (!gc->dev->dma_parms) { + gc->dev->dma_parms =devm_kzalloc (gc->dev, +sizeof (*gc->dev->dma_parms), +GFP_KERNEL ); +if (!gc->dev->dma_parms) { + err = -ENOMEM ; + goto unmap_bar; + } + +/* Clear the pointer before devres frees the allocation. */ + err =devm_add_action_or_reset (gc->dev, mana_cdx_clear_dma_parms, + gc->dev); +if (err) + goto unmap_bar; + }/* gc->is_pf is intentionally left clear: this device uses the VF * register layout, and the vPort and MAC filter registration that
Signature · 1 line
2.43.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues.
Quoted 8 lines
From: Zhiyue Qiu <[email protected]>
Enable MANA RDMA on CDX by depending on the shared GDMA core and matching the CDX auxiliary devices.
Map non-PCI userspace doorbells with
pgprot_device()to match the CDX transport, and report the Microsoft vendor ID directly.
Quoted 7 lines
Allocate missing DMA parameters before GDMA setup using managed memory. Register a cleanup action to clear the device pointer before the memory is freed.
Signed-off-by Zhiyue Qiu <[email protected]>Co-developed-by Konstantin Taranov <[email protected]>Signed-off-by Konstantin Taranov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1