Skip to contentSkip to conversation

Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

PATCH Superseded [email protected] 6 participants 33 messages Aug 31, 2022 – Sep 21, 2022

Subsystems (5)

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v5 cover Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

From: Long Li <[email protected]>

This patchset implements a RDMA driver for Microsoft Azure Network Adapter (MANA). In MANA, the RDMA device is modeled as an auxiliary device to the Ethernet device.

The first 11 patches modify the MANA Ethernet driver to support RDMA driver. The last patch implementes the RDMA driver.

The user-mode of the driver is being reviewed at:
https://github.com/linux-rdma/rdma-core/pull/1177

Ajay Sharma (3):
net: mana: Set the DMA device max segment size
net: mana: Define and process GDMA response code
GDMA_STATUS_MORE_ENTRIES
net: mana: Define data structures for protection domain and memory
registration

Long Li (9):
net: mana: Add support for auxiliary device
net: mana: Record the physical address for doorbell page region
net: mana: Handle vport sharing between devices
net: mana: Add functions for allocating doorbell page from GDMA
net: mana: Export Work Queue functions for use by RDMA driver
net: mana: Record port number in netdev
net: mana: Move header files to a common location
net: mana: Define max values for SGL entries RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

 MAINTAINERS                                   |   4 +
 drivers/infiniband/Kconfig                    |   1 +
 drivers/infiniband/hw/Makefile                |   1 +
 drivers/infiniband/hw/mana/Kconfig            |   7 +
 drivers/infiniband/hw/mana/Makefile           |   4 +
 drivers/infiniband/hw/mana/cq.c               |  80 +++
 drivers/infiniband/hw/mana/device.c           | 129 ++++
 drivers/infiniband/hw/mana/main.c             | 555 ++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h          | 165 ++++++
 drivers/infiniband/hw/mana/mr.c               | 133 +++++
 drivers/infiniband/hw/mana/qp.c               | 501 ++++++++++++++++
 drivers/infiniband/hw/mana/wq.c               | 114 ++++
 .../net/ethernet/microsoft/mana/gdma_main.c   |  96 ++-
 .../net/ethernet/microsoft/mana/hw_channel.c  |   6 +-
 .../net/ethernet/microsoft/mana/mana_bpf.c    |   2 +-
 drivers/net/ethernet/microsoft/mana/mana_en.c | 177 +++++-
 .../ethernet/microsoft/mana/mana_ethtool.c    |   2 +-
 .../net/ethernet/microsoft/mana/shm_channel.c |   2 +-
 .../microsoft => include/net}/mana/gdma.h     | 162 ++++-
 .../net}/mana/hw_channel.h                    |   0
 .../microsoft => include/net}/mana/mana.h     |  23 +-
 include/net/mana/mana_auxiliary.h             |  10 +
 .../net}/mana/shm_channel.h                   |   0
 include/uapi/rdma/ib_user_ioctl_verbs.h       |   1 +
 include/uapi/rdma/mana-abi.h                  |  66 +++
 25 files changed, 2196 insertions(+), 45 deletions(-)
 create mode 100644 drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c
 create mode 100644 drivers/infiniband/hw/mana/device.c
 create mode 100644 drivers/infiniband/hw/mana/main.c
 create mode 100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c
 create mode 100644 drivers/infiniband/hw/mana/qp.c
 create mode 100644 drivers/infiniband/hw/mana/wq.c
 rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h (79%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h (94%)
 create mode 100644 include/net/mana/mana_auxiliary.h
 rename {drivers/net/ethernet/microsoft => include/net}/mana/shm_channel.h (100%)
 create mode 100644 include/uapi/rdma/mana-abi.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 1/12 net: mana: Add support for auxiliary device

From: Long Li <[email protected]>

In preparation for supporting MANA RDMA driver, add support for auxiliary device in the Ethernet driver. The RDMA device is modeled as an auxiliary device to the Ethernet device.

---
Change log:
v3: define mana_adev_idx_alloc and mana_adev_idx_free as static

 drivers/net/ethernet/microsoft/mana/gdma.h    |  2 +
 .../ethernet/microsoft/mana/mana_auxiliary.h  | 10 +++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 84 ++++++++++++++++++-
 3 files changed, 95 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/microsoft/mana/mana_auxiliary.h
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
index 41ecd156e95f..d815d323be87 100644
@@ -204,6 +204,8 @@ struct gdma_dev {
 
 	/* GDMA driver specific pointer */
 	void *driver_data;
+
+	struct auxiliary_device *adev;
 };
 
 #define MINIMUM_SUPPORTED_PAGE_SIZE PAGE_SIZE
drivers/net/ethernet/microsoft/mana/mana_auxiliary.h new file +10 -0
new file mode 100644
index 000000000000..373d59756846
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2022, Microsoft Corporation. */
+
+#include "mana.h"
+#include <linux/auxiliary_bus.h>
+
+struct mana_adev {
+	struct auxiliary_device adev;
+	struct gdma_dev *mdev;
+};
drivers/net/ethernet/microsoft/mana/mana_en.c +83 -1

114 lines not on this page · raw message

Signature · 1 line
2.17.1
#3 lore raw

PATCH v5 2/12 net: mana: Record the physical address for doorbell page region

In reply to [email protected]

From: Long Li <[email protected]>

For supporting RDMA device with multiple user contexts with their individual doorbell pages, record the start address of doorbell page region for use by the RDMA driver to allocate user context doorbell IDs.

---
 drivers/net/ethernet/microsoft/mana/gdma.h      | 2 ++
 drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 ++++
 2 files changed, 6 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
index d815d323be87..c724ca410fcb 100644
@@ -350,9 +350,11 @@ struct gdma_context {
 	struct completion	eq_test_event;
 	u32			test_event_eq_id;
 
+	phys_addr_t		bar0_pa;
 	void __iomem		*bar0_va;
 	void __iomem		*shm_base;
 	void __iomem		*db_page_base;
+	phys_addr_t		phys_db_page_base;
 	u32 db_page_size;
 
 	/* Shared memory chanenl (used to bootstrap HWC) */
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -0
index 49b85ca578b0..9fafaa0c8e76 100644
@@ -27,6 +27,9 @@ static void mana_gd_init_registers(struct pci_dev *pdev)
 	gc->db_page_base = gc->bar0_va +
 				mana_gd_r64(gc, GDMA_REG_DB_PAGE_OFFSET);
 
+	gc->phys_db_page_base = gc->bar0_pa +
+				mana_gd_r64(gc, GDMA_REG_DB_PAGE_OFFSET);
+
 	gc->shm_base = gc->bar0_va + mana_gd_r64(gc, GDMA_REG_SHM_OFFSET);
 }
 
@@ -1335,6 +1338,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	mutex_init(&gc->eq_test_event_mutex);
 	pci_set_drvdata(pdev, gc);
+	gc->bar0_pa = pci_resource_start(pdev, 0);
 
 	bar0_va = pci_iomap(pdev, bar, 0);
 	if (!bar0_va)
Signature · 1 line
2.17.1
#5 lore raw

PATCH v5 3/12 net: mana: Handle vport sharing between devices

In reply to [email protected]

From: Long Li <[email protected]>

For outgoing packets, the PF requires the VF to configure the vport with corresponding protection domain and doorbell ID for the kernel or user context. The vport can't be shared between different contexts.

Implement the logic to exclusively take over the vport by either the Ethernet device or RDMA device.

---
Change log:
v2: use refcount instead of directly using atomic variables
v4: change to mutex to avoid possible race with refcount
v5: add detailed comments explaining vport sharing, use EXPORT_SYMBOL_NS

 drivers/net/ethernet/microsoft/mana/mana.h    |  7 +++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 54 ++++++++++++++++++-
 2 files changed, 59 insertions(+), 2 deletions(-)
drivers/net/ethernet/microsoft/mana/mana.h +7 -0
index d36405af9432..2643036ee3e0 100644
@@ -375,6 +375,10 @@ struct mana_port_context {
 
 	mana_handle_t port_handle;
 
+	/* Mutex for sharing access to vport_use_count */
+	struct mutex vport_mutex;
+	int vport_use_count;
+
 	u16 port_idx;
 
 	bool port_is_up;
@@ -561,4 +565,7 @@ struct mana_tx_package {
 	struct gdma_posted_wqe_info wqe_info;
 };
 
+int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
+		   u32 doorbell_pg_id);
+void mana_uncfg_vport(struct mana_port_context *apc);
 #endif /* _MANA_H */
drivers/net/ethernet/microsoft/mana/mana_en.c +52 -2

92 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw

PATCH v5 4/12 net: mana: Add functions for allocating doorbell page from GDMA

In reply to [email protected]

From: Long Li <[email protected]>

The RDMA device needs to allocate doorbell pages for each user context. Implement those functions and expose them for use by the RDMA driver.

---
Change log:
v4: use EXPORT_SYMBOL_NS

 drivers/net/ethernet/microsoft/mana/gdma.h    | 29 ++++++++++
 .../net/ethernet/microsoft/mana/gdma_main.c   | 56 +++++++++++++++++++
 2 files changed, 85 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma.h +29 -0
index c724ca410fcb..f945755760dc 100644
@@ -22,11 +22,15 @@ enum gdma_request_type {
 	GDMA_GENERATE_TEST_EQE		= 10,
 	GDMA_CREATE_QUEUE		= 12,
 	GDMA_DISABLE_QUEUE		= 13,
+	GDMA_ALLOCATE_RESOURCE_RANGE	= 22,
+	GDMA_DESTROY_RESOURCE_RANGE	= 24,
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
 };
 
+#define GDMA_RESOURCE_DOORBELL_PAGE	27
+
 enum gdma_queue_type {
 	GDMA_INVALID_QUEUE,
 	GDMA_SQ,
@@ -568,6 +572,26 @@ struct gdma_register_device_resp {
 	u32 db_id;
 }; /* HW DATA */
 
+struct gdma_allocate_resource_range_req {
+	struct gdma_req_hdr hdr;
+	u32 resource_type;
+	u32 num_resources;
+	u32 alignment;
+	u32 allocated_resources;
+};
+
+struct gdma_allocate_resource_range_resp {
+	struct gdma_resp_hdr hdr;
+	u32 allocated_resources;
+};
+
+struct gdma_destroy_resource_range_req {
+	struct gdma_req_hdr hdr;
+	u32 resource_type;
+	u32 num_resources;
+	u32 allocated_resources;
+};
+
 /* GDMA_CREATE_QUEUE */
 struct gdma_create_queue_req {
 	struct gdma_req_hdr hdr;
@@ -676,4 +700,9 @@ void mana_gd_free_memory(struct gdma_mem_info *gmi);
 
 int mana_gd_send_request(struct gdma_context *gc, u32 req_len, const void *req,
 			 u32 resp_len, void *resp);
+
+int mana_gd_allocate_doorbell_page(struct gdma_context *gc, int *doorbell_page);
+
+int mana_gd_destroy_doorbell_page(struct gdma_context *gc, int doorbell_page);
+
 #endif /* _GDMA_H */
drivers/net/ethernet/microsoft/mana/gdma_main.c +56 -0

62 lines not on this page · raw message

Signature · 1 line
2.17.1
#9 lore raw

PATCH v5 5/12 net: mana: Set the DMA device max segment size

In reply to [email protected]

From: Ajay Sharma <[email protected]>

MANA hardware doesn't have any restrictions on the DMA segment size, set it to the max allowed value.

---
Change log:
v2: Use the max allowed value as the hardware doesn't have any limit

 drivers/net/ethernet/microsoft/mana/gdma_main.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma_main.c +6 -0
index 3c391b0cfab4..9a1885e47766 100644
@@ -1387,6 +1387,12 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto release_region;
 
+	err = dma_set_max_seg_size(&pdev->dev, UINT_MAX);
+	if (err) {
+		dev_err(&pdev->dev, "Failed to set dma device segment size\n");
+		goto release_region;
+	}
+
 	err = -ENOMEM;
 	gc = vzalloc(sizeof(*gc));
 	if (!gc)
Signature · 1 line
2.17.1
#21 lore raw

PATCH v5 11/12 net: mana: Define data structures for protection domain and memory registration

In reply to [email protected]

From: Ajay Sharma <[email protected]>

The MANA hardware support protection domain and memory registration for use in RDMA environment. Add those definitions and expose them for use by the RDMA driver.

---
Change log:
v3: format/coding style changes
v5: remove unused code dealing with kernel-mode MR

 .../net/ethernet/microsoft/mana/gdma_main.c   |  27 ++--
 drivers/net/ethernet/microsoft/mana/mana_en.c |  18 +--
 include/net/mana/gdma.h                       | 120 +++++++++++++++++-
 3 files changed, 142 insertions(+), 23 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +17 -10
index 930e7f4a1dad..b65218cc19ab 100644
@@ -227,7 +227,7 @@ static int mana_gd_create_hw_eq(struct gdma_context *gc,
 	req.type = queue->type;
 	req.pdid = queue->gdma_dev->pdid;
 	req.doolbell_id = queue->gdma_dev->doorbell;
-	req.gdma_region = queue->mem_info.gdma_region;
+	req.gdma_region = queue->mem_info.dma_region_handle;
 	req.queue_size = queue->queue_size;
 	req.log2_throttle_limit = queue->eq.log2_throttle_limit;
 	req.eq_pci_msix_index = queue->eq.msix_index;
@@ -241,7 +241,7 @@ static int mana_gd_create_hw_eq(struct gdma_context *gc,
 
 	queue->id = resp.queue_index;
 	queue->eq.disable_needed = true;
-	queue->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+	queue->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
 	return 0;
 }
 
@@ -695,24 +695,30 @@ int mana_gd_create_hwc_queue(struct gdma_dev *gd,
 	return err;
 }
 
-static void mana_gd_destroy_dma_region(struct gdma_context *gc, u64 gdma_region)
+int mana_gd_destroy_dma_region(struct gdma_context *gc,
+			       gdma_obj_handle_t dma_region_handle)
 {
 	struct gdma_destroy_dma_region_req req = {};
 	struct gdma_general_resp resp = {};
 	int err;
 
-	if (gdma_region == GDMA_INVALID_DMA_REGION)
-		return;
+	if (dma_region_handle == GDMA_INVALID_DMA_REGION)
+		return 0;
 
 	mana_gd_init_req_hdr(&req.hdr, GDMA_DESTROY_DMA_REGION, sizeof(req),
 			     sizeof(resp));
-	req.gdma_region = gdma_region;
+	req.dma_region_handle = dma_region_handle;
 
 	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
-	if (err || resp.hdr.status)
+	if (err || resp.hdr.status) {
 		dev_err(gc->dev, "Failed to destroy DMA region: %d, 0x%x\n",
 			err, resp.hdr.status);
+		return -EPROTO;
+	}
+
+	return 0;
 }
+EXPORT_SYMBOL_NS(mana_gd_destroy_dma_region, NET_MANA);
 
 static int mana_gd_create_dma_region(struct gdma_dev *gd,
 				     struct gdma_mem_info *gmi)
@@ -757,14 +763,15 @@ static int mana_gd_create_dma_region(struct gdma_dev *gd,
 	if (err)
 		goto out;
 
-	if (resp.hdr.status || resp.gdma_region == GDMA_INVALID_DMA_REGION) {
+	if (resp.hdr.status ||
+	    resp.dma_region_handle == GDMA_INVALID_DMA_REGION) {
 		dev_err(gc->dev, "Failed to create DMA region: 0x%x\n",
 			resp.hdr.status);
 		err = -EPROTO;
 		goto out;
 	}
 
-	gmi->gdma_region = resp.gdma_region;
+	gmi->dma_region_handle = resp.dma_region_handle;
 out:
 	kfree(req);
 	return err;
@@ -887,7 +894,7 @@ void mana_gd_destroy_queue(struct gdma_context *gc, struct gdma_queue *queue)
 		return;
 	}
 
-	mana_gd_destroy_dma_region(gc, gmi->gdma_region);
+	mana_gd_destroy_dma_region(gc, gmi->dma_region_handle);
 	mana_gd_free_memory(gmi);
 	kfree(queue);
 }
drivers/net/ethernet/microsoft/mana/mana_en.c +10 -8
index 2ca2a63afe6c..3b17ea45ad26 100644
@@ -1399,10 +1399,10 @@ static int mana_create_txq(struct mana_port_context *apc,
 		memset(&wq_spec, 0, sizeof(wq_spec));
 		memset(&cq_spec, 0, sizeof(cq_spec));
 
-		wq_spec.gdma_region = txq->gdma_sq->mem_info.gdma_region;
+		wq_spec.gdma_region = txq->gdma_sq->mem_info.dma_region_handle;
 		wq_spec.queue_size = txq->gdma_sq->queue_size;
 
-		cq_spec.gdma_region = cq->gdma_cq->mem_info.gdma_region;
+		cq_spec.gdma_region = cq->gdma_cq->mem_info.dma_region_handle;
 		cq_spec.queue_size = cq->gdma_cq->queue_size;
 		cq_spec.modr_ctx_id = 0;
 		cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
@@ -1417,8 +1417,10 @@ static int mana_create_txq(struct mana_port_context *apc,
 		txq->gdma_sq->id = wq_spec.queue_index;
 		cq->gdma_cq->id = cq_spec.queue_index;
 
-		txq->gdma_sq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
-		cq->gdma_cq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+		txq->gdma_sq->mem_info.dma_region_handle =
+			GDMA_INVALID_DMA_REGION;
+		cq->gdma_cq->mem_info.dma_region_handle =
+			GDMA_INVALID_DMA_REGION;
 
 		txq->gdma_txq_id = txq->gdma_sq->id;
 
@@ -1629,10 +1631,10 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
 
 	memset(&wq_spec, 0, sizeof(wq_spec));
 	memset(&cq_spec, 0, sizeof(cq_spec));
-	wq_spec.gdma_region = rxq->gdma_rq->mem_info.gdma_region;
+	wq_spec.gdma_region = rxq->gdma_rq->mem_info.dma_region_handle;
 	wq_spec.queue_size = rxq->gdma_rq->queue_size;
 
-	cq_spec.gdma_region = cq->gdma_cq->mem_info.gdma_region;
+	cq_spec.gdma_region = cq->gdma_cq->mem_info.dma_region_handle;
 	cq_spec.queue_size = cq->gdma_cq->queue_size;
 	cq_spec.modr_ctx_id = 0;
 	cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
@@ -1645,8 +1647,8 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
 	rxq->gdma_rq->id = wq_spec.queue_index;
 	cq->gdma_cq->id = cq_spec.queue_index;
 
-	rxq->gdma_rq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
-	cq->gdma_cq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+	rxq->gdma_rq->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
+	cq->gdma_cq->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
 
 	rxq->gdma_id = rxq->gdma_rq->id;
 	cq->gdma_id = cq->gdma_cq->id;
include/net/mana/gdma.h +115 -5

174 lines not on this page · raw message

Signature · 1 line
2.17.1
#13 lore raw

PATCH v5 7/12 net: mana: Record port number in netdev

In reply to [email protected]

From: Long Li <[email protected]>

The port number is useful for user-mode application to identify this net device based on port index. Set to the correct value in ndev.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -0
index 4c771ce59108..eeddd4ab2fa4 100644
@@ -1988,6 +1988,7 @@ static int mana_probe_port(struct mana_context *ac, int port_idx,
 	ndev->max_mtu = ndev->mtu;
 	ndev->min_mtu = ndev->mtu;
 	ndev->needed_headroom = MANA_HEADROOM;
+	ndev->dev_port = port_idx;
 	SET_NETDEV_DEV(ndev, gc->dev);
 
 	netif_carrier_off(ndev);
Signature · 1 line
2.17.1
#11 lore raw

PATCH v5 6/12 net: mana: Export Work Queue functions for use by RDMA driver

In reply to [email protected]

From: Long Li <[email protected]>

RDMA device may need to create Ethernet device queues for use by Queue Pair type RAW. This allows a user-mode context accesses Ethernet hardware queues. Export the supporting functions for use by the RDMA driver.

---
Change log:
v3: format/coding style changes
v5: remove unused defintions, use EXPORT_SYMBOL_NS, rearrange some defintions to a later patch in the series

 drivers/net/ethernet/microsoft/mana/gdma_main.c |  1 +
 drivers/net/ethernet/microsoft/mana/mana.h      |  9 +++++++++
 drivers/net/ethernet/microsoft/mana/mana_en.c   | 16 +++++++++-------
 3 files changed, 19 insertions(+), 7 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -0
index 9a1885e47766..2bd9bf624eb3 100644
@@ -125,6 +125,7 @@ int mana_gd_send_request(struct gdma_context *gc, u32 req_len, const void *req,
 
 	return mana_hwc_send_request(hwc, req_len, req, resp_len, resp);
 }
+EXPORT_SYMBOL_NS(mana_gd_send_request, NET_MANA);
 
 int mana_gd_alloc_memory(struct gdma_context *gc, unsigned int length,
 			 struct gdma_mem_info *gmi)
drivers/net/ethernet/microsoft/mana/mana.h +9 -0
index 2643036ee3e0..f0902f83045f 100644
@@ -565,6 +565,15 @@ struct mana_tx_package {
 	struct gdma_posted_wqe_info wqe_info;
 };
 
+int mana_create_wq_obj(struct mana_port_context *apc,
+		       mana_handle_t vport,
+		       u32 wq_type, struct mana_obj_spec *wq_spec,
+		       struct mana_obj_spec *cq_spec,
+		       mana_handle_t *wq_obj);
+
+void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
+			 mana_handle_t wq_obj);
+
 int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
 		   u32 doorbell_pg_id);
 void mana_uncfg_vport(struct mana_port_context *apc);
drivers/net/ethernet/microsoft/mana/mana_en.c +9 -7
index f0bd0efd0b6a..4c771ce59108 100644
@@ -677,11 +677,11 @@ static int mana_cfg_vport_steering(struct mana_port_context *apc,
 	return err;
 }
 
-static int mana_create_wq_obj(struct mana_port_context *apc,
-			      mana_handle_t vport,
-			      u32 wq_type, struct mana_obj_spec *wq_spec,
-			      struct mana_obj_spec *cq_spec,
-			      mana_handle_t *wq_obj)
+int mana_create_wq_obj(struct mana_port_context *apc,
+		       mana_handle_t vport,
+		       u32 wq_type, struct mana_obj_spec *wq_spec,
+		       struct mana_obj_spec *cq_spec,
+		       mana_handle_t *wq_obj)
 {
 	struct mana_create_wqobj_resp resp = {};
 	struct mana_create_wqobj_req req = {};
@@ -730,9 +730,10 @@ static int mana_create_wq_obj(struct mana_port_context *apc,
 out:
 	return err;
 }
+EXPORT_SYMBOL_NS(mana_create_wq_obj, NET_MANA);
 
-static void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
-				mana_handle_t wq_obj)
+void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
+			 mana_handle_t wq_obj)
 {
 	struct mana_destroy_wqobj_resp resp = {};
 	struct mana_destroy_wqobj_req req = {};
@@ -757,6 +758,7 @@ static void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
 		netdev_err(ndev, "Failed to destroy WQ object: %d, 0x%x\n", err,
 			   resp.hdr.status);
 }
+EXPORT_SYMBOL_NS(mana_destroy_wq_obj, NET_MANA);
 
 static void mana_destroy_eq(struct mana_context *ac)
 {
Signature · 1 line
2.17.1
#19 lore raw

PATCH v5 10/12 net: mana: Define and process GDMA response code GDMA_STATUS_MORE_ENTRIES

In reply to [email protected]

From: Ajay Sharma <[email protected]>

When doing memory registration, the PF may respond with GDMA_STATUS_MORE_ENTRIES to indicate a follow request is needed. This is not an error and should be processed as expected.

---
 drivers/net/ethernet/microsoft/mana/hw_channel.c | 2 +-
 include/net/mana/gdma.h                          | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/microsoft/mana/hw_channel.c +1 -1
index e61cb3f6fbe1..24ecd193a185 100644
@@ -820,7 +820,7 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 		goto out;
 	}
 
-	if (ctx->status_code) {
+	if (ctx->status_code && ctx->status_code != GDMA_STATUS_MORE_ENTRIES) {
 		dev_err(hwc->dev, "HWC: Failed hw_channel req: 0x%x\n",
 			ctx->status_code);
 		err = -EPROTO;
include/net/mana/gdma.h +2 -0
index e9ad03ccc98b..2442e72628d9 100644
@@ -9,6 +9,8 @@
 
 #include "shm_channel.h"
 
+#define GDMA_STATUS_MORE_ENTRIES	0x00000105
+
 /* Structures labeled with "HW DATA" are exchanged with the hardware. All of
  * them are naturally aligned and hence don't need __packed.
  */
Signature · 1 line
2.17.1
#17 lore raw

PATCH v5 9/12 net: mana: Define max values for SGL entries

In reply to [email protected]

From: Long Li <[email protected]>

The number of maximum SGl entries should be computed from the maximum WQE size for the intended queue type and the corresponding OOB data size. This guarantees the hardware queue can successfully queue requests up to the queue depth exposed to the upper layer.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 2 +-
 include/net/mana/gdma.h                       | 7 +++++++
 include/net/mana/mana.h                       | 4 +---
 3 files changed, 9 insertions(+), 4 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -1
index ba776f73d7c9..2ca2a63afe6c 100644
@@ -188,7 +188,7 @@ int mana_start_xmit(struct sk_buff *skb, struct net_device *ndev)
 	pkg.wqe_req.client_data_unit = 0;
 
 	pkg.wqe_req.num_sge = 1 + skb_shinfo(skb)->nr_frags;
-	WARN_ON_ONCE(pkg.wqe_req.num_sge > 30);
+	WARN_ON_ONCE(pkg.wqe_req.num_sge > MAX_TX_WQE_SGL_ENTRIES);
 
 	if (pkg.wqe_req.num_sge <= ARRAY_SIZE(pkg.sgl_array)) {
 		pkg.wqe_req.sgl = pkg.sgl_array;
include/net/mana/gdma.h +7 -0
index f945755760dc..e9ad03ccc98b 100644
@@ -430,6 +430,13 @@ struct gdma_wqe {
 #define MAX_TX_WQE_SIZE 512
 #define MAX_RX_WQE_SIZE 256
 
+#define MAX_TX_WQE_SGL_ENTRIES	((GDMA_MAX_SQE_SIZE -			   \
+			sizeof(struct gdma_sge) - INLINE_OOB_SMALL_SIZE) / \
+			sizeof(struct gdma_sge))
+
+#define MAX_RX_WQE_SGL_ENTRIES	((GDMA_MAX_RQE_SIZE -			   \
+			sizeof(struct gdma_sge)) / sizeof(struct gdma_sge))
+
 struct gdma_cqe {
 	u32 cqe_data[GDMA_COMP_DATA_SIZE / 4];
 
include/net/mana/mana.h +1 -3
index f0902f83045f..5ad548047ce0 100644
@@ -263,8 +263,6 @@ struct mana_cq {
 	int budget;
 };
 
-#define GDMA_MAX_RQE_SGES 15
-
 struct mana_recv_buf_oob {
 	/* A valid GDMA work request representing the data buffer. */
 	struct gdma_wqe_request wqe_req;
@@ -274,7 +272,7 @@ struct mana_recv_buf_oob {
 
 	/* SGL of the buffer going to be sent has part of the work request. */
 	u32 num_sge;
-	struct gdma_sge sgl[GDMA_MAX_RQE_SGES];
+	struct gdma_sge sgl[MAX_RX_WQE_SGL_ENTRIES];
 
 	/* Required to store the result of mana_gd_post_work_request.
 	 * gdma_posted_wqe_info.wqe_size_in_bu is required for progressing the
Signature · 1 line
2.17.1
#24 lore raw

PATCH v5 12/12 RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]

From: Long Li <[email protected]>

Add a RDMA VF driver for Microsoft Azure Network Adapter (MANA).

---
Change log:
v2:
Changed coding sytles/formats
Checked undersize for udata length
Changed all logging to use ibdev_xxx()
Avoided page array copy when doing MR
Sorted driver ops
Fixed warnings reported by kernel test robot <[email protected]>

v3:
More coding sytle/format changes

v4:
Process error on hardware vport configuration

v5:
Change licenses to GPL-2.0-only
Fix error handling in mana_ib_gd_create_dma_region()

 MAINTAINERS                             |   3 +
 drivers/infiniband/Kconfig              |   1 +
 drivers/infiniband/hw/Makefile          |   1 +
 drivers/infiniband/hw/mana/Kconfig      |   7 +
 drivers/infiniband/hw/mana/Makefile     |   4 +
 drivers/infiniband/hw/mana/cq.c         |  80 ++++
 drivers/infiniband/hw/mana/device.c     | 129 ++++++
 drivers/infiniband/hw/mana/main.c       | 555 ++++++++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h    | 165 +++++++
 drivers/infiniband/hw/mana/mr.c         | 133 ++++++
 drivers/infiniband/hw/mana/qp.c         | 501 +++++++++++++++++++++
 drivers/infiniband/hw/mana/wq.c         | 114 +++++
 include/net/mana/mana.h                 |   3 +
 include/uapi/rdma/ib_user_ioctl_verbs.h |   1 +
 include/uapi/rdma/mana-abi.h            |  66 +++
 15 files changed, 1763 insertions(+)
 create mode 100644 drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c
 create mode 100644 drivers/infiniband/hw/mana/device.c
 create mode 100644 drivers/infiniband/hw/mana/main.c
 create mode 100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c
 create mode 100644 drivers/infiniband/hw/mana/qp.c
 create mode 100644 drivers/infiniband/hw/mana/wq.c
 create mode 100644 include/uapi/rdma/mana-abi.h
MAINTAINERS +3 -0
index 51bec6d5076d..1bed8444786d 100644
@@ -9078,6 +9078,7 @@ M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
 M:	Dexuan Cui <[email protected]>
+M:	Long Li <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -9095,6 +9096,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
 F:	drivers/clocksource/hyperv_timer.c
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
+F:	drivers/infiniband/hw/mana/
 F:	drivers/input/serio/hyperv-keyboard.c
 F:	drivers/iommu/hyperv-iommu.c
 F:	drivers/net/ethernet/microsoft/
@@ -9110,6 +9112,7 @@ F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
 F:	include/net/mana
 F:	include/uapi/linux/hyperv.h
+F:	include/uapi/rdma/mana-abi.h
 F:	net/vmw_vsock/hyperv_transport.c
 F:	tools/hv/
 
drivers/infiniband/Kconfig +1 -0
index 33d3ce9c888e..a062c662ecff 100644
@@ -83,6 +83,7 @@ source "drivers/infiniband/hw/qib/Kconfig"
 source "drivers/infiniband/hw/cxgb4/Kconfig"
 source "drivers/infiniband/hw/efa/Kconfig"
 source "drivers/infiniband/hw/irdma/Kconfig"
+source "drivers/infiniband/hw/mana/Kconfig"
 source "drivers/infiniband/hw/mlx4/Kconfig"
 source "drivers/infiniband/hw/mlx5/Kconfig"
 source "drivers/infiniband/hw/ocrdma/Kconfig"
drivers/infiniband/hw/Makefile +1 -0
index fba0b3be903e..f62e9e00c780 100644
@@ -4,6 +4,7 @@ obj-$(CONFIG_INFINIBAND_QIB)		+= qib/
 obj-$(CONFIG_INFINIBAND_CXGB4)		+= cxgb4/
 obj-$(CONFIG_INFINIBAND_EFA)		+= efa/
 obj-$(CONFIG_INFINIBAND_IRDMA)		+= irdma/
+obj-$(CONFIG_MANA_INFINIBAND)		+= mana/
 obj-$(CONFIG_MLX4_INFINIBAND)		+= mlx4/
 obj-$(CONFIG_MLX5_INFINIBAND)		+= mlx5/
 obj-$(CONFIG_INFINIBAND_OCRDMA)		+= ocrdma/
drivers/infiniband/hw/mana/Kconfig new file +7 -0
new file mode 100644
index 000000000000..b3ff03a23257
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config MANA_INFINIBAND
+	tristate "Microsoft Azure Network Adapter support"
+	depends on NETDEVICES && ETHERNET && PCI && MICROSOFT_MANA
+	help
+	  This driver provides low-level RDMA support for
+	  Microsoft Azure Network Adapter (MANA).
drivers/infiniband/hw/mana/Makefile new file +4 -0
new file mode 100644
index 000000000000..88655fe5e398
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_MANA_INFINIBAND) += mana_ib.o
+
+mana_ib-y := device.o main.o wq.o qp.o cq.o mr.o
drivers/infiniband/hw/mana/cq.c new file +80 -0

80 lines not on this page · raw message

drivers/infiniband/hw/mana/device.c new file +129 -0

129 lines not on this page · raw message

drivers/infiniband/hw/mana/main.c new file +555 -0

555 lines not on this page · raw message

drivers/infiniband/hw/mana/mana_ib.h new file +165 -0

165 lines not on this page · raw message

drivers/infiniband/hw/mana/mr.c new file +133 -0

133 lines not on this page · raw message

drivers/infiniband/hw/mana/qp.c new file +501 -0

501 lines not on this page · raw message

drivers/infiniband/hw/mana/wq.c new file +114 -0

114 lines not on this page · raw message

include/net/mana/mana.h +3 -0
index 5ad548047ce0..9cf7bed5aafb 100644
@@ -405,6 +405,9 @@ int mana_bpf(struct net_device *ndev, struct netdev_bpf *bpf);
 
 extern const struct ethtool_ops mana_ethtool_ops;
 
+/* A CQ can be created not associated with any EQ */
+#define GDMA_CQ_NO_EQ  0xffff
+
 struct mana_obj_spec {
 	u32 queue_index;
 	u64 gdma_region;
include/uapi/rdma/ib_user_ioctl_verbs.h +1 -0
index 3072e5d6b692..081aabf536dc 100644
@@ -250,6 +250,7 @@ enum rdma_driver_id {
 	RDMA_DRIVER_QIB,
 	RDMA_DRIVER_EFA,
 	RDMA_DRIVER_SIW,
+	RDMA_DRIVER_MANA,
 };
 
 enum ib_uverbs_gid_type {
include/uapi/rdma/mana-abi.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.17.1
#15 lore raw

PATCH v5 8/12 net: mana: Move header files to a common location

In reply to [email protected]

From: Long Li <[email protected]>

In preparation to add MANA RDMA driver, move all the required header files to a common location for use by both Ethernet and RDMA drivers.

---
Change log:
v2: Move headers to include/net/mana, instead of include/linux/mana

 MAINTAINERS                                                   | 1 +
 drivers/net/ethernet/microsoft/mana/gdma_main.c               | 2 +-
 drivers/net/ethernet/microsoft/mana/hw_channel.c              | 4 ++--
 drivers/net/ethernet/microsoft/mana/mana_bpf.c                | 2 +-
 drivers/net/ethernet/microsoft/mana/mana_en.c                 | 4 ++--
 drivers/net/ethernet/microsoft/mana/mana_ethtool.c            | 2 +-
 drivers/net/ethernet/microsoft/mana/shm_channel.c             | 2 +-
 {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h   | 0
 .../net/ethernet/microsoft => include/net}/mana/hw_channel.h  | 0
 {drivers/net/ethernet/microsoft => include/net}/mana/mana.h   | 0
 .../ethernet/microsoft => include/net}/mana/mana_auxiliary.h  | 0
 .../net/ethernet/microsoft => include/net}/mana/shm_channel.h | 0
 12 files changed, 9 insertions(+), 8 deletions(-)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana_auxiliary.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/shm_channel.h (100%)
MAINTAINERS +1 -0
index 40fa1955ca3f..51bec6d5076d 100644
@@ -9108,6 +9108,7 @@ F:	include/asm-generic/hyperv-tlfs.h
 F:	include/asm-generic/mshyperv.h
 F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
+F:	include/net/mana
 F:	include/uapi/linux/hyperv.h
 F:	net/vmw_vsock/hyperv_transport.c
 F:	tools/hv/
drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -1
index 2bd9bf624eb3..930e7f4a1dad 100644
@@ -6,7 +6,7 @@
 #include <linux/utsname.h>
 #include <linux/version.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 static u32 mana_gd_r32(struct gdma_context *g, u64 offset)
 {
drivers/net/ethernet/microsoft/mana/hw_channel.c +2 -2
index 078d6a5a0768..e61cb3f6fbe1 100644
@@ -1,8 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
 /* Copyright (c) 2021, Microsoft Corporation. */
 
-#include "gdma.h"
-#include "hw_channel.h"
+#include <net/mana/gdma.h>
+#include <net/mana/hw_channel.h>
 
 static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id)
 {
drivers/net/ethernet/microsoft/mana/mana_bpf.c +1 -1
index 1d2f948b5c00..97a79659af45 100644
@@ -8,7 +8,7 @@
 #include <linux/bpf_trace.h>
 #include <net/xdp.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 void mana_xdp_tx(struct sk_buff *skb, struct net_device *ndev)
 {
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
index eeddd4ab2fa4..ba776f73d7c9 100644
@@ -11,8 +11,8 @@
 #include <net/checksum.h>
 #include <net/ip6_checksum.h>
 
-#include "mana.h"
-#include "mana_auxiliary.h"
+#include <net/mana/mana.h>
+#include <net/mana/mana_auxiliary.h>
 
 static DEFINE_IDA(mana_adev_ida);
 
drivers/net/ethernet/microsoft/mana/mana_ethtool.c +1 -1
index e13f2453eabb..ebc6595d02fe 100644
@@ -5,7 +5,7 @@
 #include <linux/etherdevice.h>
 #include <linux/ethtool.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 static const struct {
 	char name[ETH_GSTRING_LEN];
drivers/net/ethernet/microsoft/mana/shm_channel.c +1 -1
index da255da62176..5553af9c8085 100644
@@ -6,7 +6,7 @@
 #include <linux/io.h>
 #include <linux/mm.h>
 
-#include "shm_channel.h"
+#include <net/mana/shm_channel.h>
 
 #define PAGE_FRAME_L48_WIDTH_BYTES 6
 #define PAGE_FRAME_L48_WIDTH_BITS (PAGE_FRAME_L48_WIDTH_BYTES * 8)
include/net/mana/gdma.h renamed from drivers/net/ethernet/microsoft/mana/gdma.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/gdma.h
rename to include/net/mana/gdma.h
include/net/mana/hw_channel.h renamed from drivers/net/ethernet/microsoft/mana/hw_channel.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/hw_channel.h
rename to include/net/mana/hw_channel.h
include/net/mana/mana.h renamed from drivers/net/ethernet/microsoft/mana/mana.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/mana.h
rename to include/net/mana/mana.h
include/net/mana/mana_auxiliary.h renamed from drivers/net/ethernet/microsoft/mana/mana_auxiliary.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/mana_auxiliary.h
rename to include/net/mana/mana_auxiliary.h
include/net/mana/shm_channel.h renamed from drivers/net/ethernet/microsoft/mana/shm_channel.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/shm_channel.h
rename to include/net/mana/shm_channel.h
Signature · 1 line
2.17.1
#29 lore raw
In reply to [email protected]
Quoted 84 lines

Subject: [Patch v5 00/12] Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

From: Long Li <[email protected]>

This patchset implements a RDMA driver for Microsoft Azure Network Adapter (MANA). In MANA, the RDMA device is modeled as an auxiliary device to the Ethernet device.

The first 11 patches modify the MANA Ethernet driver to support RDMA driver.
The last patch implementes the RDMA driver.

The user-mode of the driver is being reviewed at:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2Flinux-rdma%2Frdma- core%2Fpull%2F1177&amp;data=05%7C01%7Clongli%40microsoft.com%7Cc1 6f986dc7c34750c1de08da8ae896d4%7C72f988bf86f141af91ab2d7cd011db47% 7C1%7C0%7C637975028849495970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300 0%7C%7C%7C&amp;sdata=VoflZKD8KVbKxpv%2BH8w4rgsrlOlv5NAjul6K6fuD 9jY%3D&amp;reserved=0

Ajay Sharma (3):
net: mana: Set the DMA device max segment size
net: mana: Define and process GDMA response code
GDMA_STATUS_MORE_ENTRIES
net: mana: Define data structures for protection domain and memory
registration

Long Li (9):
net: mana: Add support for auxiliary device
net: mana: Record the physical address for doorbell page region
net: mana: Handle vport sharing between devices
net: mana: Add functions for allocating doorbell page from GDMA
net: mana: Export Work Queue functions for use by RDMA driver
net: mana: Record port number in netdev
net: mana: Move header files to a common location
net: mana: Define max values for SGL entries RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

 MAINTAINERS                                   |   4 +
 drivers/infiniband/Kconfig                    |   1 +
 drivers/infiniband/hw/Makefile                |   1 +
 drivers/infiniband/hw/mana/Kconfig            |   7 +
 drivers/infiniband/hw/mana/Makefile           |   4 +
 drivers/infiniband/hw/mana/cq.c               |  80 +++
 drivers/infiniband/hw/mana/device.c           | 129 ++++
 drivers/infiniband/hw/mana/main.c             | 555 ++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h          | 165 ++++++
 drivers/infiniband/hw/mana/mr.c               | 133 +++++
 drivers/infiniband/hw/mana/qp.c               | 501 ++++++++++++++++
 drivers/infiniband/hw/mana/wq.c               | 114 ++++
 .../net/ethernet/microsoft/mana/gdma_main.c   |  96 ++-
 .../net/ethernet/microsoft/mana/hw_channel.c  |   6 +-
 .../net/ethernet/microsoft/mana/mana_bpf.c    |   2 +-
 drivers/net/ethernet/microsoft/mana/mana_en.c | 177 +++++-
 .../ethernet/microsoft/mana/mana_ethtool.c    |   2 +-
 .../net/ethernet/microsoft/mana/shm_channel.c |   2 +-
 .../microsoft => include/net}/mana/gdma.h     | 162 ++++-
 .../net}/mana/hw_channel.h                    |   0
 .../microsoft => include/net}/mana/mana.h     |  23 +-
 include/net/mana/mana_auxiliary.h             |  10 +
 .../net}/mana/shm_channel.h                   |   0
 include/uapi/rdma/ib_user_ioctl_verbs.h       |   1 +
 include/uapi/rdma/mana-abi.h                  |  66 +++
 25 files changed, 2196 insertions(+), 45 deletions(-)  create mode 100644
drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c  create mode 100644
drivers/infiniband/hw/mana/device.c
 create mode 100644 drivers/infiniband/hw/mana/main.c  create mode
100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c  create mode 100644

drivers/infiniband/hw/mana/qp.c create mode 100644 drivers/infiniband/hw/mana/wq.c rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h (79%) rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h (100%) rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h (94%) create mode 100644 include/net/mana/mana_auxiliary.h rename {drivers/net/ethernet/microsoft => include/net}/mana/shm_channel.h (100%) create mode 100644 include/uapi/rdma/mana-abi.h

Quoted 2 lines

--
2.17.1

Hi Jason,

Can you take a look at this patch set. I have addressed all the comments from previous review.

Please let me know if I should make any changes.

Thanks,
Long

#30 lore raw
Quoted 2 lines · Long Li

Can you take a look at this patch set. I have addressed all the comments from previous review.

The last time I looked I thought it was looking OK, I was thinking of putting it in linux-next for a while to get the static checkers happy. But the netdev patches gave me pause on that plan.

However, Leon and I will be at LPC all next week so I don't know if it will happen.

I would also like to see that the netdev patches are acked, and ask how you expect this cross-tree series to be merged?

Jason

#31 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 2 lines · Long Li

Can you take a look at this patch set. I have addressed all the comments from previous review.

The last time I looked I thought it was looking OK, I was thinking of putting it in linux-next for a while to get the static checkers happy. But the netdev patches gave me pause on that plan.

Quoted 5 lines

However, Leon and I will be at LPC all next week so I don't know if it will happen.

I would also like to see that the netdev patches are acked, and ask how you expect this cross-tree series to be merged?

??????

Quoted 2 lines

Jason

maintainer Reviewed-by on v5 11/12
#22 lore raw

RE: [Patch v5 11/12] net: mana: Define data structures for protection domain and memory registration

In reply to [email protected]
Quoted 5 lines

From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 5:35 PM
...
From: Ajay Sharma <[email protected]>

Quoted 7 lines

The MANA hardware support protection domain and memory registration for use
in RDMA environment. Add those definitions and expose them for use by the RDMA driver.

maintainer Reviewed-by on v5 12/12
#25 lore raw

RE: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]
Quoted 10 lines

From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 5:35 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang
...
From: Long Li <[email protected]>

Add a RDMA VF driver for Microsoft Azure Network Adapter (MANA).

...

The patch looks good to me. Just a few small nitpicks (see the below).

drivers/infiniband/hw/mana/device.c line 56

Quoted 8 lines
drivers/infiniband/hw/mana/device.c new file +8 -0
+static int mana_ib_probe(struct auxiliary_device *adev,
+			 const struct auxiliary_device_id *id)
+{
+	struct mana_adev *madev = container_of(adev, struct mana_adev, adev);
+	struct gdma_dev *mdev = madev->mdev;
+	struct mana_context *mc;
+	struct mana_ib_dev *dev;
+	int ret = 0;

Small nitpick: the 'ret' doesn't have to be initialized to 0.

drivers/infiniband/hw/mana/main.c

Quoted 8 lines

+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +5 -0
+				 mana_handle_t *gdma_region, u64 page_sz)
+{
...
+
+if (!err)
+	return 0;

Please add a Tab character to the above 2 lines.

#26 lore raw

RE: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 20 lines

Subject: RE: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 8 lines

From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 5:35 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang ...
From: Long Li <[email protected]>

Add a RDMA VF driver for Microsoft Azure Network Adapter (MANA).

The patch looks good to me. Just a few small nitpicks (see the below).

Review

Quoted 3 lines
(quoted diff) +3 -0
+static int mana_ib_probe(struct auxiliary_device *adev,
+			 const struct auxiliary_device_id *id) {
+	struct mana_adev *madev = container_of(adev, struct mana_adev,

adev);

Quoted 6 lines

drivers/infiniband/hw/mana/device.c line 56

Quoted 4 lines
drivers/infiniband/hw/mana/device.c new file +4 -0
+	struct gdma_dev *mdev = madev->mdev;
+	struct mana_context *mc;
+	struct mana_ib_dev *dev;
+	int ret = 0;

Small nitpick: the 'ret' doesn't have to be initialized to 0.

Will fix this.

Quoted 11 lines

drivers/infiniband/hw/mana/main.c line 156

Quoted 4 lines

+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +1 -0
+				 mana_handle_t *gdma_region, u64 page_sz)

{

drivers/infiniband/hw/mana/main.c line 312

Quoted 4 lines

...

drivers/infiniband/hw/mana/main.c new file +3 -0
+
+if (!err)
+	return 0;

Please add a Tab character to the above 2 lines.

Will fix this.

Thank you,
Long

maintainer Acked-by on v5 1/12
#2 lore raw

RE: [Patch v5 01/12] net: mana: Add support for auxiliary device

In reply to [email protected]
Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 01/12] net: mana: Add support for auxiliary device

Quoted 8 lines

From: Long Li <[email protected]>

In preparation for supporting MANA RDMA driver, add support for auxiliary device in the Ethernet driver. The RDMA device is modeled as an auxiliary device to the Ethernet device.

maintainer Acked-by on v5 2/12
#4 lore raw

RE: [Patch v5 02/12] net: mana: Record the physical address for doorbell page region

In reply to [email protected]
Quoted 19 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 02/12] net: mana: Record the physical address for doorbell page region

Quoted 8 lines

From: Long Li <[email protected]>

For supporting RDMA device with multiple user contexts with their individual doorbell pages, record the start address of doorbell page region for use by the RDMA driver to allocate user context doorbell IDs.

maintainer Acked-by on v5 3/12
#6 lore raw

RE: [Patch v5 03/12] net: mana: Handle vport sharing between devices

In reply to [email protected]
Quoted 24 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 03/12] net: mana: Handle vport sharing between devices

From: Long Li <[email protected]>

For outgoing packets, the PF requires the VF to configure the vport with corresponding protection domain and doorbell ID for the kernel or user context. The vport can't be shared between different contexts.

Quoted 5 lines

Implement the logic to exclusively take over the vport by either the Ethernet device or RDMA device.

maintainer Acked-by on v5 4/12
#8 lore raw

RE: [Patch v5 04/12] net: mana: Add functions for allocating doorbell page from GDMA

In reply to [email protected]
Quoted 19 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 04/12] net: mana: Add functions for allocating doorbell page from GDMA

Quoted 7 lines

From: Long Li <[email protected]>

The RDMA device needs to allocate doorbell pages for each user context. Implement those functions and expose them for use by the RDMA driver.

maintainer Acked-by on v5 5/12
#10 lore raw

RE: [Patch v5 05/12] net: mana: Set the DMA device max segment size

In reply to [email protected]
Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 05/12] net: mana: Set the DMA device max segment size

Quoted 8 lines

From: Ajay Sharma <[email protected]>

MANA hardware doesn't have any restrictions on the DMA segment size, set it to the max allowed value.

maintainer Acked-by on v5 6/12
#12 lore raw

RE: [Patch v5 06/12] net: mana: Export Work Queue functions for use by RDMA driver

In reply to [email protected]
Quoted 19 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 06/12] net: mana: Export Work Queue functions for use by RDMA driver

Quoted 8 lines

From: Long Li <[email protected]>

RDMA device may need to create Ethernet device queues for use by Queue Pair type RAW. This allows a user-mode context accesses Ethernet hardware queues. Export the supporting functions for use by the RDMA driver.

maintainer Acked-by on v5 7/12
#14 lore raw

RE: [Patch v5 07/12] net: mana: Record port number in netdev

In reply to [email protected]
Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 07/12] net: mana: Record port number in netdev

Quoted 7 lines

From: Long Li <[email protected]>

The port number is useful for user-mode application to identify this net device based on port index. Set to the correct value in ndev.

maintainer Acked-by on v5 8/12
#16 lore raw

RE: [Patch v5 08/12] net: mana: Move header files to a common location

In reply to [email protected]
Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 08/12] net: mana: Move header files to a common location

Quoted 7 lines

From: Long Li <[email protected]>

In preparation to add MANA RDMA driver, move all the required header files to a common location for use by both Ethernet and RDMA drivers.

maintainer Acked-by on v5 9/12
#18 lore raw

RE: [Patch v5 09/12] net: mana: Define max values for SGL entries

In reply to [email protected]
Quoted 20 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 09/12] net: mana: Define max values for SGL entries

From: Long Li <[email protected]>

Quoted 7 lines

The number of maximum SGl entries should be computed from the maximum WQE size for the intended queue type and the corresponding OOB data size. This guarantees the hardware queue can successfully queue requests up to the queue depth exposed to the upper layer.

maintainer Acked-by on v5 10/12
#20 lore raw

RE: [Patch v5 10/12] net: mana: Define and process GDMA response code GDMA_STATUS_MORE_ENTRIES

In reply to [email protected]
Quoted 21 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:34 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 10/12] net: mana: Define and process GDMA response code GDMA_STATUS_MORE_ENTRIES

From: Ajay Sharma <[email protected]>

Quoted 7 lines

When doing memory registration, the PF may respond with GDMA_STATUS_MORE_ENTRIES to indicate a follow request is needed. This is not an error and should be processed as expected.

maintainer Acked-by on v5 11/12
#23 lore raw

RE: [Patch v5 11/12] net: mana: Define data structures for protection domain and memory registration

In reply to [email protected]
Quoted 21 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, August 30, 2022 8:35 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; Long Li <[email protected]>
Subject: [Patch v5 11/12] net: mana: Define data structures for protection domain and memory registration

From: Ajay Sharma <[email protected]>

Quoted 7 lines

The MANA hardware support protection domain and memory registration for use
in RDMA environment. Add those definitions and expose them for use by the RDMA driver.

#32 lore raw
In reply to Leon Romanovsky
Quoted 13 lines

Subject: Re: [Patch v5 00/12] Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

Quoted 9 lines · Jason Gunthorpe
Quoted 2 lines · Long Li

Can you take a look at this patch set. I have addressed all the comments from previous review.

The last time I looked I thought it was looking OK, I was thinking of putting it in linux-next for a while to get the static checkers happy. But the netdev patches gave me pause on that plan.

Quoted 7 lines
Quoted 5 lines

However, Leon and I will be at LPC all next week so I don't know if it will happen.

I would also like to see that the netdev patches are acked, and ask how you expect this cross-tree series to be merged?

??????

I apologize for the late response.

The MANA ethernet maintainers have reviewed and acknowledged all the patches to the ethernet driver.

Can we merge the patch series through linux-rdma tree? I will rebase and send the v6 of the patch series for rdma-next.

Thanks,
Long

#27 lore raw

Re: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Dexuan Cui
Quoted 10 lines · Dexuan Cui

drivers/infiniband/hw/mana/main.c

Quoted 8 lines

+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +5 -0
+				 mana_handle_t *gdma_region, u64 page_sz)
+{
...
+
+if (!err)
+	return 0;

Please add a Tab character to the above 2 lines.

How are we still at the point where we have trivial style errors in this series at v6?? This is not OK, please handle reviews for this basic stuff internally.

Jason

#28 lore raw

RE: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 16 lines

Subject: Re: [Patch v5 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 3 lines · Dexuan Cui

drivers/infiniband/hw/mana/main.c line 156

Quoted 3 lines

+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +1 -0
+				 mana_handle_t *gdma_region, u64 page_sz)

{

Quoted 6 lines

drivers/infiniband/hw/mana/main.c line 312

Quoted 4 lines

...

drivers/infiniband/hw/mana/main.c new file +3 -0
+
+if (!err)
+	return 0;

Please add a Tab character to the above 2 lines.

Quoted 4 lines

How are we still at the point where we have trivial style errors in this series at v6?? This is not OK, please handle reviews for this basic stuff internally.

Jason

I'm sorry for this style error. We'll review everything.

Long