Skip to contentSkip to conversation

Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

PATCH Superseded [email protected] 8 participants 56 messages Jun 16, 2022 – Aug 3, 2022

Subsystems (5)

v4 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v4 cover Introduce Microsoft Azure Network Adapter (MANA) RDMA driver

From: Long Li <[email protected]>

This patchset implements a RDMA driver for Microsoft Azure Network Adapter (MANA). In MANA, the RDMA device is modeled as an auxiliary device to the Ethernet device.

The first 11 patches modify the MANA Ethernet driver to support RDMA driver. The last patch implementes the RDMA driver.

The user-mode of the driver is being reviewed at:
https://github.com/linux-rdma/rdma-core/pull/1177

Ajay Sharma (3):
net: mana: Set the DMA device max segment size
net: mana: Define data structures for protection domain and memory
registration
net: mana: Define and process GDMA response code
GDMA_STATUS_MORE_ENTRIES

Long Li (9):
net: mana: Add support for auxiliary device
net: mana: Record the physical address for doorbell page region
net: mana: Handle vport sharing between devices
net: mana: Add functions for allocating doorbell page from GDMA
net: mana: Export Work Queue functions for use by RDMA driver
net: mana: Record port number in netdev
net: mana: Move header files to a common location
net: mana: Define max values for SGL entries RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

 MAINTAINERS                                   |   4 +
 drivers/infiniband/Kconfig                    |   1 +
 drivers/infiniband/hw/Makefile                |   1 +
 drivers/infiniband/hw/mana/Kconfig            |   7 +
 drivers/infiniband/hw/mana/Makefile           |   4 +
 drivers/infiniband/hw/mana/cq.c               |  80 ++
 drivers/infiniband/hw/mana/main.c             | 681 ++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h          | 145 ++++
 drivers/infiniband/hw/mana/mr.c               | 133 ++++
 drivers/infiniband/hw/mana/qp.c               | 501 +++++++++++++
 drivers/infiniband/hw/mana/wq.c               | 114 +++
 .../net/ethernet/microsoft/mana/gdma_main.c   |  96 ++-
 .../net/ethernet/microsoft/mana/hw_channel.c  |   6 +-
 .../net/ethernet/microsoft/mana/mana_bpf.c    |   2 +-
 drivers/net/ethernet/microsoft/mana/mana_en.c | 162 ++++-
 .../ethernet/microsoft/mana/mana_ethtool.c    |   2 +-
 .../net/ethernet/microsoft/mana/shm_channel.c |   2 +-
 .../microsoft => include/net}/mana/gdma.h     | 188 ++++-
 .../net}/mana/hw_channel.h                    |   0
 .../microsoft => include/net}/mana/mana.h     |  29 +-
 .../net}/mana/shm_channel.h                   |   0
 include/uapi/rdma/ib_user_ioctl_verbs.h       |   1 +
 include/uapi/rdma/mana-abi.h                  |  66 ++
 23 files changed, 2180 insertions(+), 45 deletions(-)
 create mode 100644 drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c
 create mode 100644 drivers/infiniband/hw/mana/main.c
 create mode 100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c
 create mode 100644 drivers/infiniband/hw/mana/qp.c
 create mode 100644 drivers/infiniband/hw/mana/wq.c
 rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h (77%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h (93%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/shm_channel.h (100%)
 create mode 100644 include/uapi/rdma/mana-abi.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v4 1/12 net: mana: Add support for auxiliary device

From: Long Li <[email protected]>

In preparation for supporting MANA RDMA driver, add support for auxiliary device in the Ethernet driver. The RDMA device is modeled as an auxiliary device to the Ethernet device.

---
Change log:
v3: define mana_adev_idx_alloc and mana_adev_idx_free as static

 drivers/net/ethernet/microsoft/mana/gdma.h    |  2 +
 drivers/net/ethernet/microsoft/mana/mana.h    |  6 ++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 83 ++++++++++++++++++-
 3 files changed, 90 insertions(+), 1 deletion(-)
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
index 41ecd156e95f..d815d323be87 100644
@@ -204,6 +204,8 @@ struct gdma_dev {
 
 	/* GDMA driver specific pointer */
 	void *driver_data;
+
+	struct auxiliary_device *adev;
 };
 
 #define MINIMUM_SUPPORTED_PAGE_SIZE PAGE_SIZE
drivers/net/ethernet/microsoft/mana/mana.h +6 -0
index d36405af9432..51bff91b63ee 100644
@@ -6,6 +6,7 @@
 
 #include "gdma.h"
 #include "hw_channel.h"
+#include <linux/auxiliary_bus.h>
 
 /* Microsoft Azure Network Adapter (MANA)'s definitions
  *
@@ -561,4 +562,9 @@ struct mana_tx_package {
 	struct gdma_posted_wqe_info wqe_info;
 };
 
+struct mana_adev {
+	struct auxiliary_device adev;
+	struct gdma_dev *mdev;
+};
+
 #endif /* _MANA_H */
drivers/net/ethernet/microsoft/mana/mana_en.c +82 -1

113 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

PATCH v4 2/12 net: mana: Record the physical address for doorbell page region

In reply to [email protected]

From: Long Li <[email protected]>

For supporting RDMA device with multiple user contexts with their individual doorbell pages, record the start address of doorbell page region for use by the RDMA driver to allocate user context doorbell IDs.

---
 drivers/net/ethernet/microsoft/mana/gdma.h      | 2 ++
 drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 ++++
 2 files changed, 6 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
index d815d323be87..c724ca410fcb 100644
@@ -350,9 +350,11 @@ struct gdma_context {
 	struct completion	eq_test_event;
 	u32			test_event_eq_id;
 
+	phys_addr_t		bar0_pa;
 	void __iomem		*bar0_va;
 	void __iomem		*shm_base;
 	void __iomem		*db_page_base;
+	phys_addr_t		phys_db_page_base;
 	u32 db_page_size;
 
 	/* Shared memory chanenl (used to bootstrap HWC) */
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -0
index 49b85ca578b0..9fafaa0c8e76 100644
@@ -27,6 +27,9 @@ static void mana_gd_init_registers(struct pci_dev *pdev)
 	gc->db_page_base = gc->bar0_va +
 				mana_gd_r64(gc, GDMA_REG_DB_PAGE_OFFSET);
 
+	gc->phys_db_page_base = gc->bar0_pa +
+				mana_gd_r64(gc, GDMA_REG_DB_PAGE_OFFSET);
+
 	gc->shm_base = gc->bar0_va + mana_gd_r64(gc, GDMA_REG_SHM_OFFSET);
 }
 
@@ -1335,6 +1338,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	mutex_init(&gc->eq_test_event_mutex);
 	pci_set_drvdata(pdev, gc);
+	gc->bar0_pa = pci_resource_start(pdev, 0);
 
 	bar0_va = pci_iomap(pdev, bar, 0);
 	if (!bar0_va)
Signature · 1 line
2.17.1
#19 lore raw

PATCH v4 4/12 net: mana: Add functions for allocating doorbell page from GDMA

In reply to [email protected]

From: Long Li <[email protected]>

The RDMA device needs to allocate doorbell pages for each user context. Implement those functions and expose them for use by the RDMA driver.

---
 drivers/net/ethernet/microsoft/mana/gdma.h    | 29 ++++++++++
 .../net/ethernet/microsoft/mana/gdma_main.c   | 56 +++++++++++++++++++
 2 files changed, 85 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma.h +29 -0
index c724ca410fcb..f945755760dc 100644
@@ -22,11 +22,15 @@ enum gdma_request_type {
 	GDMA_GENERATE_TEST_EQE		= 10,
 	GDMA_CREATE_QUEUE		= 12,
 	GDMA_DISABLE_QUEUE		= 13,
+	GDMA_ALLOCATE_RESOURCE_RANGE	= 22,
+	GDMA_DESTROY_RESOURCE_RANGE	= 24,
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
 };
 
+#define GDMA_RESOURCE_DOORBELL_PAGE	27
+
 enum gdma_queue_type {
 	GDMA_INVALID_QUEUE,
 	GDMA_SQ,
@@ -568,6 +572,26 @@ struct gdma_register_device_resp {
 	u32 db_id;
 }; /* HW DATA */
 
+struct gdma_allocate_resource_range_req {
+	struct gdma_req_hdr hdr;
+	u32 resource_type;
+	u32 num_resources;
+	u32 alignment;
+	u32 allocated_resources;
+};
+
+struct gdma_allocate_resource_range_resp {
+	struct gdma_resp_hdr hdr;
+	u32 allocated_resources;
+};
+
+struct gdma_destroy_resource_range_req {
+	struct gdma_req_hdr hdr;
+	u32 resource_type;
+	u32 num_resources;
+	u32 allocated_resources;
+};
+
 /* GDMA_CREATE_QUEUE */
 struct gdma_create_queue_req {
 	struct gdma_req_hdr hdr;
@@ -676,4 +700,9 @@ void mana_gd_free_memory(struct gdma_mem_info *gmi);
 
 int mana_gd_send_request(struct gdma_context *gc, u32 req_len, const void *req,
 			 u32 resp_len, void *resp);
+
+int mana_gd_allocate_doorbell_page(struct gdma_context *gc, int *doorbell_page);
+
+int mana_gd_destroy_doorbell_page(struct gdma_context *gc, int doorbell_page);
+
 #endif /* _GDMA_H */
drivers/net/ethernet/microsoft/mana/gdma_main.c +56 -0

62 lines not on this page · raw message

Signature · 1 line
2.17.1
#6 lore raw

PATCH v4 3/12 net: mana: Handle vport sharing between devices

In reply to [email protected]

From: Long Li <[email protected]>

For outgoing packets, the PF requires the VF to configure the vport with corresponding protection domain and doorbell ID for the kernel or user context. The vport can't be shared between different contexts.

Implement the logic to exclusively take over the vport by either the Ethernet device or RDMA device.

---
Change log:
v2: use refcount instead of directly using atomic variables
v4: change to mutex to avoid possible race with refcount

 drivers/net/ethernet/microsoft/mana/mana.h    |  7 ++++
 drivers/net/ethernet/microsoft/mana/mana_en.c | 40 ++++++++++++++++++-
 2 files changed, 45 insertions(+), 2 deletions(-)
drivers/net/ethernet/microsoft/mana/mana.h +7 -0
index 51bff91b63ee..8e58abdce906 100644
@@ -376,6 +376,10 @@ struct mana_port_context {
 
 	mana_handle_t port_handle;
 
+	/* Mutex for sharing access to vport_use_count */
+	struct mutex vport_mutex;
+	int vport_use_count;
+
 	u16 port_idx;
 
 	bool port_is_up;
@@ -567,4 +571,7 @@ struct mana_adev {
 	struct gdma_dev *mdev;
 };
 
+int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
+		   u32 doorbell_pg_id);
+void mana_uncfg_vport(struct mana_port_context *apc);
 #endif /* _MANA_H */
drivers/net/ethernet/microsoft/mana/mana_en.c +38 -2
index 745a9783dd70..23e7e423a544 100644
@@ -530,13 +530,31 @@ static int mana_query_vport_cfg(struct mana_port_context *apc, u32 vport_index,
 	return 0;
 }
 
-static int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
-			  u32 doorbell_pg_id)
+void mana_uncfg_vport(struct mana_port_context *apc)
+{
+	mutex_lock(&apc->vport_mutex);
+	apc->vport_use_count--;
+	WARN_ON(apc->vport_use_count < 0);
+	mutex_unlock(&apc->vport_mutex);
+}
+EXPORT_SYMBOL_GPL(mana_uncfg_vport);
+
+int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
+		   u32 doorbell_pg_id)
 {
 	struct mana_config_vport_resp resp = {};
 	struct mana_config_vport_req req = {};
 	int err;
 
+	/* Ethernet driver and IB driver can't take the port at the same time */
+	mutex_lock(&apc->vport_mutex);
+	if (apc->vport_use_count > 0) {
+		mutex_unlock(&apc->vport_mutex);
+		return -ENODEV;
+	}
+	apc->vport_use_count++;
+	mutex_unlock(&apc->vport_mutex);
+
 	mana_gd_init_req_hdr(&req.hdr, MANA_CONFIG_VPORT_TX,
 			     sizeof(req), sizeof(resp));
 	req.vport = apc->port_handle;
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);
 out:
+	if (err) {
+		mutex_lock(&apc->vport_mutex);
+		apc->vport_use_count--;
+		mutex_unlock(&apc->vport_mutex);
+	}
+
 	return err;
 }
+EXPORT_SYMBOL_GPL(mana_cfg_vport);
 
 static int mana_cfg_vport_steering(struct mana_port_context *apc,
 				   enum TRI_STATE rx,
@@ -626,6 +654,9 @@ static int mana_cfg_vport_steering(struct mana_port_context *apc,
 			   resp.hdr.status);
 		err = -EPROTO;
 	}
+
+	netdev_info(ndev, "Configured steering vPort %llu entries %u\n",
+		    apc->port_handle, num_entries);
 out:
 	kfree(req);
 	return err;
@@ -1678,6 +1709,8 @@ static void mana_destroy_vport(struct mana_port_context *apc)
 	}
 
 	mana_destroy_txq(apc);
+
+	mana_uncfg_vport(apc);
 }
 
 static int mana_create_vport(struct mana_port_context *apc,
@@ -1929,6 +1962,9 @@ static int mana_probe_port(struct mana_context *ac, int port_idx,
 	apc->port_handle = INVALID_MANA_HANDLE;
 	apc->port_idx = port_idx;
 
+	mutex_init(&apc->vport_mutex);
+	apc->vport_use_count = 0;
+
 	ndev->netdev_ops = &mana_devops;
 	ndev->ethtool_ops = &mana_ethtool_ops;
 	ndev->mtu = ETH_DATA_LEN;
Signature · 1 line
2.17.1
#34 lore raw

PATCH v4 8/12 net: mana: Record port number in netdev

In reply to [email protected]

From: Long Li <[email protected]>

The port number is useful for user-mode application to identify this net device based on port index. Set to the correct value in ndev.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -0
index b769fccc076d..963a4dbb46ea 100644
@@ -1975,6 +1975,7 @@ static int mana_probe_port(struct mana_context *ac, int port_idx,
 	ndev->max_mtu = ndev->mtu;
 	ndev->min_mtu = ndev->mtu;
 	ndev->needed_headroom = MANA_HEADROOM;
+	ndev->dev_port = port_idx;
 	SET_NETDEV_DEV(ndev, gc->dev);
 
 	netif_carrier_off(ndev);
Signature · 1 line
2.17.1
#23 lore raw

PATCH v4 5/12 net: mana: Set the DMA device max segment size

In reply to [email protected]

From: Ajay Sharma <[email protected]>

MANA hardware doesn't have any restrictions on the DMA segment size, set it to the max allowed value.

---
Change log:
v2: Use the max allowed value as the hardware doesn't have any limit

 drivers/net/ethernet/microsoft/mana/gdma_main.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ethernet/microsoft/mana/gdma_main.c +6 -0
index 7b42b78b7ddf..0c38c9a539f9 100644
@@ -1387,6 +1387,12 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto release_region;
 
+	err = dma_set_max_seg_size(&pdev->dev, UINT_MAX);
+	if (err) {
+		dev_err(&pdev->dev, "Failed to set dma device segment size\n");
+		goto release_region;
+	}
+
 	err = -ENOMEM;
 	gc = vzalloc(sizeof(*gc));
 	if (!gc)
Signature · 1 line
2.17.1
#31 lore raw

PATCH v4 7/12 net: mana: Export Work Queue functions for use by RDMA driver

In reply to [email protected]

From: Long Li <[email protected]>

RDMA device may need to create Ethernet device queues for use by Queue Pair type RAW. This allows a user-mode context accesses Ethernet hardware queues. Export the supporting functions for use by the RDMA driver.

---
 drivers/net/ethernet/microsoft/mana/gdma_main.c |  1 +
 drivers/net/ethernet/microsoft/mana/mana.h      |  9 +++++++++
 drivers/net/ethernet/microsoft/mana/mana_en.c   | 16 +++++++++-------
 3 files changed, 19 insertions(+), 7 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -0
index 60cc1270b7d5..272facd272a4 100644
@@ -125,6 +125,7 @@ int mana_gd_send_request(struct gdma_context *gc, u32 req_len, const void *req,
 
 	return mana_hwc_send_request(hwc, req_len, req, resp_len, resp);
 }
+EXPORT_SYMBOL(mana_gd_send_request);
 
 int mana_gd_alloc_memory(struct gdma_context *gc, unsigned int length,
 			 struct gdma_mem_info *gmi)
drivers/net/ethernet/microsoft/mana/mana.h +9 -0
index 8e58abdce906..aca95c6ba8b3 100644
@@ -571,6 +571,15 @@ struct mana_adev {
 	struct gdma_dev *mdev;
 };
 
+int mana_create_wq_obj(struct mana_port_context *apc,
+		       mana_handle_t vport,
+		       u32 wq_type, struct mana_obj_spec *wq_spec,
+		       struct mana_obj_spec *cq_spec,
+		       mana_handle_t *wq_obj);
+
+void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
+			 mana_handle_t wq_obj);
+
 int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
 		   u32 doorbell_pg_id);
 void mana_uncfg_vport(struct mana_port_context *apc);
drivers/net/ethernet/microsoft/mana/mana_en.c +9 -7
index c18c358607a7..b769fccc076d 100644
@@ -662,11 +662,11 @@ static int mana_cfg_vport_steering(struct mana_port_context *apc,
 	return err;
 }
 
-static int mana_create_wq_obj(struct mana_port_context *apc,
-			      mana_handle_t vport,
-			      u32 wq_type, struct mana_obj_spec *wq_spec,
-			      struct mana_obj_spec *cq_spec,
-			      mana_handle_t *wq_obj)
+int mana_create_wq_obj(struct mana_port_context *apc,
+		       mana_handle_t vport,
+		       u32 wq_type, struct mana_obj_spec *wq_spec,
+		       struct mana_obj_spec *cq_spec,
+		       mana_handle_t *wq_obj)
 {
 	struct mana_create_wqobj_resp resp = {};
 	struct mana_create_wqobj_req req = {};
@@ -715,9 +715,10 @@ static int mana_create_wq_obj(struct mana_port_context *apc,
 out:
 	return err;
 }
+EXPORT_SYMBOL_GPL(mana_create_wq_obj);
 
-static void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
-				mana_handle_t wq_obj)
+void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
+			 mana_handle_t wq_obj)
 {
 	struct mana_destroy_wqobj_resp resp = {};
 	struct mana_destroy_wqobj_req req = {};
@@ -742,6 +743,7 @@ static void mana_destroy_wq_obj(struct mana_port_context *apc, u32 wq_type,
 		netdev_err(ndev, "Failed to destroy WQ object: %d, 0x%x\n", err,
 			   resp.hdr.status);
 }
+EXPORT_SYMBOL_GPL(mana_destroy_wq_obj);
 
 static void mana_destroy_eq(struct mana_context *ac)
 {
Signature · 1 line
2.17.1
#41 lore raw

PATCH v4 11/12 net: mana: Define and process GDMA response code GDMA_STATUS_MORE_ENTRIES

In reply to [email protected]

From: Ajay Sharma <[email protected]>

When doing memory registration, the PF may respond with GDMA_STATUS_MORE_ENTRIES to indicate a follow request is needed. This is not an error and should be processed as expected.

---
 drivers/net/ethernet/microsoft/mana/hw_channel.c | 2 +-
 include/net/mana/gdma.h                          | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/microsoft/mana/hw_channel.c +1 -1
index e61cb3f6fbe1..24ecd193a185 100644
@@ -820,7 +820,7 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 		goto out;
 	}
 
-	if (ctx->status_code) {
+	if (ctx->status_code && ctx->status_code != GDMA_STATUS_MORE_ENTRIES) {
 		dev_err(hwc->dev, "HWC: Failed hw_channel req: 0x%x\n",
 			ctx->status_code);
 		err = -EPROTO;
include/net/mana/gdma.h +2 -0
index ef5de92dd98d..0485902d96c9 100644
@@ -9,6 +9,8 @@
 
 #include "shm_channel.h"
 
+#define GDMA_STATUS_MORE_ENTRIES	0x00000105
+
 /* Structures labeled with "HW DATA" are exchanged with the hardware. All of
  * them are naturally aligned and hence don't need __packed.
  */
Signature · 1 line
2.17.1
#36 lore raw

PATCH v4 9/12 net: mana: Move header files to a common location

In reply to [email protected]

From: Long Li <[email protected]>

In preparation to add MANA RDMA driver, move all the required header files to a common location for use by both Ethernet and RDMA drivers.

---
Change log:
v2: Move headers to include/net/mana, instead of include/linux/mana

 MAINTAINERS                                                   | 1 +
 drivers/net/ethernet/microsoft/mana/gdma_main.c               | 2 +-
 drivers/net/ethernet/microsoft/mana/hw_channel.c              | 4 ++--
 drivers/net/ethernet/microsoft/mana/mana_bpf.c                | 2 +-
 drivers/net/ethernet/microsoft/mana/mana_en.c                 | 2 +-
 drivers/net/ethernet/microsoft/mana/mana_ethtool.c            | 2 +-
 drivers/net/ethernet/microsoft/mana/shm_channel.c             | 2 +-
 {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h   | 0
 .../net/ethernet/microsoft => include/net}/mana/hw_channel.h  | 0
 {drivers/net/ethernet/microsoft => include/net}/mana/mana.h   | 0
 .../net/ethernet/microsoft => include/net}/mana/shm_channel.h | 0
 11 files changed, 8 insertions(+), 7 deletions(-)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h (100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/shm_channel.h (100%)
MAINTAINERS +1 -0
index 40fa1955ca3f..51bec6d5076d 100644
@@ -9108,6 +9108,7 @@ F:	include/asm-generic/hyperv-tlfs.h
 F:	include/asm-generic/mshyperv.h
 F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
+F:	include/net/mana
 F:	include/uapi/linux/hyperv.h
 F:	net/vmw_vsock/hyperv_transport.c
 F:	tools/hv/
drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -1
index 272facd272a4..ab3d8e75fb69 100644
@@ -6,7 +6,7 @@
 #include <linux/utsname.h>
 #include <linux/version.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 static u32 mana_gd_r32(struct gdma_context *g, u64 offset)
 {
drivers/net/ethernet/microsoft/mana/hw_channel.c +2 -2
index 078d6a5a0768..e61cb3f6fbe1 100644
@@ -1,8 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
 /* Copyright (c) 2021, Microsoft Corporation. */
 
-#include "gdma.h"
-#include "hw_channel.h"
+#include <net/mana/gdma.h>
+#include <net/mana/hw_channel.h>
 
 static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id)
 {
drivers/net/ethernet/microsoft/mana/mana_bpf.c +1 -1
index 1d2f948b5c00..97a79659af45 100644
@@ -8,7 +8,7 @@
 #include <linux/bpf_trace.h>
 #include <net/xdp.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 void mana_xdp_tx(struct sk_buff *skb, struct net_device *ndev)
 {
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -1
index 963a4dbb46ea..5aab7afc9143 100644
@@ -11,7 +11,7 @@
 #include <net/checksum.h>
 #include <net/ip6_checksum.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 static DEFINE_IDA(mana_adev_ida);
 
drivers/net/ethernet/microsoft/mana/mana_ethtool.c +1 -1
index e13f2453eabb..ebc6595d02fe 100644
@@ -5,7 +5,7 @@
 #include <linux/etherdevice.h>
 #include <linux/ethtool.h>
 
-#include "mana.h"
+#include <net/mana/mana.h>
 
 static const struct {
 	char name[ETH_GSTRING_LEN];
drivers/net/ethernet/microsoft/mana/shm_channel.c +1 -1
index da255da62176..5553af9c8085 100644
@@ -6,7 +6,7 @@
 #include <linux/io.h>
 #include <linux/mm.h>
 
-#include "shm_channel.h"
+#include <net/mana/shm_channel.h>
 
 #define PAGE_FRAME_L48_WIDTH_BYTES 6
 #define PAGE_FRAME_L48_WIDTH_BITS (PAGE_FRAME_L48_WIDTH_BYTES * 8)
include/net/mana/gdma.h renamed from drivers/net/ethernet/microsoft/mana/gdma.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/gdma.h
rename to include/net/mana/gdma.h
include/net/mana/hw_channel.h renamed from drivers/net/ethernet/microsoft/mana/hw_channel.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/hw_channel.h
rename to include/net/mana/hw_channel.h
include/net/mana/mana.h renamed from drivers/net/ethernet/microsoft/mana/mana.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/mana.h
rename to include/net/mana/mana.h
include/net/mana/shm_channel.h renamed from drivers/net/ethernet/microsoft/mana/shm_channel.h +0 -0
similarity index 100%
rename from drivers/net/ethernet/microsoft/mana/shm_channel.h
rename to include/net/mana/shm_channel.h
Signature · 1 line
2.17.1
#43 lore raw

PATCH v4 12/12 RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]

From: Long Li <[email protected]>

Add a RDMA VF driver for Microsoft Azure Network Adapter (MANA).

---
Change log:
v2:
Changed coding sytles/formats
Checked undersize for udata length
Changed all logging to use ibdev_xxx()
Avoided page array copy when doing MR
Sorted driver ops
Fixed warnings reported by kernel test robot <[email protected]>

v3:
More coding sytle/format changes

v4:
Process error on hardware vport configuration

 MAINTAINERS                             |   3 +
 drivers/infiniband/Kconfig              |   1 +
 drivers/infiniband/hw/Makefile          |   1 +
 drivers/infiniband/hw/mana/Kconfig      |   7 +
 drivers/infiniband/hw/mana/Makefile     |   4 +
 drivers/infiniband/hw/mana/cq.c         |  80 +++
 drivers/infiniband/hw/mana/main.c       | 681 ++++++++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h    | 145 +++++
 drivers/infiniband/hw/mana/mr.c         | 133 +++++
 drivers/infiniband/hw/mana/qp.c         | 501 +++++++++++++++++
 drivers/infiniband/hw/mana/wq.c         | 114 ++++
 include/net/mana/mana.h                 |   3 +
 include/uapi/rdma/ib_user_ioctl_verbs.h |   1 +
 include/uapi/rdma/mana-abi.h            |  66 +++
 14 files changed, 1740 insertions(+)
 create mode 100644 drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c
 create mode 100644 drivers/infiniband/hw/mana/main.c
 create mode 100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c
 create mode 100644 drivers/infiniband/hw/mana/qp.c
 create mode 100644 drivers/infiniband/hw/mana/wq.c
 create mode 100644 include/uapi/rdma/mana-abi.h
MAINTAINERS +3 -0
index 51bec6d5076d..1bed8444786d 100644
@@ -9078,6 +9078,7 @@ M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
 M:	Dexuan Cui <[email protected]>
+M:	Long Li <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -9095,6 +9096,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
 F:	drivers/clocksource/hyperv_timer.c
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
+F:	drivers/infiniband/hw/mana/
 F:	drivers/input/serio/hyperv-keyboard.c
 F:	drivers/iommu/hyperv-iommu.c
 F:	drivers/net/ethernet/microsoft/
@@ -9110,6 +9112,7 @@ F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
 F:	include/net/mana
 F:	include/uapi/linux/hyperv.h
+F:	include/uapi/rdma/mana-abi.h
 F:	net/vmw_vsock/hyperv_transport.c
 F:	tools/hv/
 
drivers/infiniband/Kconfig +1 -0
index 33d3ce9c888e..a062c662ecff 100644
@@ -83,6 +83,7 @@ source "drivers/infiniband/hw/qib/Kconfig"
 source "drivers/infiniband/hw/cxgb4/Kconfig"
 source "drivers/infiniband/hw/efa/Kconfig"
 source "drivers/infiniband/hw/irdma/Kconfig"
+source "drivers/infiniband/hw/mana/Kconfig"
 source "drivers/infiniband/hw/mlx4/Kconfig"
 source "drivers/infiniband/hw/mlx5/Kconfig"
 source "drivers/infiniband/hw/ocrdma/Kconfig"
drivers/infiniband/hw/Makefile +1 -0
index fba0b3be903e..f62e9e00c780 100644
@@ -4,6 +4,7 @@ obj-$(CONFIG_INFINIBAND_QIB)		+= qib/
 obj-$(CONFIG_INFINIBAND_CXGB4)		+= cxgb4/
 obj-$(CONFIG_INFINIBAND_EFA)		+= efa/
 obj-$(CONFIG_INFINIBAND_IRDMA)		+= irdma/
+obj-$(CONFIG_MANA_INFINIBAND)		+= mana/
 obj-$(CONFIG_MLX4_INFINIBAND)		+= mlx4/
 obj-$(CONFIG_MLX5_INFINIBAND)		+= mlx5/
 obj-$(CONFIG_INFINIBAND_OCRDMA)		+= ocrdma/
drivers/infiniband/hw/mana/Kconfig new file +7 -0
new file mode 100644
index 000000000000..b3ff03a23257
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config MANA_INFINIBAND
+	tristate "Microsoft Azure Network Adapter support"
+	depends on NETDEVICES && ETHERNET && PCI && MICROSOFT_MANA
+	help
+	  This driver provides low-level RDMA support for
+	  Microsoft Azure Network Adapter (MANA).
drivers/infiniband/hw/mana/Makefile new file +4 -0
new file mode 100644
index 000000000000..a799fe264c5a
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_MANA_INFINIBAND) += mana_ib.o
+
+mana_ib-y := main.o wq.o qp.o cq.o mr.o
drivers/infiniband/hw/mana/cq.c new file +80 -0

80 lines not on this page · raw message

drivers/infiniband/hw/mana/main.c new file +681 -0

681 lines not on this page · raw message

drivers/infiniband/hw/mana/mana_ib.h new file +145 -0

145 lines not on this page · raw message

drivers/infiniband/hw/mana/mr.c new file +133 -0

133 lines not on this page · raw message

drivers/infiniband/hw/mana/qp.c new file +501 -0

501 lines not on this page · raw message

drivers/infiniband/hw/mana/wq.c new file +114 -0

114 lines not on this page · raw message

include/net/mana/mana.h +3 -0
index 3a0bc6e0b730..1ff6e0d07cfd 100644
@@ -406,6 +406,9 @@ int mana_bpf(struct net_device *ndev, struct netdev_bpf *bpf);
 
 extern const struct ethtool_ops mana_ethtool_ops;
 
+/* A CQ can be created not associated with any EQ */
+#define GDMA_CQ_NO_EQ  0xffff
+
 struct mana_obj_spec {
 	u32 queue_index;
 	u64 gdma_region;
include/uapi/rdma/ib_user_ioctl_verbs.h +1 -0
index 3072e5d6b692..081aabf536dc 100644
@@ -250,6 +250,7 @@ enum rdma_driver_id {
 	RDMA_DRIVER_QIB,
 	RDMA_DRIVER_EFA,
 	RDMA_DRIVER_SIW,
+	RDMA_DRIVER_MANA,
 };
 
 enum ib_uverbs_gid_type {
include/uapi/rdma/mana-abi.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.17.1
#25 lore raw

PATCH v4 6/12 net: mana: Define data structures for protection domain and memory registration

In reply to [email protected]

From: Ajay Sharma <[email protected]>

The MANA hardware support protection domain and memory registration for use in RDMA environment. Add those definitions and expose them for use by the RDMA driver.

---
Change log:
v3: format/coding style changes

 drivers/net/ethernet/microsoft/mana/gdma.h    | 146 +++++++++++++++++-
 .../net/ethernet/microsoft/mana/gdma_main.c   |  27 ++--
 drivers/net/ethernet/microsoft/mana/mana_en.c |  18 ++-
 3 files changed, 168 insertions(+), 23 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma.h +141 -5
index f945755760dc..b1bec8ab5695 100644
@@ -27,6 +27,10 @@ enum gdma_request_type {
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
+	GDMA_CREATE_PD			= 29,
+	GDMA_DESTROY_PD			= 30,
+	GDMA_CREATE_MR			= 31,
+	GDMA_DESTROY_MR			= 32,
 };
 
 #define GDMA_RESOURCE_DOORBELL_PAGE	27
@@ -59,6 +63,8 @@ enum {
 	GDMA_DEVICE_MANA	= 2,
 };
 
+typedef u64 gdma_obj_handle_t;
+
 struct gdma_resource {
 	/* Protect the bitmap */
 	spinlock_t lock;
@@ -192,7 +198,7 @@ struct gdma_mem_info {
 	u64 length;
 
 	/* Allocated by the PF driver */
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 };
 
 #define REGISTER_ATB_MST_MKEY_LOWER_SIZE 8
@@ -599,7 +605,7 @@ struct gdma_create_queue_req {
 	u32 reserved1;
 	u32 pdid;
 	u32 doolbell_id;
-	u64 gdma_region;
+	gdma_obj_handle_t gdma_region;
 	u32 reserved2;
 	u32 queue_size;
 	u32 log2_throttle_limit;
@@ -626,6 +632,28 @@ struct gdma_disable_queue_req {
 	u32 alloc_res_id_on_creation;
 }; /* HW DATA */
 
+enum atb_page_size {
+	ATB_PAGE_SIZE_4K,
+	ATB_PAGE_SIZE_8K,
+	ATB_PAGE_SIZE_16K,
+	ATB_PAGE_SIZE_32K,
+	ATB_PAGE_SIZE_64K,
+	ATB_PAGE_SIZE_128K,
+	ATB_PAGE_SIZE_256K,
+	ATB_PAGE_SIZE_512K,
+	ATB_PAGE_SIZE_1M,
+	ATB_PAGE_SIZE_2M,
+	ATB_PAGE_SIZE_MAX,
+};
+
+enum gdma_mr_access_flags {
+	GDMA_ACCESS_FLAG_LOCAL_READ = (1 << 0),
+	GDMA_ACCESS_FLAG_LOCAL_WRITE = (1 << 1),
+	GDMA_ACCESS_FLAG_REMOTE_READ = (1 << 2),
+	GDMA_ACCESS_FLAG_REMOTE_WRITE = (1 << 3),
+	GDMA_ACCESS_FLAG_REMOTE_ATOMIC = (1 << 4),
+};
+
 /* GDMA_CREATE_DMA_REGION */
 struct gdma_create_dma_region_req {
 	struct gdma_req_hdr hdr;
@@ -652,14 +680,14 @@ struct gdma_create_dma_region_req {
 
 struct gdma_create_dma_region_resp {
 	struct gdma_resp_hdr hdr;
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
 /* GDMA_DMA_REGION_ADD_PAGES */
 struct gdma_dma_region_add_pages_req {
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 
 	u32 page_addr_list_len;
 	u32 reserved3;
@@ -671,9 +699,114 @@ struct gdma_dma_region_add_pages_req {
 struct gdma_destroy_dma_region_req {
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
+enum gdma_pd_flags {
+	GDMA_PD_FLAG_ALLOW_GPA_MR = (1 << 0),
+	GDMA_PD_FLAG_ALLOW_FMR_MR = (1 << 1),
+};
+
+struct gdma_create_pd_req {
+	struct gdma_req_hdr hdr;
+	enum gdma_pd_flags flags;
+	u32 reserved;
+};
+
+struct gdma_create_pd_resp {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	u32 pd_id;
+	u32 reserved;
+};
+
+struct gdma_destroy_pd_req {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+};
+
+struct gdma_destory_pd_resp {
+	struct gdma_resp_hdr hdr;
+};
+
+enum gdma_mr_type {
+	/* Guest Physical Address - MRs of this type allow access
+	 * to any DMA-mapped memory using bus-logical address
+	 */
+	GDMA_MR_TYPE_GPA = 1,
+
+	/* Guest Virtual Address - MRs of this type allow access
+	 * to memory mapped by PTEs associated with this MR using a virtual
+	 * address that is set up in the MST
+	 */
+	GDMA_MR_TYPE_GVA,
+
+	/* Fast Memory Register - Like GVA but the MR is initially put in the
+	 * FREE state (as opposed to Valid), and the specified number of
+	 * PTEs are reserved for future fast memory reservations.
+	 */
+	GDMA_MR_TYPE_FMR,
+};
+
+struct gdma_create_mr_params {
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	union {
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;
+		} gva;
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;
+		struct {
+			enum atb_page_size page_size;
+			u32  reserved_pte_count;
+		} fmr;
+	};
+};
+
+struct gdma_create_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	u32 reserved;
+
+	union {
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;
+
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;
+		} gva;
+
+		struct {
+			enum atb_page_size page_size;
+			u32 reserved_pte_count;
+		} fmr;
+	};
+};
+
+struct gdma_create_mr_response {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+	u32 lkey;
+	u32 rkey;
+};
+
+struct gdma_destroy_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+};
+
+struct gdma_destroy_mr_response {
+	struct gdma_resp_hdr hdr;
+};
+
 int mana_gd_verify_vf_version(struct pci_dev *pdev);
 
 int mana_gd_register_device(struct gdma_dev *gd);
@@ -705,4 +838,7 @@ int mana_gd_allocate_doorbell_page(struct gdma_context *gc, int *doorbell_page);
 
 int mana_gd_destroy_doorbell_page(struct gdma_context *gc, int doorbell_page);
 
+int mana_gd_destroy_dma_region(struct gdma_context *gc,
+			       gdma_obj_handle_t dma_region_handle);
+
 #endif /* _GDMA_H */
drivers/net/ethernet/microsoft/mana/gdma_main.c +17 -10
index 0c38c9a539f9..60cc1270b7d5 100644
@@ -226,7 +226,7 @@ static int mana_gd_create_hw_eq(struct gdma_context *gc,
 	req.type = queue->type;
 	req.pdid = queue->gdma_dev->pdid;
 	req.doolbell_id = queue->gdma_dev->doorbell;
-	req.gdma_region = queue->mem_info.gdma_region;
+	req.gdma_region = queue->mem_info.dma_region_handle;
 	req.queue_size = queue->queue_size;
 	req.log2_throttle_limit = queue->eq.log2_throttle_limit;
 	req.eq_pci_msix_index = queue->eq.msix_index;
@@ -240,7 +240,7 @@ static int mana_gd_create_hw_eq(struct gdma_context *gc,
 
 	queue->id = resp.queue_index;
 	queue->eq.disable_needed = true;
-	queue->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+	queue->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
 	return 0;
 }
 
@@ -694,24 +694,30 @@ int mana_gd_create_hwc_queue(struct gdma_dev *gd,
 	return err;
 }
 
-static void mana_gd_destroy_dma_region(struct gdma_context *gc, u64 gdma_region)
+int mana_gd_destroy_dma_region(struct gdma_context *gc,
+			       gdma_obj_handle_t dma_region_handle)
 {
 	struct gdma_destroy_dma_region_req req = {};
 	struct gdma_general_resp resp = {};
 	int err;
 
-	if (gdma_region == GDMA_INVALID_DMA_REGION)
-		return;
+	if (dma_region_handle == GDMA_INVALID_DMA_REGION)
+		return 0;
 
 	mana_gd_init_req_hdr(&req.hdr, GDMA_DESTROY_DMA_REGION, sizeof(req),
 			     sizeof(resp));
-	req.gdma_region = gdma_region;
+	req.dma_region_handle = dma_region_handle;
 
 	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
-	if (err || resp.hdr.status)
+	if (err || resp.hdr.status) {
 		dev_err(gc->dev, "Failed to destroy DMA region: %d, 0x%x\n",
 			err, resp.hdr.status);
+		return -EPROTO;
+	}
+
+	return 0;
 }
+EXPORT_SYMBOL(mana_gd_destroy_dma_region);
 
 static int mana_gd_create_dma_region(struct gdma_dev *gd,
 				     struct gdma_mem_info *gmi)
@@ -756,14 +762,15 @@ static int mana_gd_create_dma_region(struct gdma_dev *gd,
 	if (err)
 		goto out;
 
-	if (resp.hdr.status || resp.gdma_region == GDMA_INVALID_DMA_REGION) {
+	if (resp.hdr.status ||
+	    resp.dma_region_handle == GDMA_INVALID_DMA_REGION) {
 		dev_err(gc->dev, "Failed to create DMA region: 0x%x\n",
 			resp.hdr.status);
 		err = -EPROTO;
 		goto out;
 	}
 
-	gmi->gdma_region = resp.gdma_region;
+	gmi->dma_region_handle = resp.dma_region_handle;
 out:
 	kfree(req);
 	return err;
@@ -886,7 +893,7 @@ void mana_gd_destroy_queue(struct gdma_context *gc, struct gdma_queue *queue)
 		return;
 	}
 
-	mana_gd_destroy_dma_region(gc, gmi->gdma_region);
+	mana_gd_destroy_dma_region(gc, gmi->dma_region_handle);
 	mana_gd_free_memory(gmi);
 	kfree(queue);
 }
drivers/net/ethernet/microsoft/mana/mana_en.c +10 -8
index 23e7e423a544..c18c358607a7 100644
@@ -1382,10 +1382,10 @@ static int mana_create_txq(struct mana_port_context *apc,
 		memset(&wq_spec, 0, sizeof(wq_spec));
 		memset(&cq_spec, 0, sizeof(cq_spec));
 
-		wq_spec.gdma_region = txq->gdma_sq->mem_info.gdma_region;
+		wq_spec.gdma_region = txq->gdma_sq->mem_info.dma_region_handle;
 		wq_spec.queue_size = txq->gdma_sq->queue_size;
 
-		cq_spec.gdma_region = cq->gdma_cq->mem_info.gdma_region;
+		cq_spec.gdma_region = cq->gdma_cq->mem_info.dma_region_handle;
 		cq_spec.queue_size = cq->gdma_cq->queue_size;
 		cq_spec.modr_ctx_id = 0;
 		cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
@@ -1400,8 +1400,10 @@ static int mana_create_txq(struct mana_port_context *apc,
 		txq->gdma_sq->id = wq_spec.queue_index;
 		cq->gdma_cq->id = cq_spec.queue_index;
 
-		txq->gdma_sq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
-		cq->gdma_cq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+		txq->gdma_sq->mem_info.dma_region_handle =
+			GDMA_INVALID_DMA_REGION;
+		cq->gdma_cq->mem_info.dma_region_handle =
+			GDMA_INVALID_DMA_REGION;
 
 		txq->gdma_txq_id = txq->gdma_sq->id;
 
@@ -1612,10 +1614,10 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
 
 	memset(&wq_spec, 0, sizeof(wq_spec));
 	memset(&cq_spec, 0, sizeof(cq_spec));
-	wq_spec.gdma_region = rxq->gdma_rq->mem_info.gdma_region;
+	wq_spec.gdma_region = rxq->gdma_rq->mem_info.dma_region_handle;
 	wq_spec.queue_size = rxq->gdma_rq->queue_size;
 
-	cq_spec.gdma_region = cq->gdma_cq->mem_info.gdma_region;
+	cq_spec.gdma_region = cq->gdma_cq->mem_info.dma_region_handle;
 	cq_spec.queue_size = cq->gdma_cq->queue_size;
 	cq_spec.modr_ctx_id = 0;
 	cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
@@ -1628,8 +1630,8 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
 	rxq->gdma_rq->id = wq_spec.queue_index;
 	cq->gdma_cq->id = cq_spec.queue_index;
 
-	rxq->gdma_rq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
-	cq->gdma_cq->mem_info.gdma_region = GDMA_INVALID_DMA_REGION;
+	rxq->gdma_rq->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
+	cq->gdma_cq->mem_info.dma_region_handle = GDMA_INVALID_DMA_REGION;
 
 	rxq->gdma_id = rxq->gdma_rq->id;
 	cq->gdma_id = cq->gdma_cq->id;
Signature · 1 line
2.17.1
#38 lore raw

PATCH v4 10/12 net: mana: Define max values for SGL entries

In reply to [email protected]

From: Long Li <[email protected]>

The number of maximum SGl entries should be computed from the maximum WQE size for the intended queue type and the corresponding OOB data size. This guarantees the hardware queue can successfully queue requests up to the queue depth exposed to the upper layer.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 2 +-
 include/net/mana/gdma.h                       | 7 +++++++
 include/net/mana/mana.h                       | 4 +---
 3 files changed, 9 insertions(+), 4 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -1
index 5aab7afc9143..9a976d9b6b6f 100644
@@ -187,7 +187,7 @@ int mana_start_xmit(struct sk_buff *skb, struct net_device *ndev)
 	pkg.wqe_req.client_data_unit = 0;
 
 	pkg.wqe_req.num_sge = 1 + skb_shinfo(skb)->nr_frags;
-	WARN_ON_ONCE(pkg.wqe_req.num_sge > 30);
+	WARN_ON_ONCE(pkg.wqe_req.num_sge > MAX_TX_WQE_SGL_ENTRIES);
 
 	if (pkg.wqe_req.num_sge <= ARRAY_SIZE(pkg.sgl_array)) {
 		pkg.wqe_req.sgl = pkg.sgl_array;
include/net/mana/gdma.h +7 -0
index b1bec8ab5695..ef5de92dd98d 100644
@@ -436,6 +436,13 @@ struct gdma_wqe {
 #define MAX_TX_WQE_SIZE 512
 #define MAX_RX_WQE_SIZE 256
 
+#define MAX_TX_WQE_SGL_ENTRIES	((GDMA_MAX_SQE_SIZE - \
+			sizeof(struct gdma_sge) - INLINE_OOB_SMALL_SIZE) / \
+			sizeof(struct gdma_sge))
+
+#define MAX_RX_WQE_SGL_ENTRIES	((GDMA_MAX_RQE_SIZE - \
+			sizeof(struct gdma_sge)) / sizeof(struct gdma_sge))
+
 struct gdma_cqe {
 	u32 cqe_data[GDMA_COMP_DATA_SIZE / 4];
 
include/net/mana/mana.h +1 -3
index aca95c6ba8b3..3a0bc6e0b730 100644
@@ -264,8 +264,6 @@ struct mana_cq {
 	int budget;
 };
 
-#define GDMA_MAX_RQE_SGES 15
-
 struct mana_recv_buf_oob {
 	/* A valid GDMA work request representing the data buffer. */
 	struct gdma_wqe_request wqe_req;
@@ -275,7 +273,7 @@ struct mana_recv_buf_oob {
 
 	/* SGL of the buffer going to be sent has part of the work request. */
 	u32 num_sge;
-	struct gdma_sge sgl[GDMA_MAX_RQE_SGES];
+	struct gdma_sge sgl[MAX_RX_WQE_SGL_ENTRIES];
 
 	/* Required to store the result of mana_gd_post_work_request.
 	 * gdma_posted_wqe_info.wqe_size_in_bu is required for progressing the
Signature · 1 line
2.17.1
#44 lore raw

RE: [EXTERNAL] [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]

Hello Maintainers,
Any idea when these patches would make it into the next kernel release ?

Ajay

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 9:07 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>
Subject: [EXTERNAL] [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

From: Long Li <[email protected]>

Add a RDMA VF driver for Microsoft Azure Network Adapter (MANA).

---
Change log:
v2:
Changed coding sytles/formats
Checked undersize for udata length
Changed all logging to use ibdev_xxx()
Avoided page array copy when doing MR
Sorted driver ops
Fixed warnings reported by kernel test robot <[email protected]>

v3:
More coding sytle/format changes

v4:
Process error on hardware vport configuration

 MAINTAINERS                             |   3 +
 drivers/infiniband/Kconfig              |   1 +
 drivers/infiniband/hw/Makefile          |   1 +
 drivers/infiniband/hw/mana/Kconfig      |   7 +
 drivers/infiniband/hw/mana/Makefile     |   4 +
 drivers/infiniband/hw/mana/cq.c         |  80 +++
 drivers/infiniband/hw/mana/main.c       | 681 ++++++++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h    | 145 +++++
 drivers/infiniband/hw/mana/mr.c         | 133 +++++
 drivers/infiniband/hw/mana/qp.c         | 501 +++++++++++++++++
 drivers/infiniband/hw/mana/wq.c         | 114 ++++
 include/net/mana/mana.h                 |   3 +
 include/uapi/rdma/ib_user_ioctl_verbs.h |   1 +
 include/uapi/rdma/mana-abi.h            |  66 +++
 14 files changed, 1740 insertions(+)
 create mode 100644 drivers/infiniband/hw/mana/Kconfig
 create mode 100644 drivers/infiniband/hw/mana/Makefile
 create mode 100644 drivers/infiniband/hw/mana/cq.c
 create mode 100644 drivers/infiniband/hw/mana/main.c
 create mode 100644 drivers/infiniband/hw/mana/mana_ib.h
 create mode 100644 drivers/infiniband/hw/mana/mr.c
 create mode 100644 drivers/infiniband/hw/mana/qp.c
 create mode 100644 drivers/infiniband/hw/mana/wq.c
 create mode 100644 include/uapi/rdma/mana-abi.h
MAINTAINERS +3 -0
index 51bec6d5076d..1bed8444786d 100644
@@ -9078,6 +9078,7 @@ M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
 M:	Dexuan Cui <[email protected]>
+M:	Long Li <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -9095,6 +9096,7 @@ F:	arch/x86/kernel/cpu/mshyperv.c
 F:	drivers/clocksource/hyperv_timer.c
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
+F:	drivers/infiniband/hw/mana/
 F:	drivers/input/serio/hyperv-keyboard.c
 F:	drivers/iommu/hyperv-iommu.c
 F:	drivers/net/ethernet/microsoft/
@@ -9110,6 +9112,7 @@ F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
 F:	include/net/mana
 F:	include/uapi/linux/hyperv.h
+F:	include/uapi/rdma/mana-abi.h
 F:	net/vmw_vsock/hyperv_transport.c
 F:	tools/hv/
 
drivers/infiniband/Kconfig +1 -0
index 33d3ce9c888e..a062c662ecff 100644
@@ -83,6 +83,7 @@ source "drivers/infiniband/hw/qib/Kconfig"
 source "drivers/infiniband/hw/cxgb4/Kconfig"
 source "drivers/infiniband/hw/efa/Kconfig"
 source "drivers/infiniband/hw/irdma/Kconfig"
+source "drivers/infiniband/hw/mana/Kconfig"
 source "drivers/infiniband/hw/mlx4/Kconfig"
 source "drivers/infiniband/hw/mlx5/Kconfig"
 source "drivers/infiniband/hw/ocrdma/Kconfig"
drivers/infiniband/hw/Makefile +1 -0
index fba0b3be903e..f62e9e00c780 100644
@@ -4,6 +4,7 @@ obj-$(CONFIG_INFINIBAND_QIB)		+= qib/
 obj-$(CONFIG_INFINIBAND_CXGB4)		+= cxgb4/
 obj-$(CONFIG_INFINIBAND_EFA)		+= efa/
 obj-$(CONFIG_INFINIBAND_IRDMA)		+= irdma/
+obj-$(CONFIG_MANA_INFINIBAND)		+= mana/
 obj-$(CONFIG_MLX4_INFINIBAND)		+= mlx4/
 obj-$(CONFIG_MLX5_INFINIBAND)		+= mlx5/
 obj-$(CONFIG_INFINIBAND_OCRDMA)		+= ocrdma/
drivers/infiniband/hw/mana/Kconfig new file +7 -0
new file mode 100644
index 000000000000..b3ff03a23257
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config MANA_INFINIBAND
+	tristate "Microsoft Azure Network Adapter support"
+	depends on NETDEVICES && ETHERNET && PCI && MICROSOFT_MANA
+	help
+	  This driver provides low-level RDMA support for
+	  Microsoft Azure Network Adapter (MANA).
drivers/infiniband/hw/mana/Makefile new file +4 -0
new file mode 100644
index 000000000000..a799fe264c5a
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_MANA_INFINIBAND) += mana_ib.o
+
+mana_ib-y := main.o wq.o qp.o cq.o mr.o
drivers/infiniband/hw/mana/cq.c new file +80 -0

80 lines not on this page · raw message

drivers/infiniband/hw/mana/main.c new file +681 -0

681 lines not on this page · raw message

drivers/infiniband/hw/mana/mana_ib.h new file +145 -0

145 lines not on this page · raw message

drivers/infiniband/hw/mana/mr.c new file +133 -0

133 lines not on this page · raw message

drivers/infiniband/hw/mana/qp.c new file +501 -0

501 lines not on this page · raw message

drivers/infiniband/hw/mana/wq.c new file +114 -0

114 lines not on this page · raw message

include/net/mana/mana.h +3 -0
index 3a0bc6e0b730..1ff6e0d07cfd 100644
@@ -406,6 +406,9 @@ int mana_bpf(struct net_device *ndev, struct netdev_bpf *bpf);
 
 extern const struct ethtool_ops mana_ethtool_ops;
 
+/* A CQ can be created not associated with any EQ */
+#define GDMA_CQ_NO_EQ  0xffff
+
 struct mana_obj_spec {
 	u32 queue_index;
 	u64 gdma_region;
include/uapi/rdma/ib_user_ioctl_verbs.h +1 -0
index 3072e5d6b692..081aabf536dc 100644
@@ -250,6 +250,7 @@ enum rdma_driver_id {
 	RDMA_DRIVER_QIB,
 	RDMA_DRIVER_EFA,
 	RDMA_DRIVER_SIW,
+	RDMA_DRIVER_MANA,
 };
 
 enum ib_uverbs_gid_type {
include/uapi/rdma/mana-abi.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.17.1
#45 lore raw

Re: [EXTERNAL] [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 2 lines · Ajay Sharma

Hello Maintainers,
Any idea when these patches would make it into the next kernel release ?

New rdma drivers typically take a long time to get merged due to their typical huge size. Currently I'm working through ERDMA. Reviewing the ERDMA submission would be helpful, I generally prefer it if people proposing new drivers review other new drivers being submitted.

In this case it seems smaller, so you might make this cycle, though I haven't even opened the userspace portion yet.

Jason

maintainer Reviewed-by on v4 1/12
#2 lore raw

RE: [Patch v4 01/12] net: mana: Add support for auxiliary device

In reply to [email protected]
Quoted 2 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

...

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_probe_port(struct mana_context *ac, int port_idx, · line 1996

Quoted 3 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+static int add_adev(struct gdma_dev *gd)
+{
+	int ret = 0;

No need to initialize it to 0.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_probe_port(struct mana_context *ac, int port_idx, · line 1998

Quoted 2 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
+	struct mana_adev *madev;
+	struct auxiliary_device *adev;

davem would require the reverse xmas tree order :-)

maintainer Reviewed-by on v4 2/12
#5 lore raw

RE: [Patch v4 02/12] net: mana: Record the physical address for doorbell page region

In reply to [email protected]
Quoted 8 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

For supporting RDMA device with multiple user contexts with their individual doorbell pages, record the start address of doorbell page region for use by the RDMA driver to allocate user context doorbell IDs.

maintainer Reviewed-by on v4 3/12
#7 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to [email protected]
Quoted 13 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

drivers/net/ethernet/microsoft/mana/mana_en.c +11 -0
+void mana_uncfg_vport(struct mana_port_context *apc)
+{
+	mutex_lock(&apc->vport_mutex);
+	apc->vport_use_count--;
+	WARN_ON(apc->vport_use_count < 0);
+	mutex_unlock(&apc->vport_mutex);
+}
+EXPORT_SYMBOL_GPL(mana_uncfg_vport);
+
+int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
+		   u32 doorbell_pg_id)

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_query_vport_cfg(struct mana_port_context *apc, u32 vport_index, · line 553

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +5 -0
 {
 	struct mana_config_vport_resp resp = {};
 	struct mana_config_vport_req req = {};
 	int err;
 
+	/* Ethernet driver and IB driver can't take the port at the same time */
+	mutex_lock(&apc->vport_mutex);
+	if (apc->vport_use_count > 0) {
+		mutex_unlock(&apc->vport_mutex);
+		return -ENODEV;

Maybe -EBUSY is better?

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct mana_port_context · line 587

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct mana_port_context
 *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id, · line 592

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +5 -0
 out:
+	if (err) {
+		mutex_lock(&apc->vport_mutex);
+		apc->vport_use_count--;
+		mutex_unlock(&apc->vport_mutex);
+	}

Change this to the blelow?

    if (err)
        mana_uncfg_vport(apc);

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport_steering(struct · line 660

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -626,6 +654,9 @@ static int mana_cfg_vport_steering(struct
 mana_port_context *apc,
 			   resp.hdr.status);
 		err = -EPROTO;
 	}
+
+	netdev_info(ndev, "Configured steering vPort %llu entries %u\n",
+		    apc->port_handle, num_entries);

netdev_dbg()?

In general, the patch looks good to me.

maintainer
#26 lore raw

RE: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

In reply to [email protected]
Quoted 4 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

The MANA hardware support protection domain and memory registration for

s/support/supports

drivers/net/ethernet/microsoft/mana/gdma.h enum gdma_request_type { · line 33

Quoted 13 lines
drivers/net/ethernet/microsoft/mana/gdma.h +4 -0
index f945755760dc..b1bec8ab5695 100644
@@ -27,6 +27,10 @@ enum gdma_request_type {
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
+	GDMA_CREATE_PD			= 29,
+	GDMA_DESTROY_PD			= 30,
+	GDMA_CREATE_MR			= 31,
+	GDMA_DESTROY_MR			= 32,

These are not used in this patch. They're used in the 12th patch for the first time. Can we move these to that patch?

Quoted 11 lines
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
 #define GDMA_RESOURCE_DOORBELL_PAGE	27
@@ -59,6 +63,8 @@ enum {
 	GDMA_DEVICE_MANA	= 2,
 };
 
+typedef u64 gdma_obj_handle_t;
+
 struct gdma_resource {
 	/* Protect the bitmap */
 	spinlock_t lock;

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_mem_info { · line 201

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/gdma.h +1 -1
@@ -192,7 +198,7 @@ struct gdma_mem_info {
 	u64 length;
 
 	/* Allocated by the PF driver */
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;

The old name "gdma_region" is shorter and it has "gdma" rather than "dma".

The new name is longer. When one starts to read the code for the first time, I feel that "dma_region_handle" might be confusing as it's similar to "dma_handle" (which is the DMA address returned by dma_alloc_coherent()). "dma_region_handle" is an integer rather than a memory address.

You use the new name probably because there is a "mr_handle " in the 12 patch. I prefer the old name, though the new name is also ok to me. If you decide to use the new name, it would be great if this patch could split into two patches: one for the renaming only, and the other for the real changes.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_create_queue_req { · line 608

Quoted 7 lines
drivers/net/ethernet/microsoft/mana/gdma.h +1 -1
 #define REGISTER_ATB_MST_MKEY_LOWER_SIZE 8
@@ -599,7 +605,7 @@ struct gdma_create_queue_req {
 	u32 reserved1;
 	u32 pdid;
 	u32 doolbell_id;
-	u64 gdma_region;
+	gdma_obj_handle_t gdma_region;

If we decide to use the new name "dma_region_handle", should we change the field/param names in the below structs and functions as well (this may not be a complete list)?
struct mana_ib_wq
struct mana_ib_cq
mana_ib_gd_create_dma_region mana_ib_gd_destroy_dma_region

Quoted 18 lines
drivers/net/ethernet/microsoft/mana/gdma.h +11 -0
 	u32 reserved2;
 	u32 queue_size;
 	u32 log2_throttle_limit;
@@ -626,6 +632,28 @@ struct gdma_disable_queue_req {
 	u32 alloc_res_id_on_creation;
 }; /* HW DATA */
 
+enum atb_page_size {
+	ATB_PAGE_SIZE_4K,
+	ATB_PAGE_SIZE_8K,
+	ATB_PAGE_SIZE_16K,
+	ATB_PAGE_SIZE_32K,
+	ATB_PAGE_SIZE_64K,
+	ATB_PAGE_SIZE_128K,
+	ATB_PAGE_SIZE_256K,
+	ATB_PAGE_SIZE_512K,
+	ATB_PAGE_SIZE_1M,
+	ATB_PAGE_SIZE_2M,

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_disable_queue_req { · line 655

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+	ATB_PAGE_SIZE_MAX,
+};
+
+enum gdma_mr_access_flags {
+	GDMA_ACCESS_FLAG_LOCAL_READ = (1 << 0),
+	GDMA_ACCESS_FLAG_LOCAL_WRITE = (1 << 1),
+	GDMA_ACCESS_FLAG_REMOTE_READ = (1 << 2),
+	GDMA_ACCESS_FLAG_REMOTE_WRITE = (1 << 3),
+	GDMA_ACCESS_FLAG_REMOTE_ATOMIC = (1 << 4),
+};

It would be better to use BIT_ULL(0), BIT_ULL(1), etc.

Quoted 22 lines
drivers/net/ethernet/microsoft/mana/gdma.h +2 -2
 /* GDMA_CREATE_DMA_REGION */
 struct gdma_create_dma_region_req {
 	struct gdma_req_hdr hdr;
@@ -652,14 +680,14 @@ struct gdma_create_dma_region_req {
 
 struct gdma_create_dma_region_resp {
 	struct gdma_resp_hdr hdr;
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
 /* GDMA_DMA_REGION_ADD_PAGES */
 struct gdma_dma_region_add_pages_req {
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 
 	u32 page_addr_list_len;
 	u32 reserved3;
@@ -671,9 +699,114 @@ struct gdma_dma_region_add_pages_req {
 struct gdma_destroy_dma_region_req {

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 708

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +5 -1
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
+enum gdma_pd_flags {
+	GDMA_PD_FLAG_ALLOW_GPA_MR = (1 << 0),
+	GDMA_PD_FLAG_ALLOW_FMR_MR = (1 << 1),
+};

Use BIT_ULL(0), BIT_ULL(1) ?

Quoted 40 lines
drivers/net/ethernet/microsoft/mana/gdma.h +40 -0
+struct gdma_create_pd_req {
+	struct gdma_req_hdr hdr;
+	enum gdma_pd_flags flags;
+	u32 reserved;
+};
+
+struct gdma_create_pd_resp {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	u32 pd_id;
+	u32 reserved;
+};
+
+struct gdma_destroy_pd_req {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+};
+
+struct gdma_destory_pd_resp {
+	struct gdma_resp_hdr hdr;
+};
+
+enum gdma_mr_type {
+	/* Guest Physical Address - MRs of this type allow access
+	 * to any DMA-mapped memory using bus-logical address
+	 */
+	GDMA_MR_TYPE_GPA = 1,
+
+	/* Guest Virtual Address - MRs of this type allow access
+	 * to memory mapped by PTEs associated with this MR using a virtual
+	 * address that is set up in the MST
+	 */
+	GDMA_MR_TYPE_GVA,
+
+	/* Fast Memory Register - Like GVA but the MR is initially put in the
+	 * FREE state (as opposed to Valid), and the specified number of
+	 * PTEs are reserved for future fast memory reservations.
+	 */
+	GDMA_MR_TYPE_FMR,
+};

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 759

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+struct gdma_create_mr_params {
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	union {
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;
+		} gva;

Add an empty line to make it more readable?

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 762

Quoted 3 lines
drivers/net/ethernet/microsoft/mana/gdma.h +3 -0
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;

Add an empty line?

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 768

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+		struct {
+			enum atb_page_size page_size;
+			u32  reserved_pte_count;
+		} fmr;
+	};
+};

The definition of struct gdma_create_mr_params is not naturally aligned. This can potenially cause issues.

According to my test, sizeof(struct gdma_create_mr_params) is 40 bytes, meaning the compiler adds two "hidden" fields:

struct gdma_create_mr_params {
gdma_obj_handle_t pd_handle; // offset = 0
enum gdma_mr_type mr_type; // offset = 8

+       u32 hidden_field_a;
        union {                                            // offset = 0x10
                struct {
                        gdma_obj_handle_t dma_region_handle;   // offset =0x10
                        u64 virtual_address;                    // offset =0x18
                        enum gdma_mr_access_flags access_flags;  // offset =0x20
+                       u32 hidden_field_b;
                } gva;

We'll run into trouble some day if the Linux VF driver or the host PF driver adds something like __attribute__((packed)).

Can we work with the host team to improve the definition? If it's hard/impossible to change the PF driver side definition, both sides should at least explicitly define the two hidden fields as reserved fields.

BTW, can we assume the size of "enum" is 4 bytes? I prefer using u32 explicitly when a struct is used to talk to the PF driver or the device.

If we decide to use "enum", I suggest we add BUILD_BUG_ON(sizeof(struct gdma_create_mr_params) != 40) to make sure the assumptin is true.

BTW, Haiyang added "/* HW DATA */ " to other definitions, e.g. gdma_create_queue_resp. Can you please add the same comment for consistency?

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/gdma.h +5 -0
+struct gdma_create_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	u32 reserved;

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 784

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+	union {
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;
+
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;

Similarly, there is a hidden u32 field here. We should explicitly define it.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 783

Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma.h +1 -0
+		} gva;

Can we use the same order of "gva; gpa" used in
struct gdma_create_mr_params?

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 768

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+		struct {
+			enum atb_page_size page_size;
+			u32 reserved_pte_count;
+		} fmr;
+	};
+};

Add BUILD_BUG_ON(sizeof(struct gdma_create_mr_request) != 80) ? Add /* HW DATA */ ?

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+struct gdma_create_mr_response {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+	u32 lkey;
+	u32 rkey;
+};

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 809

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+struct gdma_destroy_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+};
+
+struct gdma_destroy_mr_response {
+	struct gdma_resp_hdr hdr;
+};
+

None of the new defines are really used in this patch:

+enum atb_page_size {
+enum gdma_mr_access_flags {
+enum gdma_pd_flags {
+struct gdma_create_pd_req {
+struct gdma_create_pd_resp {
+struct gdma_destroy_pd_req {
+struct gdma_destory_pd_resp {
+enum gdma_mr_type {
+struct gdma_create_mr_params {
+struct gdma_create_mr_request {
+struct gdma_create_mr_response {
+struct gdma_destroy_mr_request {
+struct gdma_destroy_mr_response

The new defines are used in the 12th patch for the first time. Can we move these to that patch or at least move these defines to before the 12th patch?

maintainer Reviewed-by on v4 5/12
#24 lore raw

RE: [Patch v4 05/12] net: mana: Set the DMA device max segment size

In reply to [email protected]
Quoted 10 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM
...
From: Ajay Sharma <[email protected]>

MANA hardware doesn't have any restrictions on the DMA segment size, set it to the max allowed value.

maintainer Reviewed-by on v4 7/12
#32 lore raw

RE: [Patch v4 07/12] net: mana: Export Work Queue functions for use by RDMA driver

In reply to [email protected]

drivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_send_request(struct gdma_context *gc, · line 129

Quoted 8 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -0
@@ -125,6 +125,7 @@ int mana_gd_send_request(struct gdma_context *gc,
 u32 req_len, const void *req,
 
 	return mana_hwc_send_request(hwc, req_len, req, resp_len, resp);
 }
+EXPORT_SYMBOL(mana_gd_send_request);

Can we use EXPORT_SYMBOL_GPL?

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_create_wq_obj(struct · line 719

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -0
@@ -715,9 +715,10 @@ static int mana_create_wq_obj(struct
 mana_port_context *apc,
 out:
 	return err;
 }
+EXPORT_SYMBOL_GPL(mana_create_wq_obj);

Well, here we use EXPORT_SYMBOL_GPL. If there is a rule to decide which one should be used, please add a comment.

In general, the patch looks good to me.

maintainer Reviewed-by on v4 9/12
#37 lore raw

RE: [Patch v4 09/12] net: mana: Move header files to a common location

In reply to [email protected]
Quoted 11 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM
...
In preparation to add MANA RDMA driver, move all the required header files to a common location for use by both Ethernet and RDMA drivers.

---
Change log:
v2: Move headers to include/net/mana, instead of include/linux/mana
...

Quoted 8 lines
 rename {drivers/net/ethernet/microsoft => include/net}/mana/gdma.h
(100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/hw_channel.h
(100%)
 rename {drivers/net/ethernet/microsoft => include/net}/mana/mana.h
(100%)
 rename {drivers/net/ethernet/microsoft =>

include/net}/mana/shm_channel.h (100%)

While I'm giving my Reviewed-by, I hope someone who has a better judgement can share thoughts as well.

maintainer Reviewed-by on v4 10/12
#39 lore raw

RE: [Patch v4 10/12] net: mana: Define max values for SGL entries

In reply to [email protected]
Quoted 4 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM
...
The number of maximum SGl entries should be computed from the maximum

s/SGl/SGL

include/net/mana/gdma.h struct gdma_wqe { · line 444

Quoted 10 lines
include/net/mana/gdma.h +6 -0
@@ -436,6 +436,13 @@ struct gdma_wqe {
 #define MAX_TX_WQE_SIZE 512
 #define MAX_RX_WQE_SIZE 256
 
+#define MAX_TX_WQE_SGL_ENTRIES	((GDMA_MAX_SQE_SIZE - \
+			sizeof(struct gdma_sge) - INLINE_OOB_SMALL_SIZE) / \
+			sizeof(struct gdma_sge))
+
+#define MAX_RX_WQE_SGL_ENTRIES	((GDMA_MAX_RQE_SIZE - \
+			sizeof(struct gdma_sge)) / sizeof(struct gdma_sge))

Can we make these '\' chars aligned? :-)
Please refer to the definiton of "lock_requestor" in include/linux/hyperv.h.

maintainer Reviewed-by on v4 11/12
#42 lore raw

RE: [Patch v4 11/12] net: mana: Define and process GDMA response code GDMA_STATUS_MORE_ENTRIES

In reply to [email protected]
Quoted 9 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

When doing memory registration, the PF may respond with GDMA_STATUS_MORE_ENTRIES to indicate a follow request is needed. This is not an error and should be processed as expected.

maintainer
#46 lore raw

RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]
Quoted 6 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM
...
+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +1 -0
+				 mana_handle_t *gdma_region, u64 page_sz)

drivers/infiniband/hw/mana/main.c line 233

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+{
+ ...
+	err = mana_gd_send_request(gc, create_req_msg_size, create_req,
+				   sizeof(create_resp), &create_resp);
+	kfree(create_req);
+
+	if (err || create_resp.hdr.status) {
+		ibdev_err(&dev->ib_dev,
+			  "Failed to create DMA region: %d, 0x%x\n", err,
+			  create_resp.hdr.status);
    if (!err)
        err = -EPROTO;

drivers/infiniband/hw/mana/main.c line 300

Quoted 11 lines
drivers/infiniband/hw/mana/main.c new file +11 -0
+		goto error;
+	}
+ ...
+			err = mana_gd_send_request(gc, add_req_msg_size,
+						   add_req, sizeof(add_resp),
+						   &add_resp);
+			if (!err || add_resp.hdr.status != expected_status) {
+				ibdev_err(&dev->ib_dev,
+					  "Failed put DMA pages %u: %d,0x%x\n",
+					  i, err, add_resp.hdr.status);
+				err = -EPROTO;

Should we try to undo what has been done by calling GDMA_DESTROY_DMA_REGION?

Quoted 47 lines
drivers/infiniband/hw/mana/main.c new file +19 -0
+				goto free_req;
+			}
+
+			num_pages_cur += num_pages_to_handle;
+			num_pages_to_handle =
+				min_t(size_t, num_pages_total - num_pages_cur,
+				      max_pgs_add_cmd);
+			add_req_msg_size = sizeof(*add_req) +
+					   num_pages_to_handle * sizeof(u64);
+		}
+free_req:
+		kfree(add_req);
+	}
+
+error:
+	return err;
+}
+ ...
+int mana_ib_gd_create_mr(struct mana_ib_dev *dev, struct mana_ib_mr

*mr,

drivers/infiniband/hw/mana/main.c new file +27 -0
+			 struct gdma_create_mr_params *mr_params)
+{
+	struct gdma_create_mr_response resp = {};
+	struct gdma_create_mr_request req = {};
+	struct gdma_dev *mdev = dev->gdma_dev;
+	struct gdma_context *gc;
+	int err;
+
+	gc = mdev->gdma_context;
+
+	mana_gd_init_req_hdr(&req.hdr, GDMA_CREATE_MR, sizeof(req),
+			     sizeof(resp));
+	req.pd_handle = mr_params->pd_handle;
+
+	switch (mr_params->mr_type) {
+	case GDMA_MR_TYPE_GVA:
+		req.mr_type = GDMA_MR_TYPE_GVA;
+		req.gva.dma_region_handle = mr_params->gva.dma_region_handle;
+		req.gva.virtual_address = mr_params->gva.virtual_address;
+		req.gva.access_flags = mr_params->gva.access_flags;
+		break;
+
+	case GDMA_MR_TYPE_GPA:
+		req.mr_type = GDMA_MR_TYPE_GPA;
+		req.gpa.access_flags = mr_params->gpa.access_flags;
+		break;
+

drivers/infiniband/hw/mana/main.c line 429

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+	case GDMA_MR_TYPE_FMR:
+		req.mr_type = GDMA_MR_TYPE_FMR;
+		req.fmr.page_size = mr_params->fmr.page_size;
+		req.fmr.reserved_pte_count = mr_params->fmr.reserved_pte_count;
+		break;
+
+	default:
+		ibdev_dbg(&dev->ib_dev,
+			  "invalid param (GDMA_MR_TYPE) passed, type %d\n",
+			  req.mr_type);

Here req.mr_type is always 0.
We should remove the 3 above lines of "req.mr_type = ...", and add a line "req.mr_type = mr_params->mr_type;" before the "switch" line..

drivers/infiniband/hw/mana/main.c line 438

Quoted 9 lines
drivers/infiniband/hw/mana/main.c new file +9 -0
+		err = -EINVAL;
+		goto error;
+	}
+
+	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
+
+	if (err || resp.hdr.status) {
+		ibdev_err(&dev->ib_dev, "Failed to create mr %d, %u", err,
+			  resp.hdr.status);
    if (!err)
        err = -EPROTO;
Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+		goto error;
+	}
+
+	mr->ibmr.lkey = resp.lkey;
+	mr->ibmr.rkey = resp.rkey;
+	mr->mr_handle = resp.mr_handle;
+
+	return 0;
+error:
+	return err;

drivers/infiniband/hw/mana/main.c line 608

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+}
+ ...
+static int mana_ib_probe(struct auxiliary_device *adev,
+			 const struct auxiliary_device_id *id)
+{
+	struct mana_adev *madev = container_of(adev, struct mana_adev, adev);
+	struct gdma_dev *mdev = madev->mdev;
+	struct mana_context *mc;
+	struct mana_ib_dev *dev;
+	int ret = 0;

No need to initialize 'ret' to 0.

drivers/infiniband/hw/mana/mr.c line 124

Quoted 11 lines
drivers/infiniband/hw/mana/mr.c new file +11 -0
+int mana_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
+{
+	struct mana_ib_mr *mr = container_of(ibmr, struct mana_ib_mr, ibmr);
+	struct ib_device *ibdev = ibmr->device;
+	struct mana_ib_dev *dev;
+	int err;
+
+	dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
+
+	err = mana_ib_gd_destroy_mr(dev, mr->mr_handle);
+	if (err)

Should we return here without calling ib_umem_release() and kfree(mr)?

Quoted 10 lines
drivers/infiniband/hw/mana/mr.c new file +9 -0
+		return err;
 
+
+	if (mr->umem)
+		ib_umem_release(mr->umem);
+
+	kfree(mr);
+
+	return 0;
+}
#3 lore raw

RE: [Patch v4 01/12] net: mana: Add support for auxiliary device

In reply to Dexuan Cui
Quoted 5 lines

Subject: RE: [Patch v4 01/12] net: mana: Add support for auxiliary device

Quoted 2 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

...

Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+static int add_adev(struct gdma_dev *gd) {
+	int ret = 0;

No need to initialize it to 0.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_probe_port(struct mana_context *ac, int port_idx, · line 1998

Quoted 2 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
+	struct mana_adev *madev;
+	struct auxiliary_device *adev;

davem would require the reverse xmas tree order :-)

Thank you, will send v5 to fix this.

Quoted 2 lines
#8 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Dexuan Cui
Quoted 19 lines

Subject: RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 17 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

drivers/net/ethernet/microsoft/mana/mana_en.c +10 -0
+void mana_uncfg_vport(struct mana_port_context *apc) {
+	mutex_lock(&apc->vport_mutex);
+	apc->vport_use_count--;
+	WARN_ON(apc->vport_use_count < 0);
+	mutex_unlock(&apc->vport_mutex);
+}
+EXPORT_SYMBOL_GPL(mana_uncfg_vport);
+
+int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id,
+		   u32 doorbell_pg_id)
 {
 	struct mana_config_vport_resp resp = {};
 	struct mana_config_vport_req req = {};
 	int err;
Quoted 6 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_query_vport_cfg(struct mana_port_context *apc, u32 vport_index, · line 553

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +5 -0
+	/* Ethernet driver and IB driver can't take the port at the same time */
+	mutex_lock(&apc->vport_mutex);
+	if (apc->vport_use_count > 0) {
+		mutex_unlock(&apc->vport_mutex);
+		return -ENODEV;

Maybe -EBUSY is better?

I agree with you, EBUSY is a better value. Will change this in v5.

Quoted 4 lines
Quoted 4 lines
(quoted diff) +0 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
 mana_port_context *apc, u32 protection_dom_id,
 
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id, · line 586

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

The reason netdev_info () is used is that this message is important for troubleshooting initial setup issues with Ethernet driver. We rely on user to get this configured right to share the same hardware port between Ethernet and RDMA driver. As far as I know, there is no easy way for a driver to "take over" an exclusive hardware resource from another driver.

If it is acceptable that we have one such message for each opened Ethernet port on the system, I suggest we keep it this way.

Quoted 25 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id, · line 592

Quoted 7 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +5 -0
 out:
+	if (err) {
+		mutex_lock(&apc->vport_mutex);
+		apc->vport_use_count--;
+		mutex_unlock(&apc->vport_mutex);
+	}

Change this to the blelow?

    if (err)
        mana_uncfg_vport(apc);

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport_steering(struct · line 660

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -626,6 +654,9 @@ static int mana_cfg_vport_steering(struct
 mana_port_context *apc,
 			   resp.hdr.status);
 		err = -EPROTO;
 	}
+
+	netdev_info(ndev, "Configured steering vPort %llu entries %u\n",
+		    apc->port_handle, num_entries);

netdev_dbg()?

In general, the patch looks good to me.

#21 lore raw

RE: [Patch v4 04/12] net: mana: Add functions for allocating doorbell page from GDMA

In reply to Dexuan Cui
Quoted 10 lines

Subject: RE: [Patch v4 04/12] net: mana: Add functions for allocating doorbell page from GDMA

Quoted 3 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM ... +EXPORT_SYMBOL(mana_gd_destroy_doorbell_page);

Can this be EXPORT_SYMBOL_GPL?

Quoted 1 line

+EXPORT_SYMBOL(mana_gd_allocate_doorbell_page);

EXPORT_SYMBOL_GPL?

Will fix in v5.

Quoted 2 lines
#33 lore raw

RE: [Patch v4 07/12] net: mana: Export Work Queue functions for use by RDMA driver

In reply to Dexuan Cui
Quoted 10 lines

Subject: RE: [Patch v4 07/12] net: mana: Export Work Queue functions for use by RDMA driver

Review

Quoted 7 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM @@ -125,6 +125,7 @@ int mana_gd_send_request(struct gdma_context *gc,
u32 req_len, const void *req,

(quoted diff) +1 -0
 	return mana_hwc_send_request(hwc, req_len, req, resp_len, resp);  }
+EXPORT_SYMBOL(mana_gd_send_request);
Quoted 9 lines

Can we use EXPORT_SYMBOL_GPL?

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_create_wq_obj(struct · line 719

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -0
@@ -715,9 +715,10 @@ static int mana_create_wq_obj(struct
 mana_port_context *apc,
 out:
 	return err;
 }
+EXPORT_SYMBOL_GPL(mana_create_wq_obj);

Will fix this in v5.

Quoted 6 lines

Well, here we use EXPORT_SYMBOL_GPL. If there is a rule to decide which one should be used, please add a comment.

In general, the patch looks good to me.

#40 lore raw

RE: [Patch v4 10/12] net: mana: Define max values for SGL entries

In reply to Dexuan Cui
Quoted 10 lines

Subject: RE: [Patch v4 10/12] net: mana: Define max values for SGL entries

Quoted 3 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM ...
The number of maximum SGl entries should be computed from the maximum

s/SGl/SGL

Quoted 3 lines
include/net/mana/gdma.h +0 -0
@@ -436,6 +436,13 @@ struct gdma_wqe {  #define MAX_TX_WQE_SIZE 512
 #define MAX_RX_WQE_SIZE 256
 
Quoted 9 lines

include/net/mana/gdma.h struct gdma_wqe { · line 444

Quoted 6 lines
include/net/mana/gdma.h +6 -0
+#define MAX_TX_WQE_SGL_ENTRIES	((GDMA_MAX_SQE_SIZE - \
+			sizeof(struct gdma_sge) - INLINE_OOB_SMALL_SIZE) / \
+			sizeof(struct gdma_sge))
+
+#define MAX_RX_WQE_SGL_ENTRIES	((GDMA_MAX_RQE_SIZE - \
+			sizeof(struct gdma_sge)) / sizeof(struct gdma_sge))

Can we make these '\' chars aligned? :-) Please refer to the definiton of "lock_requestor" in include/linux/hyperv.h.

Will fix this.

Quoted 2 lines
#49 lore raw

Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]

drivers/infiniband/hw/mana/cq.c line 1

Quoted 4 lines · On Wed, Jun 15, 2022 at 07:07:20PM -0700, [email protected] wrote:
drivers/infiniband/hw/mana/cq.c new file +1 -0
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB

Why perpetuate this mess that the OpenIB people created? I thought that no new drivers were going to be added with this, why does this one need to have it as well if it is new?

thanks,

greg k-h

#50 lore raw

RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 4 lines

Subject: Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

On Wed, Jun 15, 2022 at 07:07:20PM -0700, [email protected] wrote:

Quoted 8 lines

drivers/infiniband/hw/mana/cq.c line 1

Quoted 4 lines
drivers/infiniband/hw/mana/cq.c new file +1 -0
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB

Why perpetuate this mess that the OpenIB people created? I thought that no new drivers were going to be added with this, why does this one need to have it as well if it is new?

I apologize for the incorrect license language. I followed other RDMA driver's license terms but didn't' realized their licensing language is not up to the standard.

The newly introduced EFA RDMA driver used the following license terms:
(drivers/infiniband/hw/efa/efa_main.c)
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause

Is it acceptable that we use the same license terms?

Thanks,
Long

#47 lore raw

RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Dexuan Cui

Please see comments inline

-----Original Message-----
From: Dexuan Cui <[email protected]>
Sent: Sunday, July 10, 2022 8:43 PM
To: Long Li <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

drivers/infiniband/hw/mana/main.c line 233

Quoted 13 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM ...
+int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c new file +9 -0
+				 mana_handle_t *gdma_region, u64 page_sz) {  ...
+	err = mana_gd_send_request(gc, create_req_msg_size, create_req,
+				   sizeof(create_resp), &create_resp);
+	kfree(create_req);
+
+	if (err || create_resp.hdr.status) {
+		ibdev_err(&dev->ib_dev,
+			  "Failed to create DMA region: %d, 0x%x\n", err,
+			  create_resp.hdr.status);
    if (!err)
        err = -EPROTO;

drivers/infiniband/hw/mana/main.c line 300

Quoted 11 lines
drivers/infiniband/hw/mana/main.c new file +11 -0
+		goto error;
+	}
+ ...
+			err = mana_gd_send_request(gc, add_req_msg_size,
+						   add_req, sizeof(add_resp),
+						   &add_resp);
+			if (!err || add_resp.hdr.status != expected_status) {
+				ibdev_err(&dev->ib_dev,
+					  "Failed put DMA pages %u: %d,0x%x\n",
+					  i, err, add_resp.hdr.status);
+				err = -EPROTO;

Should we try to undo what has been done by calling GDMA_DESTROY_DMA_REGION? Yes, I updated the patch.

Quoted 46 lines
drivers/infiniband/hw/mana/main.c new file +19 -0
+				goto free_req;
+			}
+
+			num_pages_cur += num_pages_to_handle;
+			num_pages_to_handle =
+				min_t(size_t, num_pages_total - num_pages_cur,
+				      max_pgs_add_cmd);
+			add_req_msg_size = sizeof(*add_req) +
+					   num_pages_to_handle * sizeof(u64);
+		}
+free_req:
+		kfree(add_req);
+	}
+
+error:
+	return err;
+}
+ ...
+int mana_ib_gd_create_mr(struct mana_ib_dev *dev, struct mana_ib_mr

*mr,

drivers/infiniband/hw/mana/main.c new file +26 -0
+			 struct gdma_create_mr_params *mr_params) {
+	struct gdma_create_mr_response resp = {};
+	struct gdma_create_mr_request req = {};
+	struct gdma_dev *mdev = dev->gdma_dev;
+	struct gdma_context *gc;
+	int err;
+
+	gc = mdev->gdma_context;
+
+	mana_gd_init_req_hdr(&req.hdr, GDMA_CREATE_MR, sizeof(req),
+			     sizeof(resp));
+	req.pd_handle = mr_params->pd_handle;
+
+	switch (mr_params->mr_type) {
+	case GDMA_MR_TYPE_GVA:
+		req.mr_type = GDMA_MR_TYPE_GVA;
+		req.gva.dma_region_handle = mr_params->gva.dma_region_handle;
+		req.gva.virtual_address = mr_params->gva.virtual_address;
+		req.gva.access_flags = mr_params->gva.access_flags;
+		break;
+
+	case GDMA_MR_TYPE_GPA:
+		req.mr_type = GDMA_MR_TYPE_GPA;
+		req.gpa.access_flags = mr_params->gpa.access_flags;
+		break;
+

drivers/infiniband/hw/mana/main.c line 429

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+	case GDMA_MR_TYPE_FMR:
+		req.mr_type = GDMA_MR_TYPE_FMR;
+		req.fmr.page_size = mr_params->fmr.page_size;
+		req.fmr.reserved_pte_count = mr_params->fmr.reserved_pte_count;
+		break;
+
+	default:
+		ibdev_dbg(&dev->ib_dev,
+			  "invalid param (GDMA_MR_TYPE) passed, type %d\n",
+			  req.mr_type);

Here req.mr_type is always 0.
We should remove the 3 above lines of "req.mr_type = ...", and add a line "req.mr_type = mr_params->mr_type;" before the "switch" line..

No, That's incorrect. The mr_type is being explicitly set here to control what regions get exposed to the user and kernel. GPA and FMR are never exposed to user. So we cannot assign req.mr_type = mr_params->mr_type.

drivers/infiniband/hw/mana/main.c line 438

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+		err = -EINVAL;
+		goto error;
+	}
+
+	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), 
+&resp);
+
+	if (err || resp.hdr.status) {
+		ibdev_err(&dev->ib_dev, "Failed to create mr %d, %u", err,
+			  resp.hdr.status);
    if (!err)
        err = -EPROTO;
Quoted 9 lines
drivers/infiniband/hw/mana/main.c new file +9 -0
+		goto error;
+	}
+
+	mr->ibmr.lkey = resp.lkey;
+	mr->ibmr.rkey = resp.rkey;
+	mr->mr_handle = resp.mr_handle;
+
+	return 0;
+error:

drivers/infiniband/hw/mana/main.c

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+	return err;
+}
+ ...
+static int mana_ib_probe(struct auxiliary_device *adev,
+			 const struct auxiliary_device_id *id) {
+	struct mana_adev *madev = container_of(adev, struct mana_adev, adev);
+	struct gdma_dev *mdev = madev->mdev;
+	struct mana_context *mc;
+	struct mana_ib_dev *dev;
+	int ret = 0;

No need to initialize 'ret' to 0.
Agreed. Updated the patch.

drivers/infiniband/hw/mana/mr.c line 124

Quoted 10 lines
drivers/infiniband/hw/mana/mr.c new file +10 -0
+int mana_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) {
+	struct mana_ib_mr *mr = container_of(ibmr, struct mana_ib_mr, ibmr);
+	struct ib_device *ibdev = ibmr->device;
+	struct mana_ib_dev *dev;
+	int err;
+
+	dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
+
+	err = mana_ib_gd_destroy_mr(dev, mr->mr_handle);
+	if (err)

Should we return here without calling ib_umem_release() and kfree(mr)?
Yes, if the device fails to deallocate the resources and we release them back to kernel it will lead to unexpected results.

Quoted 10 lines
drivers/infiniband/hw/mana/mr.c new file +9 -0
+		return err;
 
+
+	if (mr->umem)
+		ib_umem_release(mr->umem);
+
+	kfree(mr);
+
+	return 0;
+}
#27 lore raw

RE: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

In reply to Dexuan Cui

Please see inline.

-----Original Message-----
From: Dexuan Cui <[email protected]>
Sent: Sunday, July 10, 2022 8:29 PM
To: Long Li <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

Quoted 5 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

The MANA hardware support protection domain and memory registration for

s/support/supports

drivers/net/ethernet/microsoft/mana/gdma.h enum gdma_request_type { · line 33

Quoted 13 lines
drivers/net/ethernet/microsoft/mana/gdma.h +4 -0
index f945755760dc..b1bec8ab5695 100644
@@ -27,6 +27,10 @@ enum gdma_request_type {
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
+	GDMA_CREATE_PD			= 29,
+	GDMA_DESTROY_PD			= 30,
+	GDMA_CREATE_MR			= 31,
+	GDMA_DESTROY_MR			= 32,

These are not used in this patch. They're used in the 12th patch for the first time. Can we move these to that patch?

Quoted 11 lines
drivers/net/ethernet/microsoft/mana/gdma.h +2 -0
 #define GDMA_RESOURCE_DOORBELL_PAGE	27
@@ -59,6 +63,8 @@ enum {
 	GDMA_DEVICE_MANA	= 2,
 };
 
+typedef u64 gdma_obj_handle_t;
+
 struct gdma_resource {
 	/* Protect the bitmap */
 	spinlock_t lock;

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_mem_info { · line 201

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/gdma.h +1 -1
@@ -192,7 +198,7 @@ struct gdma_mem_info {
 	u64 length;
 
 	/* Allocated by the PF driver */
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;

The old name "gdma_region" is shorter and it has "gdma" rather than "dma".

The new name is longer. When one starts to read the code for the first time, I feel that "dma_region_handle" might be confusing as it's similar to "dma_handle" (which is the DMA address returned by dma_alloc_coherent()). "dma_region_handle" is an integer rather than a memory address.

You use the new name probably because there is a "mr_handle "
in the 12 patch. I prefer the old name, though the new name is also ok to me. If you decide to use the new name, it would be great if this patch could split into two patches: one for the renaming only, and the other for the real changes.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_create_queue_req { · line 608

Quoted 7 lines
drivers/net/ethernet/microsoft/mana/gdma.h +0 -0
 #define REGISTER_ATB_MST_MKEY_LOWER_SIZE 8 @@ -599,7 +605,7 @@ struct 
 gdma_create_queue_req {
 	u32 reserved1;
drivers/net/ethernet/microsoft/mana/gdma.h +1 -1
 	u32 pdid;
 	u32 doolbell_id;
-	u64 gdma_region;
+	gdma_obj_handle_t gdma_region;

If we decide to use the new name "dma_region_handle", should we change the field/param names in the below structs and functions as well (this may not be a complete list)?
struct mana_ib_wq
struct mana_ib_cq
mana_ib_gd_create_dma_region
mana_ib_gd_destroy_dma_region

Quoted 17 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
 	u32 reserved2;
 	u32 queue_size;
 	u32 log2_throttle_limit;
@@ -626,6 +632,28 @@ struct gdma_disable_queue_req {
 	u32 alloc_res_id_on_creation;
 }; /* HW DATA */
 
+enum atb_page_size {
+	ATB_PAGE_SIZE_4K,
+	ATB_PAGE_SIZE_8K,
+	ATB_PAGE_SIZE_16K,
+	ATB_PAGE_SIZE_32K,
+	ATB_PAGE_SIZE_64K,
+	ATB_PAGE_SIZE_128K,
+	ATB_PAGE_SIZE_256K,
+	ATB_PAGE_SIZE_512K,
+	ATB_PAGE_SIZE_1M,

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_disable_queue_req { · line 654

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+	ATB_PAGE_SIZE_2M,
+	ATB_PAGE_SIZE_MAX,
+};
+
+enum gdma_mr_access_flags {
+	GDMA_ACCESS_FLAG_LOCAL_READ = (1 << 0),
+	GDMA_ACCESS_FLAG_LOCAL_WRITE = (1 << 1),
+	GDMA_ACCESS_FLAG_REMOTE_READ = (1 << 2),
+	GDMA_ACCESS_FLAG_REMOTE_WRITE = (1 << 3),
+	GDMA_ACCESS_FLAG_REMOTE_ATOMIC = (1 << 4), };

It would be better to use BIT_ULL(0), BIT_ULL(1), etc. Agreed, updated in the new patch.

Quoted 21 lines
drivers/net/ethernet/microsoft/mana/gdma.h +2 -2
 /* GDMA_CREATE_DMA_REGION */
 struct gdma_create_dma_region_req {
 	struct gdma_req_hdr hdr;
@@ -652,14 +680,14 @@ struct gdma_create_dma_region_req {
 
 struct gdma_create_dma_region_resp {
 	struct gdma_resp_hdr hdr;
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
 /* GDMA_DMA_REGION_ADD_PAGES */
 struct gdma_dma_region_add_pages_req {
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 
 	u32 page_addr_list_len;
 	u32 reserved3;

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { struct · line 707

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +4 -1
@@ -671,9 +699,114 @@ struct gdma_dma_region_add_pages_req {  struct 
 gdma_destroy_dma_region_req {
 	struct gdma_req_hdr hdr;
 
-	u64 gdma_region;
+	gdma_obj_handle_t dma_region_handle;
 }; /* HW DATA */
 
+enum gdma_pd_flags {
+	GDMA_PD_FLAG_ALLOW_GPA_MR = (1 << 0),
+	GDMA_PD_FLAG_ALLOW_FMR_MR = (1 << 1), };

Use BIT_ULL(0), BIT_ULL(1) ?
Agreed and updated the patch

Quoted 40 lines
drivers/net/ethernet/microsoft/mana/gdma.h +40 -0
+struct gdma_create_pd_req {
+	struct gdma_req_hdr hdr;
+	enum gdma_pd_flags flags;
+	u32 reserved;
+};
+
+struct gdma_create_pd_resp {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	u32 pd_id;
+	u32 reserved;
+};
+
+struct gdma_destroy_pd_req {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+};
+
+struct gdma_destory_pd_resp {
+	struct gdma_resp_hdr hdr;
+};
+
+enum gdma_mr_type {
+	/* Guest Physical Address - MRs of this type allow access
+	 * to any DMA-mapped memory using bus-logical address
+	 */
+	GDMA_MR_TYPE_GPA = 1,
+
+	/* Guest Virtual Address - MRs of this type allow access
+	 * to memory mapped by PTEs associated with this MR using a virtual
+	 * address that is set up in the MST
+	 */
+	GDMA_MR_TYPE_GVA,
+
+	/* Fast Memory Register - Like GVA but the MR is initially put in the
+	 * FREE state (as opposed to Valid), and the specified number of
+	 * PTEs are reserved for future fast memory reservations.
+	 */
+	GDMA_MR_TYPE_FMR,
+};

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 759

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+struct gdma_create_mr_params {
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	union {
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;
+		} gva;

Add an empty line to make it more readable?
Done.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 762

Quoted 3 lines
drivers/net/ethernet/microsoft/mana/gdma.h +3 -0
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;

Add an empty line?

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 768

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+		struct {
+			enum atb_page_size page_size;
+			u32  reserved_pte_count;
+		} fmr;
+	};
+};

The definition of struct gdma_create_mr_params is not naturally aligned.
This can potenially cause issues.
This is union and so the biggest element is aligned to word. I feel since this is not passed to the hw it should be fine.

According to my test, sizeof(struct gdma_create_mr_params) is 40 bytes, meaning the compiler adds two "hidden" fields:

struct gdma_create_mr_params {
gdma_obj_handle_t pd_handle; // offset = 0
enum gdma_mr_type mr_type; // offset = 8

+       u32 hidden_field_a;
        union {                                            // offset = 0x10
                struct {
                        gdma_obj_handle_t dma_region_handle;   // offset =0x10
                        u64 virtual_address;                    // offset =0x18
                        enum gdma_mr_access_flags access_flags;  // offset =0x20
+                       u32 hidden_field_b;
                } gva;

We'll run into trouble some day if the Linux VF driver or the host PF driver adds something like __attribute__((packed)).

Can we work with the host team to improve the definition? If it's hard/impossible to change the PF driver side definition, both sides should at least explicitly define the two hidden fields as reserved fields.

BTW, can we assume the size of "enum" is 4 bytes? I prefer using u32 explicitly when a struct is used to talk to the PF driver or the device.

If we decide to use "enum", I suggest we add BUILD_BUG_ON(sizeof(struct gdma_create_mr_params) != 40) to make sure the assumptin is true.

BTW, Haiyang added "/* HW DATA */ " to other definitions, e.g. gdma_create_queue_resp. Can you please add the same comment for consistency?

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/gdma.h +5 -0
+struct gdma_create_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	u32 reserved;

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 784

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+	union {
+		struct {
+			enum gdma_mr_access_flags access_flags;
+		} gpa;
+
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;

Similarly, there is a hidden u32 field here. We should explicitly define it.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { struct · line 782

Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma.h +1 -0
+		} gva;

Can we use the same order of "gva; gpa" used in struct gdma_create_mr_params? Done, although it shouldn't matter in union case.

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 768

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+		struct {
+			enum atb_page_size page_size;
+			u32 reserved_pte_count;
+		} fmr;
+	};
+};

Add BUILD_BUG_ON(sizeof(struct gdma_create_mr_request) != 80) ? Add /* HW DATA */ ?

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+struct gdma_create_mr_response {
+	struct gdma_resp_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+	u32 lkey;
+	u32 rkey;
+};

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 809

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma.h +10 -0
+
+struct gdma_destroy_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t mr_handle;
+};
+
+struct gdma_destroy_mr_response {
+	struct gdma_resp_hdr hdr;
+};
+

None of the new defines are really used in this patch:

+enum atb_page_size {
+enum gdma_mr_access_flags {
+enum gdma_pd_flags {
+struct gdma_create_pd_req {
+struct gdma_create_pd_resp {
+struct gdma_destroy_pd_req {
+struct gdma_destory_pd_resp {
+enum gdma_mr_type {
+struct gdma_create_mr_params {
+struct gdma_create_mr_request {
+struct gdma_create_mr_response {
+struct gdma_destroy_mr_request {
+struct gdma_destroy_mr_response

The new defines are used in the 12th patch for the first time.
Can we move these to that patch or at least move these defines to before the 12th patch?

#51 lore raw

Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Long Li
Quoted 13 lines · Long Li
Quoted 4 lines

Subject: Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

On Wed, Jun 15, 2022 at 07:07:20PM -0700, [email protected] wrote:

Quoted 8 lines

drivers/infiniband/hw/mana/cq.c line 1

Quoted 4 lines
drivers/infiniband/hw/mana/cq.c new file +1 -0
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB

Why perpetuate this mess that the OpenIB people created? I thought that no new drivers were going to be added with this, why does this one need to have it as well if it is new?

Quoted 1 line

I apologize for the incorrect license language. I followed other RDMA driver's license terms but didn't' realized their licensing language is not up to the standard.

You need to follow the license rules of your employer, please consult with them as they know what to do here.

Quoted 5 lines

The newly introduced EFA RDMA driver used the following license terms:
(drivers/infiniband/hw/efa/efa_main.c)
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause

Is it acceptable that we use the same license terms?

Again, discuss this with the lawyers at your company. But if you are going to use a dual-license, you must be prepared to defend why you are doing so.

thanks,

greg k-h

maintainer
#28 lore raw

RE: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

In reply to Ajay Sharma
Quoted 8 lines

From: Ajay Sharma <[email protected]>
Sent: Tuesday, July 12, 2022 9:39 PM
To: Dexuan Cui <[email protected]>; Long Li <[email protected]>; KY
...

Quoted 2 lines

The definition of struct gdma_create_mr_params is not naturally aligned. This can potenially cause issues.

This is union and so the biggest element is aligned to word. I feel since this is not passed to the hw it should be fine.

Ajay, you're right. I didn't realize struct gdma_create_mr_params is not really passed to the PF driver or the device. Please ignore my comments on struct gdma_create_mr_params. Sorry for the confusion!

Quoted 3 lines
Quoted 3 lines

BTW, Haiyang added "/* HW DATA */ " to other definitions, e.g. gdma_create_queue_resp. Can you please add the same comment for consistency?

It's still recommended that we add the tag "/* HW DATA */ " to new definitions that are passed to the PF driver or the device.

Quoted 9 lines
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/gdma.h +9 -0
+struct gdma_create_mr_request {
+	struct gdma_req_hdr hdr;
+	gdma_obj_handle_t pd_handle;
+	enum gdma_mr_type mr_type;
+	u32 reserved;
+
+	union {
+		struct {
+			enum gdma_mr_access_flags access_flags;
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/gdma.h struct gdma_dma_region_add_pages_req { · line 784

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma.h +6 -0
+		} gpa;
+
+		struct {
+			gdma_obj_handle_t dma_region_handle;
+			u64 virtual_address;
+			enum gdma_mr_access_flags access_flags;

Similarly, there is a hidden u32 field here. We should explicitly define it.

The issue with struct gdma_create_mr_request is valid, since it's passed to the PF driver. We should explicitly define the hidden field.

maintainer
#48 lore raw

RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Ajay Sharma
Quoted 31 lines

From: Ajay Sharma <[email protected]>
Sent: Tuesday, July 12, 2022 9:33 PM

Quoted 14 lines

...

(quoted diff) +13 -0
+	switch (mr_params->mr_type) {
+	case GDMA_MR_TYPE_GVA:
+		req.mr_type = GDMA_MR_TYPE_GVA;
+		req.gva.dma_region_handle = mr_params->gva.dma_region_handle;
+		req.gva.virtual_address = mr_params->gva.virtual_address;
+		req.gva.access_flags = mr_params->gva.access_flags;
+		break;
+
+	case GDMA_MR_TYPE_GPA:
+		req.mr_type = GDMA_MR_TYPE_GPA;
+		req.gpa.access_flags = mr_params->gpa.access_flags;
+		break;
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+	case GDMA_MR_TYPE_FMR:
+		req.mr_type = GDMA_MR_TYPE_FMR;
+		req.fmr.page_size = mr_params->fmr.page_size;
+		req.fmr.reserved_pte_count = mr_params->fmr.reserved_pte_count;
+		break;
+
+	default:
+		ibdev_dbg(&dev->ib_dev,
+			  "invalid param (GDMA_MR_TYPE) passed, type %d\n",
+			  req.mr_type);

Here req.mr_type is always 0.
We should remove the 3 above lines of "req.mr_type = ...", and add a line "req.mr_type = mr_params->mr_type;" before the "switch" line..

Quoted 3 lines

No, That's incorrect. The mr_type is being explicitly set here to control what regions get exposed to the user and kernel. GPA and FMR are never exposed to user. So we cannot assign req.mr_type = mr_params->mr_type.

I'm not following you. I meant the below change, which should have no functional change, right? In the "default:" branch , we just "goto error;", so there is no functional change either.

infiniband/hw/mana/main.c renamed from infiniband/hw/mana/main.c.orig +1 -3
@@ -394,21 +394,19 @@
                             sizeof(resp));
        req.pd_handle = mr_params->pd_handle;
 
+       req.mr_type = mr_params->mr_type;
        switch (mr_params->mr_type) {
        case GDMA_MR_TYPE_GVA:
-               req.mr_type = GDMA_MR_TYPE_GVA;
                req.gva.dma_region_handle = mr_params->gva.dma_region_handle;
                req.gva.virtual_address = mr_params->gva.virtual_address;
                req.gva.access_flags = mr_params->gva.access_flags;
                break;
 
        case GDMA_MR_TYPE_GPA:
-               req.mr_type = GDMA_MR_TYPE_GPA;
                req.gpa.access_flags = mr_params->gpa.access_flags;
                break;
 
        case GDMA_MR_TYPE_FMR:
-               req.mr_type = GDMA_MR_TYPE_FMR;
                req.fmr.page_size = mr_params->fmr.page_size;
                req.fmr.reserved_pte_count = mr_params->fmr.reserved_pte_count;
                break;
#9 lore raw

Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Long Li
Quoted 12 lines · Long Li
Quoted 11 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct · line 587

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
 mana_port_context *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

Quoted 6 lines

The reason netdev_info () is used is that this message is important for troubleshooting initial setup issues with Ethernet driver. We rely on user to get this configured right to share the same hardware port between Ethernet and RDMA driver. As far as I know, there is no easy way for a driver to "take over" an exclusive hardware resource from another driver.

This seems like a really strange statement.

Exactly how does all of this work?

Jason

#22 lore raw

Re: [Patch v4 04/12] net: mana: Add functions for allocating doorbell page from GDMA

In reply to Dexuan Cui
Quoted 8 lines · Dexuan Cui
Quoted 4 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM
...
+EXPORT_SYMBOL(mana_gd_destroy_doorbell_page);

Can this be EXPORT_SYMBOL_GPL?

Quoted 1 line

+EXPORT_SYMBOL(mana_gd_allocate_doorbell_page);

EXPORT_SYMBOL_GPL?

Can you think about using the symbol namespaces here?

Nobody else has done it yet, but I think we should be...

Jason

#29 lore raw

Re: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

In reply to Dexuan Cui
Quoted 6 lines · Dexuan Cui
Quoted 4 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

The MANA hardware support protection domain and memory registration for

s/support/supports

Quoted 15 lines

drivers/net/ethernet/microsoft/mana/gdma.h enum gdma_request_type { · line 33

Quoted 13 lines
drivers/net/ethernet/microsoft/mana/gdma.h +4 -0
index f945755760dc..b1bec8ab5695 100644
@@ -27,6 +27,10 @@ enum gdma_request_type {
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
+	GDMA_CREATE_PD			= 29,
+	GDMA_DESTROY_PD			= 30,
+	GDMA_CREATE_MR			= 31,
+	GDMA_DESTROY_MR			= 32,

These are not used in this patch. They're used in the 12th patch for the first time. Can we move these to that patch?

This looks like RDMA code anyhow, why is it under net/ethernet?

Jason

maintainer
#52 lore raw

Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]

drivers/infiniband/hw/mana/main.c line 92

Quoted 13 lines · On Wed, Jun 15, 2022 at 07:07:20PM -0700, [email protected] wrote:
drivers/infiniband/hw/mana/main.c new file +13 -0
+static int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
+{
+	struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
+	struct ib_device *ibdev = ibpd->device;
+	enum gdma_pd_flags flags = 0;
+	struct mana_ib_dev *dev;
+	int ret;
+
+	dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
+
+	/* Set flags if this is a kernel request */
+	if (!ibpd->uobject)
+		flags = GDMA_PD_FLAG_ALLOW_GPA_MR | GDMA_PD_FLAG_ALLOW_FMR_MR;

I'm confused, this driver doesn't seem to support kverbs:

Quoted 17 lines
drivers/infiniband/hw/mana/main.c new file +17 -0
+static const struct ib_device_ops mana_ib_dev_ops = {
+	.owner = THIS_MODULE,
+	.driver_id = RDMA_DRIVER_MANA,
+	.uverbs_abi_ver = MANA_IB_UVERBS_ABI_VERSION,
+
+	.alloc_pd = mana_ib_alloc_pd,
+	.alloc_ucontext = mana_ib_alloc_ucontext,
+	.create_cq = mana_ib_create_cq,
+	.create_qp = mana_ib_create_qp,
+	.create_rwq_ind_table = mana_ib_create_rwq_ind_table,
+	.create_wq = mana_ib_create_wq,
+	.dealloc_pd = mana_ib_dealloc_pd,
+	.dealloc_ucontext = mana_ib_dealloc_ucontext,
+	.dereg_mr = mana_ib_dereg_mr,
+	.destroy_cq = mana_ib_destroy_cq,
+	.destroy_qp = mana_ib_destroy_qp,
+	.destroy_rwq_ind_table = mana_ib_destroy_rwq_ind_table,

drivers/infiniband/hw/mana/main.c line 591

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+	.destroy_wq = mana_ib_destroy_wq,
+	.disassociate_ucontext = mana_ib_disassociate_ucontext,
+	.get_port_immutable = mana_ib_get_port_immutable,
+	.mmap = mana_ib_mmap,
+	.modify_qp = mana_ib_modify_qp,
+	.modify_wq = mana_ib_modify_wq,
+	.query_device = mana_ib_query_device,
+	.query_gid = mana_ib_query_gid,
+	.query_port = mana_ib_query_port,
+	.reg_user_mr = mana_ib_reg_user_mr,

eg there is no way to create a kernel MR..

So, why do I see so many kverbs like things - and why are things like FMR in this driver that can never be used?

Jason

#10 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Jason Gunthorpe
Quoted 23 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 12 lines · Long Li
Quoted 11 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct · line 587

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
 mana_port_context *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

Quoted 6 lines

The reason netdev_info () is used is that this message is important for troubleshooting initial setup issues with Ethernet driver. We rely on user to get this configured right to share the same hardware port between Ethernet and RDMA driver. As far as I know, there is no easy way for a driver to "take over" an exclusive hardware resource from another driver.

Quoted 5 lines

This seems like a really strange statement.

Exactly how does all of this work?

Jason

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Hardware is not responsible for tracking which one is using this "vport", it's up to the software to make sure it's correctly configured for that device.

Long

#30 lore raw

RE: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

In reply to Jason Gunthorpe
Quoted 28 lines

Subject: Re: [Patch v4 06/12] net: mana: Define data structures for protection domain and memory registration

Quoted 4 lines · Dexuan Cui
Quoted 4 lines

From: [email protected] <[email protected]>
Sent: Wednesday, June 15, 2022 7:07 PM

The MANA hardware support protection domain and memory

registration

Quoted 18 lines
Quoted 1 line

for

s/support/supports

drivers/net/ethernet/microsoft/mana/gdma.h enum gdma_request_type { · line 33

Quoted 13 lines
drivers/net/ethernet/microsoft/mana/gdma.h +4 -0
index f945755760dc..b1bec8ab5695 100644
@@ -27,6 +27,10 @@ enum gdma_request_type {
 	GDMA_CREATE_DMA_REGION		= 25,
 	GDMA_DMA_REGION_ADD_PAGES	= 26,
 	GDMA_DESTROY_DMA_REGION		= 27,
+	GDMA_CREATE_PD			= 29,
+	GDMA_DESTROY_PD			= 30,
+	GDMA_CREATE_MR			= 31,
+	GDMA_DESTROY_MR			= 32,

These are not used in this patch. They're used in the 12th patch for the first time. Can we move these to that patch?

Quoted 3 lines

This looks like RDMA code anyhow, why is it under net/ethernet?

Jason

This header file belongs to the GDMA layer (as its filename implies) . It's a hardware communication layer used by both ethernet and RDMA for communicating with the hardware.

Some of the RDMA functionalities are implemented at GDMA layer in the PF running on the host, so the message definitions are also there.

Long

#54 lore raw

Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to [email protected]
Quoted 2 lines · On 6/16/22 10:07 AM, [email protected] wrote:

From: Long Li <[email protected]>

<...>

Quoted 78 lines
drivers/infiniband/hw/mana/qp.c new file +78 -0
+
+static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
+				 struct ib_qp_init_attr *attr,
+				 struct ib_udata *udata)
+{
+	struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
+	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
+	struct mana_ib_dev *mdev =
+		container_of(ibpd->device, struct mana_ib_dev, ib_dev);
+	struct mana_ib_cq *send_cq =
+		container_of(attr->send_cq, struct mana_ib_cq, ibcq);
+	struct ib_ucontext *ib_ucontext = ibpd->uobject->context;
+	struct mana_ib_create_qp_resp resp = {};
+	struct mana_ib_ucontext *mana_ucontext;
+	struct gdma_dev *gd = mdev->gdma_dev;
+	struct mana_ib_create_qp ucmd = {};
+	struct mana_obj_spec wq_spec = {};
+	struct mana_obj_spec cq_spec = {};
+	struct mana_port_context *mpc;
+	struct mana_context *mc;
+	struct net_device *ndev;
+	struct ib_umem *umem;
+	int err;
+	u32 port;
+
+	mana_ucontext =
+		container_of(ib_ucontext, struct mana_ib_ucontext, ibucontext);
+	mc = gd->driver_data;
+
+	if (udata->inlen < sizeof(ucmd))
+		return -EINVAL;
+
+	err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata->inlen));
+	if (err) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Failed to copy from udata create qp-raw, %d\n", err);
+		return -EFAULT;
+	}
+
+	/* IB ports start with 1, MANA Ethernet ports start with 0 */
+	port = ucmd.port;
+	if (ucmd.port > mc->num_ports)
+		return -EINVAL;
+
+	if (attr->cap.max_send_wr > MAX_SEND_BUFFERS_PER_QUEUE) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Requested max_send_wr %d exceeding limit\n",
+			  attr->cap.max_send_wr);
+		return -EINVAL;
+	}
+
+	if (attr->cap.max_send_sge > MAX_TX_WQE_SGL_ENTRIES) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Requested max_send_sge %d exceeding limit\n",
+			  attr->cap.max_send_sge);
+		return -EINVAL;
+	}
+
+	ndev = mc->ports[port - 1];
+	mpc = netdev_priv(ndev);
+	ibdev_dbg(&mdev->ib_dev, "port %u ndev %p mpc %p\n", port, ndev, mpc);
+
+	err = mana_ib_cfg_vport(mdev, port - 1, pd, mana_ucontext->doorbell);
+	if (err)
+		return -ENODEV;
+
+	qp->port = port;
+
+	ibdev_dbg(&mdev->ib_dev, "ucmd sq_buf_addr 0x%llx port %u\n",
+		  ucmd.sq_buf_addr, ucmd.port);
+
+	umem = ib_umem_get(ibpd->device, ucmd.sq_buf_addr, ucmd.sq_buf_size,
+			   IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(umem)) {
+		err = PTR_ERR(umem);
+		ibdev_dbg(&mdev->ib_dev,
+			  "Failed to get umem for create qp-raw, err %d\n",
+			  err);

drivers/infiniband/hw/mana/qp.c line 331

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +10 -0
+		goto err_free_vport;
+	}
+	qp->sq_umem = umem;
+
+	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
+					   &qp->sq_gdma_region, PAGE_SIZE);
+	if (err) {
+		ibdev_err(&mdev->ib_dev,
+			  "Failed to create dma region for create qp-raw, %d\n",
+			  err);

It is better not print in userspace-triggered paths.

There are also same issues in other paths.

Thanks,
Cheng Xu

#11 lore raw

Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Long Li
Quoted 27 lines · Long Li
Quoted 27 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 12 lines · Long Li
Quoted 11 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct · line 587

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
 mana_port_context *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u DB %u\n",
+		    apc->port_handle, protection_dom_id, doorbell_pg_id);

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

Quoted 6 lines

The reason netdev_info () is used is that this message is important for troubleshooting initial setup issues with Ethernet driver. We rely on user to get this configured right to share the same hardware port between Ethernet and RDMA driver. As far as I know, there is no easy way for a driver to "take over" an exclusive hardware resource from another driver.

This seems like a really strange statement.

Exactly how does all of this work?

Quoted 7 lines
Quoted 1 line

Jason

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Sure, but how is the users problem to "get this configured right" and what exactly is the user supposed to do?

I would expect the allocation of HW resources to be completely transparent to the user. Why is it not?

Jason

#55 lore raw

RE: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Cheng Xu
Quoted 107 lines

Subject: Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

Quoted 2 lines · On 6/16/22 10:07 AM, [email protected] wrote:

From: Long Li <[email protected]>

<...>

drivers/infiniband/hw/mana/qp.c

Quoted 6 lines
drivers/infiniband/hw/mana/qp.c new file +6 -0
+
+static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
+				 struct ib_qp_init_attr *attr,
+				 struct ib_udata *udata)
+{
+	struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd,

ibpd);

Review

Quoted 1 line
(quoted diff) +1 -0
+	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp,

ibqp);

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +10 -0
+	struct mana_ib_dev *mdev =
+		container_of(ibpd->device, struct mana_ib_dev, ib_dev);
+	struct mana_ib_cq *send_cq =
+		container_of(attr->send_cq, struct mana_ib_cq, ibcq);
+	struct ib_ucontext *ib_ucontext = ibpd->uobject->context;
+	struct mana_ib_create_qp_resp resp = {};
+	struct mana_ib_ucontext *mana_ucontext;
+	struct gdma_dev *gd = mdev->gdma_dev;
+	struct mana_ib_create_qp ucmd = {};
+	struct mana_obj_spec wq_spec = {};

drivers/infiniband/hw/mana/qp.c

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +10 -0
+	struct mana_obj_spec cq_spec = {};
+	struct mana_port_context *mpc;
+	struct mana_context *mc;
+	struct net_device *ndev;
+	struct ib_umem *umem;
+	int err;
+	u32 port;
+
+	mana_ucontext =
+		container_of(ib_ucontext, struct mana_ib_ucontext,

ibucontext);

drivers/infiniband/hw/mana/qp.c

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +6 -0
+	mc = gd->driver_data;
+
+	if (udata->inlen < sizeof(ucmd))
+		return -EINVAL;
+
+	err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata-

inlen));

drivers/infiniband/hw/mana/qp.c new file +3 -0
+	if (err) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Failed to copy from udata create qp-raw, %d\n",

err);

Quoted 15 lines
drivers/infiniband/hw/mana/qp.c new file +15 -0
+		return -EFAULT;
+	}
+
+	/* IB ports start with 1, MANA Ethernet ports start with 0 */
+	port = ucmd.port;
+	if (ucmd.port > mc->num_ports)
+		return -EINVAL;
+
+	if (attr->cap.max_send_wr > MAX_SEND_BUFFERS_PER_QUEUE) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Requested max_send_wr %d exceeding limit\n",
+			  attr->cap.max_send_wr);
+		return -EINVAL;
+	}
+

drivers/infiniband/hw/mana/qp.c

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +10 -0
+	if (attr->cap.max_send_sge > MAX_TX_WQE_SGL_ENTRIES) {
+		ibdev_dbg(&mdev->ib_dev,
+			  "Requested max_send_sge %d exceeding limit\n",
+			  attr->cap.max_send_sge);
+		return -EINVAL;
+	}
+
+	ndev = mc->ports[port - 1];
+	mpc = netdev_priv(ndev);
+	ibdev_dbg(&mdev->ib_dev, "port %u ndev %p mpc %p\n", port,

ndev, mpc);

drivers/infiniband/hw/mana/qp.c

Quoted 12 lines
(quoted diff) +2 -0
+
+	err = mana_ib_cfg_vport(mdev, port - 1, pd, mana_ucontext-

doorbell);

drivers/infiniband/hw/mana/qp.c new file +9 -0
+	if (err)
+		return -ENODEV;
+
+	qp->port = port;
+
+	ibdev_dbg(&mdev->ib_dev, "ucmd sq_buf_addr 0x%llx port %u\n",
+		  ucmd.sq_buf_addr, ucmd.port);
+
+	umem = ib_umem_get(ibpd->device, ucmd.sq_buf_addr,

ucmd.sq_buf_size,

Quoted 5 lines
drivers/infiniband/hw/mana/qp.c new file +5 -0
+			   IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(umem)) {
+		err = PTR_ERR(umem);
+		ibdev_dbg(&mdev->ib_dev,
+			  "Failed to get umem for create qp-raw, err %d\n",

drivers/infiniband/hw/mana/qp.c

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c new file +10 -0
+			  err);
+		goto err_free_vport;
+	}
+	qp->sq_umem = umem;
+
+	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
+					   &qp->sq_gdma_region, PAGE_SIZE);
+	if (err) {
+		ibdev_err(&mdev->ib_dev,
+			  "Failed to create dma region for create qp-

raw, %d\n",

Quoted 8 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+			  err);

It is better not print in userspace-triggered paths.

There are also same issues in other paths.

Thanks,
Cheng Xu

Thank you, I will scan the code and make sure user-mode can't flood error messages.

This error is a hardware/PF error. It means the hardware channel has faulted. It's logged at error level.

Long

#12 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Jason Gunthorpe
Quoted 41 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 11 lines · Long Li
Quoted 11 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 7 lines · Long Li
Quoted 7 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_cfg_vport(struct · line 586

Quoted 7 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
@@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
 mana_port_context *apc, u32 protection_dom_id,
 
 	apc->tx_shortform_allowed = resp.short_form_allowed;
 	apc->tx_vp_offset = resp.tx_vport_offset;
+
+	netdev_info(apc->ndev, "Configured vPort %llu PD %u

DB %u\n",

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Review static int mana_cfg_vport(struct · line 587

Quoted 1 line
(quoted diff) +1 -0
+		    apc->port_handle, protection_dom_id,

doorbell_pg_id);

Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 3 lines

Should this be netdev_dbg()?
The log buffer can be flooded if there are many vPorts per VF PCI device and there are a lot of VFs.

The reason netdev_info () is used is that this message is important for troubleshooting initial setup issues with Ethernet driver. We rely on user to get this configured right to share the same hardware port between Ethernet and RDMA driver. As far as I know, there is no easy way for a driver to "take over" an exclusive hardware resource from another driver.

This seems like a really strange statement.

Exactly how does all of this work?

Quoted 7 lines
Quoted 1 line

Jason

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Quoted 7 lines

Sure, but how is the users problem to "get this configured right" and what exactly is the user supposed to do?

I would expect the allocation of HW resources to be completely transparent to the user. Why is it not?

Jason

In the hardware, RDMA RAW_QP shares the same hardware resource (in this case, the vPort in hardware table) with the ethernet NIC. When an RDMA user creates a RAW_QP, we can't just shut down the ethernet. The user is required to make sure the ethernet is not in used when he creates this QP type.

#13 lore raw

Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 13 lines · Long Li
Quoted 6 lines
Quoted 5 lines

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Quoted 6 lines

Sure, but how is the users problem to "get this configured right" and what exactly is the user supposed to do?

I would expect the allocation of HW resources to be completely transparent to the user. Why is it not?

Quoted 5 lines

In the hardware, RDMA RAW_QP shares the same hardware resource (in this case, the vPort in hardware table) with the ethernet NIC. When an RDMA user creates a RAW_QP, we can't just shut down the ethernet. The user is required to make sure the ethernet is not in used when he creates this QP type.

You haven't answered my question - how is the user supposed to achieve this?

And now I also want to know why the ethernet device and rdma device can even be loaded together if they cannot share the physical port? Exclusivity is not a sharing model that any driver today implements.

Jason

#53 lore raw

RE: [EXTERNAL] Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

In reply to Jason Gunthorpe
Quoted 68 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, July 20, 2022 6:49 PM
To: Long Li <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Leon Romanovsky <[email protected]>; [email protected]; [email protected]; Ajay Sharma <[email protected]>; linux- [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: [EXTERNAL] Re: [Patch v4 12/12] RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter

drivers/infiniband/hw/mana/main.c

Quoted 13 lines · On Wed, Jun 15, 2022 at 07:07:20PM -0700, [email protected] wrote:
drivers/infiniband/hw/mana/main.c new file +13 -0
+static int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata
+*udata) {
+	struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
+	struct ib_device *ibdev = ibpd->device;
+	enum gdma_pd_flags flags = 0;
+	struct mana_ib_dev *dev;
+	int ret;
+
+	dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
+
+	/* Set flags if this is a kernel request */
+	if (!ibpd->uobject)
+		flags = GDMA_PD_FLAG_ALLOW_GPA_MR |

GDMA_PD_FLAG_ALLOW_FMR_MR;

I'm confused, this driver doesn't seem to support kverbs:

Quoted 17 lines
drivers/infiniband/hw/mana/main.c new file +17 -0
+static const struct ib_device_ops mana_ib_dev_ops = {
+	.owner = THIS_MODULE,
+	.driver_id = RDMA_DRIVER_MANA,
+	.uverbs_abi_ver = MANA_IB_UVERBS_ABI_VERSION,
+
+	.alloc_pd = mana_ib_alloc_pd,
+	.alloc_ucontext = mana_ib_alloc_ucontext,
+	.create_cq = mana_ib_create_cq,
+	.create_qp = mana_ib_create_qp,
+	.create_rwq_ind_table = mana_ib_create_rwq_ind_table,
+	.create_wq = mana_ib_create_wq,
+	.dealloc_pd = mana_ib_dealloc_pd,
+	.dealloc_ucontext = mana_ib_dealloc_ucontext,
+	.dereg_mr = mana_ib_dereg_mr,
+	.destroy_cq = mana_ib_destroy_cq,
+	.destroy_qp = mana_ib_destroy_qp,
+	.destroy_rwq_ind_table = mana_ib_destroy_rwq_ind_table,

drivers/infiniband/hw/mana/main.c line 591

Quoted 10 lines
drivers/infiniband/hw/mana/main.c new file +10 -0
+	.destroy_wq = mana_ib_destroy_wq,
+	.disassociate_ucontext = mana_ib_disassociate_ucontext,
+	.get_port_immutable = mana_ib_get_port_immutable,
+	.mmap = mana_ib_mmap,
+	.modify_qp = mana_ib_modify_qp,
+	.modify_wq = mana_ib_modify_wq,
+	.query_device = mana_ib_query_device,
+	.query_gid = mana_ib_query_gid,
+	.query_port = mana_ib_query_port,
+	.reg_user_mr = mana_ib_reg_user_mr,

eg there is no way to create a kernel MR..

Quoted 4 lines

So, why do I see so many kverbs like things - and why are things like FMR in this driver that can never be used?

Jason

The idea was to introduce kernel support in future. I will remove it from the code and upload the patch.

#14 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

In reply to Jason Gunthorpe
Quoted 16 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 13 lines · Long Li
Quoted 12 lines
Quoted 5 lines

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Sure, but how is the users problem to "get this configured right" and what exactly is the user supposed to do?

I would expect the allocation of HW resources to be completely transparent to the user. Why is it not?

Quoted 7 lines
Quoted 5 lines

In the hardware, RDMA RAW_QP shares the same hardware resource (in this case, the vPort in hardware table) with the ethernet NIC. When an RDMA user creates a RAW_QP, we can't just shut down the ethernet. The user is required to make sure the ethernet is not in used when he creates this QP type.

You haven't answered my question - how is the user supposed to achieve this?

The user needs to configure the network interface so the kernel will not use it when the user creates a RAW QP on this port.

This can be done via system configuration to not bring this interface online on system boot, or equivalently doing "ifconfig xxx down" to make the interface down when creating a RAW QP on this port.

Quoted 5 lines

And now I also want to know why the ethernet device and rdma device can even be loaded together if they cannot share the physical port? Exclusivity is not a sharing model that any driver today implements.

This physical port limitation only applies to the RAW QP. For RC QP, the hardware doesn't have this limitation. The user can create RC QPs on a physical port up to the hardware limits independent of the Ethernet usage on the same port.

For Ethernet usage, the hardware supports only one active user on a physical port. The driver checks on the port usage before programming the hardware when creating the RAW QP. Because the RDMA driver doesn't know in advance which QP type the user will create, it exposes the device with all its ports. The user may not be able to create RAW QP on a port if this port is already in used by the kernel.

As a comparison, Mellanox NICs can expose both Ethernet and RDMA RAW_QP on the same physical port to software. They can work at the same time, but with some "quirks". The RDMA RAW_QP can preempt/interfere Ethernet traffic under certain conditions commonly used by DPDK (a heavy user of RAW_QP).

Here are two scenarios that a Mellanox NIC port works on both Ethernet and RAW_QP.

Scenario 1: The Ethernet loses TCP connection.
1. User A runs a program listing on a TCP port, accepts an incoming TCP connection and is communicating with the remote peer over this TCP connection.
2. User B creates an RDMA RAW_QP on the same port on the device.
3. As soon as the RAW_QP is created, the program in 1 can't send/receive data over this TCP connection. After some period of inactivity, the TCP connection terminates.

Please note that this may also pose a security risk. User B with RAW_QP can potentially hijack this TCP connection from the kernel by framing the correct Ethernet packets and send over this QP to trick the remote peer, making it believe it's User A.

Scenario 2: The Ethernet port state changes after RDMA RAW_QP is used on the port.
1. User uses "ifconfig ethx down" on the NIC, intending to make it offline
2. User creates a RDMA RAW_QP on the same port on the device.
3. User destroys this RAW_QP.
4. The ethx device in 1 reports carrier state in step 2, in many Linux distributions this makes it online without user interaction. "ifconfig ethx" shows its state changes to "up".

The two activities on Ethernet and on RDMA RAW_QP should not happen concurrently and the user either gets unexpected behavior (Scenario 1) or the user needs to explicitly serialize the use (Scenario 2). In this sense, I think MANA is not materially different to how the Mellanox NICs implement the RAW_QP. IMHO, it's better to have the user explicitly decide whether to use Ethernet or RDMA RAW_QP on a specific port.

Long

#15 lore raw

Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 22 lines · Long Li
Quoted 22 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 18 lines · Long Li
Quoted 6 lines
Quoted 5 lines

"vport" is a hardware resource that can either be used by an Ethernet device, or an RDMA device. But it can't be used by both at the same time. The "vport" is associated with a protection domain and doorbell, it's programmed in the hardware. Outgoing traffic is enforced on this vport based on how it is programmed.

Quoted 6 lines

Sure, but how is the users problem to "get this configured right" and what exactly is the user supposed to do?

I would expect the allocation of HW resources to be completely transparent to the user. Why is it not?

In the hardware, RDMA RAW_QP shares the same hardware resource (in this case, the vPort in hardware table) with the ethernet NIC. When an RDMA user creates a RAW_QP, we can't just shut down the ethernet. The user is required to make sure the ethernet is not in used when he creates this QP type.

Quoted 8 lines
Quoted 1 line

You haven't answered my question - how is the user supposed to achieve this?

The user needs to configure the network interface so the kernel will not use it when the user creates a RAW QP on this port.

This can be done via system configuration to not bring this interface online on system boot, or equivalently doing "ifconfig xxx down" to make the interface down when creating a RAW QP on this port.

That sounds horrible, why allow the user to even bind two drivers if the two drivers can't be used together?

Quoted 8 lines
Quoted 3 lines

And now I also want to know why the ethernet device and rdma device can even be loaded together if they cannot share the physical port? Exclusivity is not a sharing model that any driver today implements.

This physical port limitation only applies to the RAW QP. For RC QP, the hardware doesn't have this limitation. The user can create RC QPs on a physical port up to the hardware limits independent of the Ethernet usage on the same port.

.. and it is because you support sharing models in other cases :\

Quoted 9 lines

Scenario 1: The Ethernet loses TCP connection.

1. User A runs a program listing on a TCP port, accepts an incoming TCP connection and is communicating with the remote peer over this TCP connection.
2. User B creates an RDMA RAW_QP on the same port on the device.
3. As soon as the RAW_QP is created, the program in 1 can't send/receive data over this TCP connection. After some period of inactivity, the TCP connection terminates.

It is a little more complicated than that, but yes, that could possibly happen if the userspace captures the right traffic.

Quoted 4 lines

Please note that this may also pose a security risk. User B with RAW_QP can potentially hijack this TCP connection from the kernel by framing the correct Ethernet packets and send over this QP to trick the remote peer, making it believe it's User A.

Any root user can do this with the netstack using eg tcpdump, bpf, XDP, raw sockets, etc. This is why the capability is guarded by CAP_NET_RAW. It is nothing unusual.

Quoted 7 lines

Scenario 2: The Ethernet port state changes after RDMA RAW_QP is used on the port.
1. User uses "ifconfig ethx down" on the NIC, intending to make it offline
2. User creates a RDMA RAW_QP on the same port on the device.
3. User destroys this RAW_QP.
4. The ethx device in 1 reports carrier state in step 2, in many Linux distributions this makes it online without user
interaction. "ifconfig ethx" shows its state changes to "up".

This I'm not familiar with, it actually sounds like a bug that the RAW_QP's interfere with the netdev carrier state.

Quoted 3 lines

the Mellanox NICs implement the RAW_QP. IMHO, it's better to have the user explicitly decide whether to use Ethernet or RDMA RAW_QP on a specific port.

It should all be carefully documented someplace.

Jason

#16 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 5 lines
Quoted 3 lines

the Mellanox NICs implement the RAW_QP. IMHO, it's better to have the user explicitly decide whether to use Ethernet or RDMA RAW_QP on a specific port.

It should all be carefully documented someplace.

The use case for RAW_QP is from user-mode. Is it acceptable that we document the detailed usage in rdma-core?

Long

#17 lore raw

Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 8 lines · Long Li
Quoted 5 lines
Quoted 3 lines

the Mellanox NICs implement the RAW_QP. IMHO, it's better to have the user explicitly decide whether to use Ethernet or RDMA RAW_QP on a specific port.

It should all be carefully documented someplace.

The use case for RAW_QP is from user-mode. Is it acceptable that we document the detailed usage in rdma-core?

Yes, but add a suitable comment someplace in the kernel too

Jason

#18 lore raw

RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 7 lines

Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between devices

Quoted 4 lines · Long Li
Quoted 4 lines
Quoted 3 lines

the Mellanox NICs implement the RAW_QP. IMHO, it's better to have the user explicitly decide whether to use Ethernet or RDMA RAW_QP on a specific port.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

It should all be carefully documented someplace.

The use case for RAW_QP is from user-mode. Is it acceptable that we document the detailed usage in rdma-core?

Yes, but add a suitable comment someplace in the kernel too

Thanks. I will add detailed comments.

Long