Skip to contentSkip to conversation

verbs: Query GID table API

PATCH In review Yishai Hadas 1 participant 10 messages Sep 23, 2020 – Oct 11, 2020 InfiniBand

lore
Share & tools
lore
#0 lore raw

cover verbs: Query GID table API

When an application is not using RDMA CM and if it is using multiple RDMA devices with one or more RoCE ports, finding the right GID table entry is a long process.

For example, with two RoCE dual-port devices in a system, when IP failover is used between two RoCE ports, searching a suitable GID entry for a given source IP, matching netdevice of given RoCEv1/v2 type requires iterating over all 4 ports * 256 entry GID table.

Even though the best first match GID table for given criteria is used, when the matching entry is on the 4th port, it requires reading 3 ports * 256 entries * 3 files (GID, netdev, type) = 2304 files. The GID table needs to be referred on every QP creation during IP failover on other netdevice of an RDMA device.

We introduce this series of patches, which introduces an API to query the complete GID tables of an RDMA device, that returns all valid GID table entries.

This is done through single ioctl, eliminating 2304 read, 2304 open and 2304 close system calls to just a total of 2 calls (one for each device).

While at it, we also introduce an API to query an individual GID entry over ioctl interface, which provides all GID attributes information.

The APIs are based on the below RFC [1], the matching kernel part was sent to rdma-next.

PR was sent as well [2].

[1] https://www.spinics.net/lists/linux-rdma/msg91825.html
[2] https://github.com/linux-rdma/rdma-core/pull/828

Changelog:
v1:
* Move some GIDs stuff to cmd_device.c with better function's names.
* Cleanup in few places.

Avihai Horon (7):
verbs: Change the name of enum ibv_gid_type
verbs: Introduce a new query GID entry API
verbs: Implement ibv_query_gid and ibv_query_gid_type over ioctl
verbs: Optimize ibv_query_gid and ibv_query_gid_type
verbs: Introduce a new query GID table API
pyverbs: Add query_gid_table and query_gid_ex methods
tests: Add tests for ibv_query_gid_table and ibv_query_gid_ex

Yishai Hadas (1):
Update kernel headers

 debian/libibverbs1.symbols                |   3 +
 kernel-headers/rdma/ib_user_ioctl_cmds.h  |  16 ++
 kernel-headers/rdma/ib_user_ioctl_verbs.h |  14 ++
 kernel-headers/rdma/ib_user_verbs.h       |  11 ++
 kernel-headers/rdma/rdma_user_rxe.h       |   6 +-
 libibverbs/CMakeLists.txt                 |   2 +-
 libibverbs/cmd_device.c                   | 314 ++++++++++++++++++++++++++++++
 libibverbs/driver.h                       |  19 +-
 libibverbs/examples/devinfo.c             |  14 +-
 libibverbs/libibverbs.map.in              |   6 +
 libibverbs/man/CMakeLists.txt             |   2 +
 libibverbs/man/ibv_query_gid_ex.3.md      |  93 +++++++++
 libibverbs/man/ibv_query_gid_table.3.md   |  73 +++++++
 libibverbs/verbs.c                        | 105 ++++------
 libibverbs/verbs.h                        |  45 +++++
 providers/mlx5/verbs.c                    |   2 +-
 pyverbs/device.pxd                        |   3 +
 pyverbs/device.pyx                        | 108 +++++++++-
 pyverbs/libibverbs.pxd                    |  15 +-
 pyverbs/libibverbs_enums.pxd              |  11 +-
 tests/base.py                             |   3 +-
 tests/test_device.py                      |  32 +++
 22 files changed, 804 insertions(+), 93 deletions(-)
 create mode 100644 libibverbs/man/ibv_query_gid_ex.3.md
 create mode 100644 libibverbs/man/ibv_query_gid_table.3.md
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/8 Update kernel headers

To commit 320a6a2fef0b ("RDMA/uverbs: Expose the new GID query API to user space")

---
 kernel-headers/rdma/ib_user_ioctl_cmds.h  | 16 ++++++++++++++++
 kernel-headers/rdma/ib_user_ioctl_verbs.h | 14 ++++++++++++++
 kernel-headers/rdma/ib_user_verbs.h       | 11 +++++++++++
 kernel-headers/rdma/rdma_user_rxe.h       |  6 +++---
 4 files changed, 44 insertions(+), 3 deletions(-)
kernel-headers/rdma/ib_user_ioctl_cmds.h +16 -0
index 99dcabf..7968a18 100644
@@ -70,6 +70,8 @@ enum uverbs_methods_device {
 	UVERBS_METHOD_QUERY_PORT,
 	UVERBS_METHOD_GET_CONTEXT,
 	UVERBS_METHOD_QUERY_CONTEXT,
+	UVERBS_METHOD_QUERY_GID_TABLE,
+	UVERBS_METHOD_QUERY_GID_ENTRY,
 };
 
 enum uverbs_attrs_invoke_write_cmd_attr_ids {
@@ -352,4 +354,18 @@ enum uverbs_attrs_async_event_create {
 	UVERBS_ATTR_ASYNC_EVENT_ALLOC_FD_HANDLE,
 };
 
+enum uverbs_attrs_query_gid_table_cmd_attr_ids {
+	UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE,
+	UVERBS_ATTR_QUERY_GID_TABLE_FLAGS,
+	UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES,
+	UVERBS_ATTR_QUERY_GID_TABLE_RESP_NUM_ENTRIES,
+};
+
+enum uverbs_attrs_query_gid_entry_cmd_attr_ids {
+	UVERBS_ATTR_QUERY_GID_ENTRY_PORT,
+	UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX,
+	UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS,
+	UVERBS_ATTR_QUERY_GID_ENTRY_RESP_ENTRY,
+};
+
 #endif
kernel-headers/rdma/ib_user_ioctl_verbs.h +14 -0
index 5debab4..cfea82a 100644
@@ -250,4 +250,18 @@ enum rdma_driver_id {
 	RDMA_DRIVER_SIW,
 };
 
+enum ib_uverbs_gid_type {
+	IB_UVERBS_GID_TYPE_IB,
+	IB_UVERBS_GID_TYPE_ROCE_V1,
+	IB_UVERBS_GID_TYPE_ROCE_V2,
+};
+
+struct ib_uverbs_gid_entry {
+	__aligned_u64 gid[2];
+	__u32 gid_index;
+	__u32 port_num;
+	__u32 gid_type;
+	__u32 netdev_ifindex; /* It is 0 if there is no netdev associated with it */
+};
+
 #endif
kernel-headers/rdma/ib_user_verbs.h +11 -0
index 0474c74..456438c 100644
@@ -457,6 +457,17 @@ struct ib_uverbs_poll_cq {
 	__u32 ne;
 };
 
+enum ib_uverbs_wc_opcode {
+	IB_UVERBS_WC_SEND = 0,
+	IB_UVERBS_WC_RDMA_WRITE = 1,
+	IB_UVERBS_WC_RDMA_READ = 2,
+	IB_UVERBS_WC_COMP_SWAP = 3,
+	IB_UVERBS_WC_FETCH_ADD = 4,
+	IB_UVERBS_WC_BIND_MW = 5,
+	IB_UVERBS_WC_LOCAL_INV = 6,
+	IB_UVERBS_WC_TSO = 7,
+};
+
 struct ib_uverbs_wc {
 	__aligned_u64 wr_id;
 	__u32 status;
kernel-headers/rdma/rdma_user_rxe.h +3 -3
index aae2e69..d8f2e0e 100644
@@ -99,8 +99,8 @@ struct rxe_send_wr {
 				struct ib_mr *mr;
 				__aligned_u64 reserved;
 			};
-			__u32        key;
-			__u32        access;
+			__u32	     key;
+			__u32	     access;
 		} reg;
 	} wr;
 };
@@ -112,7 +112,7 @@ struct rxe_sge {
 };
 
 struct mminfo {
-	__aligned_u64  		offset;
+	__aligned_u64		offset;
 	__u32			size;
 	__u32			pad;
 };
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 2/8 verbs: Change the name of enum ibv_gid_type

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Change the name of enum ibv_gid_type in order to introduce a new enum ibv_gid_type in verbs.h in the next commits, which will provide a more accurate gid type info by separating IB and RoCEv1 types.

This is a preliminary step before introducing a new query GID API.

---
 libibverbs/driver.h           |  8 ++++----
 libibverbs/examples/devinfo.c | 14 +++++++-------
 libibverbs/verbs.c            | 21 +++++++++++----------
 providers/mlx5/verbs.c        |  2 +-
 pyverbs/device.pyx            |  2 +-
 pyverbs/libibverbs.pxd        |  2 +-
 pyverbs/libibverbs_enums.pxd  |  6 +++---
 tests/base.py                 |  3 ++-
 8 files changed, 30 insertions(+), 28 deletions(-)
libibverbs/driver.h +4 -4
index 4436363..046c07d 100644
@@ -72,9 +72,9 @@ enum verbs_qp_mask {
 	VERBS_QP_EX		= 1 << 1,
 };
 
-enum ibv_gid_type {
-	IBV_GID_TYPE_IB_ROCE_V1,
-	IBV_GID_TYPE_ROCE_V2,
+enum ibv_gid_type_sysfs {
+	IBV_GID_TYPE_SYSFS_IB_ROCE_V1,
+	IBV_GID_TYPE_SYSFS_ROCE_V2,
 };
 
 enum ibv_mr_type {
@@ -653,7 +653,7 @@ static inline bool check_comp_mask(uint64_t input, uint64_t supported)
 }
 
 int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
-		       unsigned int index, enum ibv_gid_type *type);
+		       unsigned int index, enum ibv_gid_type_sysfs *type);
 
 static inline int
 ibv_check_alloc_parent_domain(struct ibv_parent_domain_init_attr *attr)
libibverbs/examples/devinfo.c +7 -7
index 00ed3f9..c245217 100644
@@ -164,17 +164,17 @@ static const char *vl_str(uint8_t vl_num)
 }
 
 #define DEVINFO_INVALID_GID_TYPE	2
-static const char *gid_type_str(enum ibv_gid_type type)
+static const char *gid_type_str(enum ibv_gid_type_sysfs type)
 {
 	switch (type) {
-	case IBV_GID_TYPE_IB_ROCE_V1: return "RoCE v1";
-	case IBV_GID_TYPE_ROCE_V2: return "RoCE v2";
+	case IBV_GID_TYPE_SYSFS_IB_ROCE_V1: return "RoCE v1";
+	case IBV_GID_TYPE_SYSFS_ROCE_V2: return "RoCE v2";
 	default: return "Invalid gid type";
 	}
 }
 
 static void print_formated_gid(union ibv_gid *gid, int i,
-			       enum ibv_gid_type type, int ll)
+			       enum ibv_gid_type_sysfs type, int ll)
 {
 	char gid_str[INET6_ADDRSTRLEN] = {};
 	char str[20] = {};
@@ -182,7 +182,7 @@ static void print_formated_gid(union ibv_gid *gid, int i,
 	if (ll == IBV_LINK_LAYER_ETHERNET)
 		sprintf(str, ", %s", gid_type_str(type));
 
-	if (type == IBV_GID_TYPE_IB_ROCE_V1)
+	if (type == IBV_GID_TYPE_SYSFS_IB_ROCE_V1)
 		printf("\t\t\tGID[%3d]:\t\t%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x%s\n",
 		       i, gid->raw[0], gid->raw[1], gid->raw[2],
 		       gid->raw[3], gid->raw[4], gid->raw[5], gid->raw[6],
@@ -190,7 +190,7 @@ static void print_formated_gid(union ibv_gid *gid, int i,
 		       gid->raw[11], gid->raw[12], gid->raw[13], gid->raw[14],
 		       gid->raw[15], str);
 
-	if (type == IBV_GID_TYPE_ROCE_V2) {
+	if (type == IBV_GID_TYPE_SYSFS_ROCE_V2) {
 		inet_ntop(AF_INET6, gid->raw, gid_str, sizeof(gid_str));
 		printf("\t\t\tGID[%3d]:\t\t%s%s\n", i, gid_str, str);
 	}
@@ -200,7 +200,7 @@ static int print_all_port_gids(struct ibv_context *ctx,
 			       struct ibv_port_attr *port_attr,
 			       uint8_t port_num)
 {
-	enum ibv_gid_type type;
+	enum ibv_gid_type_sysfs type;
 	union ibv_gid gid;
 	int tbl_len;
 	int rc = 0;
libibverbs/verbs.c +11 -10
index b54e2b8..9507ffd 100644
@@ -704,7 +704,7 @@ LATEST_SYMVER_FUNC(ibv_create_ah, 1_1, "IBVERBS_1.1",
 #define V1_TYPE "IB/RoCE v1"
 #define V2_TYPE "RoCE v2"
 int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
-		       unsigned int index, enum ibv_gid_type *type)
+		       unsigned int index, enum ibv_gid_type_sysfs *type)
 {
 	struct verbs_device *verbs_device = verbs_get_device(context->device);
 	char buff[11];
@@ -723,7 +723,7 @@ int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
 			/* In IB, this file doesn't exist and the kernel sets
 			 * errno to -EINVAL.
 			 */
-			*type = IBV_GID_TYPE_IB_ROCE_V1;
+			*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
 			return 0;
 		}
 		if (asprintf(&dir_path, "%s/%s/%d/%s/",
@@ -738,7 +738,7 @@ int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
 				 * we have an old kernel and all GIDs are
 				 * IB/RoCE v1
 				 */
-				*type = IBV_GID_TYPE_IB_ROCE_V1;
+				*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
 			else
 				return -1;
 		} else {
@@ -748,9 +748,9 @@ int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
 		}
 	} else {
 		if (!strcmp(buff, V1_TYPE)) {
-			*type = IBV_GID_TYPE_IB_ROCE_V1;
+			*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
 		} else if (!strcmp(buff, V2_TYPE)) {
-			*type = IBV_GID_TYPE_ROCE_V2;
+			*type = IBV_GID_TYPE_SYSFS_ROCE_V2;
 		} else {
 			errno = ENOTSUP;
 			return -1;
@@ -761,9 +761,10 @@ int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
 }
 
 static int ibv_find_gid_index(struct ibv_context *context, uint8_t port_num,
-			      union ibv_gid *gid, enum ibv_gid_type gid_type)
+			      union ibv_gid *gid,
+			      enum ibv_gid_type_sysfs gid_type)
 {
-	enum ibv_gid_type sgid_type = 0;
+	enum ibv_gid_type_sysfs sgid_type = 0;
 	union ibv_gid sgid;
 	int i = 0, ret;
 
@@ -863,7 +864,7 @@ static inline int set_ah_attr_by_ipv4(struct ibv_context *context,
 
 	map_ipv4_addr_to_ipv6(ip4h->daddr, (struct in6_addr *)&sgid);
 	ret = ibv_find_gid_index(context, port_num, &sgid,
-				 IBV_GID_TYPE_ROCE_V2);
+				 IBV_GID_TYPE_SYSFS_ROCE_V2);
 	if (ret < 0)
 		return ret;
 
@@ -893,9 +894,9 @@ static inline int set_ah_attr_by_ipv6(struct ibv_context *context,
 
 	ah_attr->grh.dgid = grh->sgid;
 	if (grh->next_hdr == IPPROTO_UDP) {
-		sgid_type = IBV_GID_TYPE_ROCE_V2;
+		sgid_type = IBV_GID_TYPE_SYSFS_ROCE_V2;
 	} else if (grh->next_hdr == IB_NEXT_HDR) {
-		sgid_type = IBV_GID_TYPE_IB_ROCE_V1;
+		sgid_type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
 	} else {
 		errno = EPROTONOSUPPORT;
 		return -1;
providers/mlx5/verbs.c +1 -1
index 4650250..917d057 100644
@@ -2955,7 +2955,7 @@ struct ibv_ah *mlx5_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
 				       attr->grh.sgid_index, &gid_type))
 			goto err;
 
-		if (gid_type == IBV_GID_TYPE_ROCE_V2)
+		if (gid_type == IBV_GID_TYPE_SYSFS_ROCE_V2)
 			mlx5_ah_set_udp_sport(ah, attr);
 
 		/* Since RoCE packets must contain GRH, this bit is reserved
pyverbs/device.pyx +1 -1
index b75fcd0..c1323cd 100755
@@ -220,7 +220,7 @@ cdef class Context(PyverbsCM):
         return gid
 
     def query_gid_type(self, unsigned int port_num, unsigned int index):
-        cdef v.ibv_gid_type gid_type
+        cdef v.ibv_gid_type_sysfs gid_type
         rc = v.ibv_query_gid_type(self.context, port_num, index, &gid_type)
         if rc != 0:
             raise PyverbsRDMAErrno('Failed to query gid type of port {p} and gid index {g}'
pyverbs/libibverbs.pxd +1 -1
index dda33e7..c84b9fc 100755
@@ -617,6 +617,6 @@ cdef extern from 'infiniband/verbs.h':
 
 cdef extern from 'infiniband/driver.h':
     int ibv_query_gid_type(ibv_context *context, uint8_t port_num,
-                           unsigned int index, ibv_gid_type *type)
+                           unsigned int index, ibv_gid_type_sysfs *type)
     int ibv_set_ece(ibv_qp *qp, ibv_ece *ece)
     int ibv_query_ece(ibv_qp *qp, ibv_ece *ece)
pyverbs/libibverbs_enums.pxd +3 -3
index 7c1a120..83ca516 100755
@@ -443,6 +443,6 @@ _IBV_ADVISE_MR_FLAG_FLUSH = IBV_ADVISE_MR_FLAG_FLUSH
 
 
 cdef extern from '<infiniband/driver.h>':
-    cpdef enum ibv_gid_type:
-        IBV_GID_TYPE_IB_ROCE_V1
-        IBV_GID_TYPE_ROCE_V2
+    cpdef enum ibv_gid_type_sysfs:
+        IBV_GID_TYPE_SYSFS_IB_ROCE_V1
+        IBV_GID_TYPE_SYSFS_ROCE_V2
tests/base.py +2 -1
index b6c389d..0ebd728 100755
@@ -176,7 +176,8 @@ class RDMATestCase(unittest.TestCase):
                 continue
             # Avoid RoCEv2 GIDs on unsupported devices
             if port_attrs.link_layer == e.IBV_LINK_LAYER_ETHERNET and \
-                    ctx.query_gid_type(port, idx) == e.IBV_GID_TYPE_ROCE_V2 and \
+                    ctx.query_gid_type(port, idx) == \
+                    e.IBV_GID_TYPE_SYSFS_ROCE_V2 and \
                     has_roce_hw_bug(vendor_id, vendor_pid):
                 continue
             if not os.path.exists('/sys/class/infiniband/{}/device/net/'.format(dev)):
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH 3/8 verbs: Introduce a new query GID entry API

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Introduce the ibv_query_gid_ex verb which queries a specific index in the GID table of the given port of the given device. The queried data is stored in a buffer provided by the user.

If the kernel doesn't support ioctl or the needed uverbs method, the API will try to query the GID entry via sysfs.

This API provides a faster way to query a GID table entry using a single call over ioctl, instead of multiple calls to open, close and read multiple sysfs files for a single GID table entry.

---
 debian/libibverbs1.symbols           |   2 +
 libibverbs/CMakeLists.txt            |   2 +-
 libibverbs/cmd_device.c              | 105 +++++++++++++++++++++++++++++++++++
 libibverbs/libibverbs.map.in         |   5 ++
 libibverbs/man/CMakeLists.txt        |   1 +
 libibverbs/man/ibv_query_gid_ex.3.md |  93 +++++++++++++++++++++++++++++++
 libibverbs/verbs.h                   |  29 ++++++++++
 7 files changed, 236 insertions(+), 1 deletion(-)
 create mode 100644 libibverbs/man/ibv_query_gid_ex.3.md
debian/libibverbs1.symbols +2 -0
index 2efbe89..536d543 100644
@@ -8,7 +8,9 @@ libibverbs.so.1 libibverbs1 #MINVER#
  IBVERBS_1.8@IBVERBS_1.8 28
  IBVERBS_1.9@IBVERBS_1.9 30
  IBVERBS_1.10@IBVERBS_1.10 31
+ IBVERBS_1.11@IBVERBS_1.11 32
  (symver)IBVERBS_PRIVATE_25 25
+ _ibv_query_gid_ex@IBVERBS_1.11 32
  ibv_ack_async_event@IBVERBS_1.0 1.1.6
  ibv_ack_async_event@IBVERBS_1.1 1.1.6
  ibv_ack_cq_events@IBVERBS_1.0 1.1.6
libibverbs/CMakeLists.txt +1 -1
index 06a590f..0fe4256 100644
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
 
 rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
   # See Documentation/versioning.md
-  1 1.10.${PACKAGE_VERSION}
+  1 1.11.${PACKAGE_VERSION}
   all_providers.c
   cmd.c
   cmd_ah.c
libibverbs/cmd_device.c +105 -0

114 lines not on this page · raw message

libibverbs/libibverbs.map.in +5 -0
index 3240e00..dae4963 100644
@@ -142,6 +142,11 @@ IBVERBS_1.10 {
 		ibv_unimport_pd;
 } IBVERBS_1.9;
 
+IBVERBS_1.11 {
+	global:
+		_ibv_query_gid_ex;
+} IBVERBS_1.10;
+
 /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
    version. See the top level CMakeLists.txt for this setting. */
 
libibverbs/man/CMakeLists.txt +1 -0
index b925004..2dea4ff 100644
@@ -57,6 +57,7 @@ rdma_man_pages(
   ibv_query_device_ex.3
   ibv_query_ece.3.md
   ibv_query_gid.3.md
+  ibv_query_gid_ex.3.md
   ibv_query_pkey.3.md
   ibv_query_port.3
   ibv_query_qp.3
libibverbs/man/ibv_query_gid_ex.3.md new file +93 -0

93 lines not on this page · raw message

libibverbs/verbs.h +29 -0
index 2e785aa..e5bf900 100644
@@ -68,6 +68,20 @@ union ibv_gid {
 	} global;
 };
 
+enum ibv_gid_type {
+	IBV_GID_TYPE_IB,
+	IBV_GID_TYPE_ROCE_V1,
+	IBV_GID_TYPE_ROCE_V2,
+};
+
+struct ibv_gid_entry {
+	union ibv_gid gid;
+	uint32_t gid_index;
+	uint32_t port_num;
+	uint32_t gid_type; /* enum ibv_gid_type */
+	uint32_t ndev_ifindex;
+};
+
 #define vext_field_avail(type, fld, sz) (offsetof(type, fld) < (sz))
 
 #ifdef __cplusplus
@@ -2330,6 +2344,21 @@ static inline int ___ibv_query_port(struct ibv_context *context,
 int ibv_query_gid(struct ibv_context *context, uint8_t port_num,
 		  int index, union ibv_gid *gid);
 
+int _ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
+		     uint32_t gid_index, struct ibv_gid_entry *entry,
+		     uint32_t flags, size_t entry_size);
+
+/**
+ * ibv_query_gid_ex - Read a GID table entry
+ */
+static inline int ibv_query_gid_ex(struct ibv_context *context,
+				   uint32_t port_num, uint32_t gid_index,
+				   struct ibv_gid_entry *entry, uint32_t flags)
+{
+	return _ibv_query_gid_ex(context, port_num, gid_index, entry, flags,
+				 sizeof(*entry));
+}
+
 /**
  * ibv_query_pkey - Get a P_Key table entry
  */
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH 4/8 verbs: Implement ibv_query_gid and ibv_query_gid_type over ioctl

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Currently ibv_query_gid and ibv_query_gid_type are implemented over sysfs. In order to improve their performance we implement them using the new query GID entry API, so now they will use ioctl and fallback to sysfs.

---
 libibverbs/cmd_device.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++--
 libibverbs/verbs.c      | 92 +++++++++++++++----------------------------------
 2 files changed, 117 insertions(+), 67 deletions(-)
libibverbs/cmd_device.c +90 -2
index 8643cc6..f707273 100644
@@ -30,6 +30,10 @@
  * SOFTWARE.
  */
 
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <dirent.h>
 #include <infiniband/cmd_write.h>
 
 #include <net/if.h>
@@ -230,6 +234,90 @@ static int query_sysfs_gid_ndev_ifindex(struct ibv_context *context,
 	return *ndev_ifindex ? 0 : errno;
 }
 
+static int query_sysfs_gid(struct ibv_context *context, uint8_t port_num, int index,
+			   union ibv_gid *gid)
+{
+	struct verbs_device *verbs_device = verbs_get_device(context->device);
+	char attr[41];
+	uint16_t val;
+	int i;
+
+	if (ibv_read_ibdev_sysfs_file(attr, sizeof(attr), verbs_device->sysfs,
+				      "ports/%d/gids/%d", port_num, index) < 0)
+		return -1;
+
+	for (i = 0; i < 8; ++i) {
+		if (sscanf(attr + i * 5, "%hx", &val) != 1)
+			return -1;
+		gid->raw[i * 2] = val >> 8;
+		gid->raw[i * 2 + 1] = val & 0xff;
+	}
+
+	return 0;
+}
+
+/* GID types as appear in sysfs, no change is expected as of ABI
+ * compatibility.
+ */
+#define V1_TYPE "IB/RoCE v1"
+#define V2_TYPE "RoCE v2"
+static int query_sysfs_gid_type(struct ibv_context *context, uint8_t port_num,
+				unsigned int index, enum ibv_gid_type_sysfs *type)
+{
+	struct verbs_device *verbs_device = verbs_get_device(context->device);
+	char buff[11];
+
+	/* Reset errno so that we can rely on its value upon any error flow in
+	 * ibv_read_sysfs_file.
+	 */
+	errno = 0;
+	if (ibv_read_ibdev_sysfs_file(buff, sizeof(buff), verbs_device->sysfs,
+				      "ports/%d/gid_attrs/types/%d", port_num,
+				      index) <= 0) {
+		char *dir_path;
+		DIR *dir;
+
+		if (errno == EINVAL) {
+			/* In IB, this file doesn't exist and the kernel sets
+			 * errno to -EINVAL.
+			 */
+			*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
+			return 0;
+		}
+		if (asprintf(&dir_path, "%s/%s/%d/%s/",
+			     verbs_device->sysfs->ibdev_path, "ports", port_num,
+			     "gid_attrs") < 0)
+			return -1;
+		dir = opendir(dir_path);
+		free(dir_path);
+		if (!dir) {
+			if (errno == ENOENT)
+				/* Assuming that if gid_attrs doesn't exist,
+				 * we have an old kernel and all GIDs are
+				 * IB/RoCE v1
+				 */
+				*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
+			else
+				return -1;
+		} else {
+			closedir(dir);
+			errno = EFAULT;
+			return -1;
+		}
+	} else {
+		if (!strcmp(buff, V1_TYPE)) {
+			*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
+		} else if (!strcmp(buff, V2_TYPE)) {
+			*type = IBV_GID_TYPE_SYSFS_ROCE_V2;
+		} else {
+			errno = ENOTSUP;
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
 static int query_sysfs_gid_entry(struct ibv_context *context, uint32_t port_num,
 				 uint32_t gid_index,
 				 struct ibv_gid_entry *entry)
@@ -240,11 +328,11 @@ static int query_sysfs_gid_entry(struct ibv_context *context, uint32_t port_num,
 
 	entry->gid_index = gid_index;
 	entry->port_num = port_num;
-	ret = ibv_query_gid(context, port_num, gid_index, &entry->gid);
+	ret = query_sysfs_gid(context, port_num, gid_index, &entry->gid);
 	if (ret)
 		return EINVAL;
 
-	ret = ibv_query_gid_type(context, port_num, gid_index, &gid_type);
+	ret = query_sysfs_gid_type(context, port_num, gid_index, &gid_type);
 	if (ret)
 		return EINVAL;
 
libibverbs/verbs.c +27 -65

112 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#5 lore raw

PATCH 5/8 verbs: Optimize ibv_query_gid and ibv_query_gid_type

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

ibv_query_gid and ibv_query_gid_type are implemented as ioctl first and fallback to sysfs. Currently, if the fallback path is taken, all of the gid entry attributes are retrieved over sysfs.

For example, if ibv_query_gid is called and the fallback path is taken, the gid type and the gid ndev ifindex will also be read over sysfs, even though we only need the gid.

In order to eliminate these unnecessary sysfs reads, we add an attribute mask to ibv_cmd_query_gid_entry that will allow us to mark the specific gid entry attributes that we would like to query in fallback.

---
 libibverbs/cmd_device.c | 76 ++++++++++++++++++++++++++++++++-----------------
 libibverbs/driver.h     | 11 +++++++
 libibverbs/verbs.c      |  8 +++---
 3 files changed, 65 insertions(+), 30 deletions(-)
libibverbs/cmd_device.c +50 -26
index f707273..4f85010 100644
@@ -320,41 +320,49 @@ static int query_sysfs_gid_type(struct ibv_context *context, uint8_t port_num,
 
 static int query_sysfs_gid_entry(struct ibv_context *context, uint32_t port_num,
 				 uint32_t gid_index,
-				 struct ibv_gid_entry *entry)
+				 struct ibv_gid_entry *entry,
+				 uint32_t attr_mask)
 {
 	enum ibv_gid_type_sysfs gid_type;
 	struct ibv_port_attr port_attr = {};
-	int ret;
+	int ret = 0;
 
 	entry->gid_index = gid_index;
 	entry->port_num = port_num;
-	ret = query_sysfs_gid(context, port_num, gid_index, &entry->gid);
-	if (ret)
-		return EINVAL;
-
-	ret = query_sysfs_gid_type(context, port_num, gid_index, &gid_type);
-	if (ret)
-		return EINVAL;
 
-	if (gid_type == IBV_GID_TYPE_SYSFS_IB_ROCE_V1) {
-		ret = ibv_query_port(context, port_num, &port_attr);
+	if (attr_mask & VERBS_QUERY_GID_ATTR_GID) {
+		ret = query_sysfs_gid(context, port_num, gid_index, &entry->gid);
 		if (ret)
-			goto out;
+			return EINVAL;
+	}
 
-		if (port_attr.link_layer == IBV_LINK_LAYER_INFINIBAND) {
-			entry->gid_type = IBV_GID_TYPE_IB;
-		} else if (port_attr.link_layer == IBV_LINK_LAYER_ETHERNET) {
-			entry->gid_type = IBV_GID_TYPE_ROCE_V1;
+	if (attr_mask & VERBS_QUERY_GID_ATTR_TYPE) {
+		ret = query_sysfs_gid_type(context, port_num, gid_index, &gid_type);
+		if (ret)
+			return EINVAL;
+
+		if (gid_type == IBV_GID_TYPE_SYSFS_IB_ROCE_V1) {
+			ret = ibv_query_port(context, port_num, &port_attr);
+			if (ret)
+				goto out;
+
+			if (port_attr.link_layer == IBV_LINK_LAYER_INFINIBAND) {
+				entry->gid_type = IBV_GID_TYPE_IB;
+			} else if (port_attr.link_layer ==
+				   IBV_LINK_LAYER_ETHERNET) {
+				entry->gid_type = IBV_GID_TYPE_ROCE_V1;
+			} else {
+				ret = EINVAL;
+				goto out;
+			}
 		} else {
-			ret = EINVAL;
-			goto out;
+			entry->gid_type = IBV_GID_TYPE_ROCE_V2;
 		}
-	} else {
-		entry->gid_type = IBV_GID_TYPE_ROCE_V2;
 	}
 
-	ret = query_sysfs_gid_ndev_ifindex(context, port_num, gid_index,
-					   &entry->ndev_ifindex);
+	if (attr_mask & VERBS_QUERY_GID_ATTR_NDEV_IFINDEX)
+		ret = query_sysfs_gid_ndev_ifindex(context, port_num, gid_index,
+						   &entry->ndev_ifindex);
 
 out:
 	return ret;
@@ -364,9 +372,10 @@ out:
  * verbs_context_ops while async_event is and doesn't use ioctl.
  */
 #define query_gid_kernel_cap async_event
-int _ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
+int __ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
 			    uint32_t gid_index, struct ibv_gid_entry *entry,
-			    uint32_t flags, size_t entry_size)
+			    uint32_t flags, size_t entry_size,
+			    uint32_t fallback_attr_mask)
 {
 	DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_DEVICE,
 			       UVERBS_METHOD_QUERY_GID_ENTRY, 4);
@@ -386,12 +395,27 @@ int _ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
 			return EOPNOTSUPP;
 
 		ret = query_sysfs_gid_entry(context, port_num, gid_index,
-					    entry);
+					    entry, fallback_attr_mask);
 		if (ret)
 			return ret;
 
-		return is_zero_gid(&entry->gid) ? ENODATA : 0;
+		if (fallback_attr_mask & VERBS_QUERY_GID_ATTR_GID &&
+		    is_zero_gid(&entry->gid))
+			return ENODATA;
+
+		return 0;
 	default:
 		return ret;
 	}
 }
+
+int _ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
+		      uint32_t gid_index, struct ibv_gid_entry *entry,
+		      uint32_t flags, size_t entry_size)
+{
+	return __ibv_query_gid_ex(context, port_num, gid_index, entry,
+				  flags, entry_size,
+				  VERBS_QUERY_GID_ATTR_GID |
+				  VERBS_QUERY_GID_ATTR_TYPE |
+				  VERBS_QUERY_GID_ATTR_NDEV_IFINDEX);
+}
libibverbs/driver.h +11 -0
index 046c07d..479e16e 100644
@@ -77,6 +77,12 @@ enum ibv_gid_type_sysfs {
 	IBV_GID_TYPE_SYSFS_ROCE_V2,
 };
 
+enum verbs_query_gid_attr_mask {
+	VERBS_QUERY_GID_ATTR_GID		= 1 << 0,
+	VERBS_QUERY_GID_ATTR_TYPE		= 1 << 1,
+	VERBS_QUERY_GID_ATTR_NDEV_IFINDEX	= 1 << 2,
+};
+
 enum ibv_mr_type {
 	IBV_MR_TYPE_MR,
 	IBV_MR_TYPE_NULL_MR,
@@ -633,6 +639,11 @@ int ibv_cmd_reg_dm_mr(struct ibv_pd *pd, struct verbs_dm *dm,
 		      unsigned int access, struct verbs_mr *vmr,
 		      struct ibv_command_buffer *link);
 
+int __ibv_query_gid_ex(struct ibv_context *context, uint32_t port_num,
+			    uint32_t gid_index, struct ibv_gid_entry *entry,
+			    uint32_t flags, size_t entry_size,
+			    uint32_t fallback_attr_mask);
+
 /*
  * sysfs helper functions
  */
libibverbs/verbs.c +4 -4
index 99a48d5..e50ad94 100644
@@ -224,8 +224,8 @@ LATEST_SYMVER_FUNC(ibv_query_gid, 1_1, "IBVERBS_1.1",
 	struct ibv_gid_entry entry = {};
 	int ret;
 
-	ret = _ibv_query_gid_ex(context, port_num, index, &entry, 0,
-				sizeof(entry));
+	ret = __ibv_query_gid_ex(context, port_num, index, &entry, 0,
+				 sizeof(entry), VERBS_QUERY_GID_ATTR_GID);
 	/* Preserve API behavior for empty GID */
 	if (ret == ENODATA) {
 		memset(gid, 0, sizeof(*gid));
@@ -703,8 +703,8 @@ int ibv_query_gid_type(struct ibv_context *context, uint8_t port_num,
 	struct ibv_gid_entry entry = {};
 	int ret;
 
-	ret = _ibv_query_gid_ex(context, port_num, index, &entry, 0,
-				sizeof(entry));
+	ret = __ibv_query_gid_ex(context, port_num, index, &entry, 0,
+				 sizeof(entry), VERBS_QUERY_GID_ATTR_TYPE);
 	/* Preserve API behavior for empty GID */
 	if (ret == ENODATA) {
 		*type = IBV_GID_TYPE_SYSFS_IB_ROCE_V1;
Signature · 1 line
1.8.3.1
#6 lore raw

PATCH 6/8 verbs: Introduce a new query GID table API

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Introduce the ibv_query_gid_table verb which queries the GID tables of the given device and stores the queried data in a buffer provided by the user.

If the kernel doesn't support ioctl or the needed uverbs method, the API will try to query the GID tables via sysfs.

This API provides a faster way to query the GID tables of a device using a single call over ioctl, instead of multiple calls to open, close and read multiple sysfs files for a single GID table entry.

---
 debian/libibverbs1.symbols              |   1 +
 libibverbs/cmd_device.c                 | 117 +++++++++++++++++++++++++++++---
 libibverbs/libibverbs.map.in            |   1 +
 libibverbs/man/CMakeLists.txt           |   1 +
 libibverbs/man/ibv_query_gid_table.3.md |  73 ++++++++++++++++++++
 libibverbs/verbs.h                      |  16 +++++
 6 files changed, 199 insertions(+), 10 deletions(-)
 create mode 100644 libibverbs/man/ibv_query_gid_table.3.md
debian/libibverbs1.symbols +1 -0
index 536d543..99257de 100644
@@ -11,6 +11,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
  IBVERBS_1.11@IBVERBS_1.11 32
  (symver)IBVERBS_PRIVATE_25 25
  _ibv_query_gid_ex@IBVERBS_1.11 32
+ _ibv_query_gid_table@IBVERBS_1.11 32
  ibv_ack_async_event@IBVERBS_1.0 1.1.6
  ibv_ack_async_event@IBVERBS_1.1 1.1.6
  ibv_ack_cq_events@IBVERBS_1.0 1.1.6
libibverbs/cmd_device.c +107 -10

146 lines not on this page · raw message

libibverbs/libibverbs.map.in +1 -0
index dae4963..7429016 100644
@@ -145,6 +145,7 @@ IBVERBS_1.10 {
 IBVERBS_1.11 {
 	global:
 		_ibv_query_gid_ex;
+		_ibv_query_gid_table;
 } IBVERBS_1.10;
 
 /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
libibverbs/man/CMakeLists.txt +1 -0
index 2dea4ff..1fb5ac1 100644
@@ -58,6 +58,7 @@ rdma_man_pages(
   ibv_query_ece.3.md
   ibv_query_gid.3.md
   ibv_query_gid_ex.3.md
+  ibv_query_gid_table.3.md
   ibv_query_pkey.3.md
   ibv_query_port.3
   ibv_query_qp.3
libibverbs/man/ibv_query_gid_table.3.md new file +73 -0

73 lines not on this page · raw message

libibverbs/verbs.h +16 -0
index e5bf900..caf626c 100644
@@ -43,6 +43,7 @@
 #include <string.h>
 #include <linux/types.h>
 #include <stdint.h>
+#include <sys/types.h>
 #include <infiniband/verbs_api.h>
 
 #ifdef __cplusplus
@@ -2359,6 +2360,21 @@ static inline int ibv_query_gid_ex(struct ibv_context *context,
 				 sizeof(*entry));
 }
 
+ssize_t _ibv_query_gid_table(struct ibv_context *context,
+			     struct ibv_gid_entry *entries, size_t max_entries,
+			     uint32_t flags, size_t entry_size);
+
+/*
+ * ibv_query_gid_table - Get all valid GID table entries
+ */
+static inline ssize_t ibv_query_gid_table(struct ibv_context *context,
+					  struct ibv_gid_entry *entries,
+					  size_t max_entries, uint32_t flags)
+{
+	return _ibv_query_gid_table(context, entries, max_entries, flags,
+				    sizeof(*entries));
+}
+
 /**
  * ibv_query_pkey - Get a P_Key table entry
  */
Signature · 1 line
1.8.3.1
#7 lore raw

PATCH 7/8 pyverbs: Add query_gid_table and query_gid_ex methods

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Add two new methods to Context class: query_gid_table and query_gid_ex.

query_gid_table queries all GID tables of the device and returns a list of GIDEntry objects containing all valid GID entries.

query_gid_ex queries the GID table of the given port in the given index and returns a GIDEntry object.

---
 pyverbs/device.pxd           |   3 ++
 pyverbs/device.pyx           | 106 +++++++++++++++++++++++++++++++++++++++++++
 pyverbs/libibverbs.pxd       |  13 ++++++
 pyverbs/libibverbs_enums.pxd |   5 ++
 4 files changed, 127 insertions(+)
pyverbs/device.pxd +3 -0
index 73328d3..0519c4b 100755
@@ -64,3 +64,6 @@ cdef class DM(PyverbsCM):
 
 cdef class PortAttr(PyverbsObject):
     cdef v.ibv_port_attr attr
+
+cdef class GIDEntry(PyverbsObject):
+    cdef v.ibv_gid_entry entry
pyverbs/device.pyx +106 -0

124 lines not on this page · raw message

pyverbs/libibverbs.pxd +13 -0
index c84b9fc..6fbba54 100755
@@ -483,6 +483,13 @@ cdef extern from 'infiniband/verbs.h':
         uint32_t options
         uint32_t comp_mask
 
+    cdef struct ibv_gid_entry:
+        ibv_gid gid
+        uint32_t gid_index
+        uint32_t port_num
+        uint32_t gid_type
+        uint32_t ndev_ifindex
+
     ibv_device **ibv_get_device_list(int *n)
     int ibv_get_device_index(ibv_device *device);
     void ibv_free_device_list(ibv_device **list)
@@ -613,6 +620,12 @@ cdef extern from 'infiniband/verbs.h':
     void ibv_unimport_mr(ibv_mr *mr)
     ibv_pd *ibv_import_pd(ibv_context *context, uint32_t handle)
     void ibv_unimport_pd(ibv_pd *pd)
+    int ibv_query_gid_ex(ibv_context *context, uint32_t port_num,
+                         uint32_t gid_index, ibv_gid_entry *entry,
+                         uint32_t flags)
+    ssize_t ibv_query_gid_table(ibv_context *context,
+                                ibv_gid_entry *entries, size_t max_entries,
+                                uint32_t flags)
 
 
 cdef extern from 'infiniband/driver.h':
pyverbs/libibverbs_enums.pxd +5 -0
index 83ca516..a5c07b3 100755
@@ -427,6 +427,11 @@ cdef extern from '<infiniband/verbs.h>':
 
     cdef void *IBV_ALLOCATOR_USE_DEFAULT
 
+    cpdef enum ibv_gid_type:
+        IBV_GID_TYPE_IB
+        IBV_GID_TYPE_ROCE_V1
+        IBV_GID_TYPE_ROCE_V2
+
 
 cdef extern from "<infiniband/verbs_api.h>":
     cdef unsigned long long IBV_ADVISE_MR_ADVICE_PREFETCH
Signature · 1 line
1.8.3.1
#8 lore raw

PATCH 8/8 tests: Add tests for ibv_query_gid_table and ibv_query_gid_ex

In reply to Yishai Hadas

From: Avihai Horon <[email protected]>

Add a test for ibv_query_gid_table and another one for ibv_query_gid_ex.

---
 tests/test_device.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
tests/test_device.py +32 -0
index 1eb1a81..94c0e40 100644
@@ -6,6 +6,7 @@ Test module for pyverbs' device module.
 import unittest
 import resource
 import random
+import errno
 
 from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError
 from tests.base import PyverbsAPITestCase
@@ -80,6 +81,37 @@ class DeviceTest(PyverbsAPITestCase):
             with d.Context(name=dev.name.decode()) as ctx:
                 ctx.query_gid(port_num=1, index=0)
 
+    def test_query_gid_table(self):
+        """
+        Test ibv_query_gid_table()
+        """
+        devs = self.get_device_list()
+        with d.Context(name=devs[0].name.decode()) as ctx:
+            device_attr = ctx.query_device()
+            port_attr = ctx.query_port(1)
+            max_entries = device_attr.phys_port_cnt * port_attr.gid_tbl_len
+            try:
+                ctx.query_gid_table(max_entries)
+            except PyverbsRDMAError as ex:
+                if ex.error_code in [errno.EOPNOTSUPP, errno.EPROTONOSUPPORT]:
+                    raise unittest.SkipTest('ibv_query_gid_table is not'\
+                                            ' supported on this device')
+                raise ex
+
+    def test_query_gid_ex(self):
+        """
+        Test ibv_query_gid_ex()
+        """
+        devs = self.get_device_list()
+        with d.Context(name=devs[0].name.decode()) as ctx:
+            try:
+                ctx.query_gid_ex(port_num=1, gid_index=0)
+            except PyverbsRDMAError as ex:
+                if ex.error_code in [errno.EOPNOTSUPP, errno.EPROTONOSUPPORT]:
+                    raise unittest.SkipTest('ibv_query_gid_ex is not'\
+                                            ' supported on this device')
+                raise ex
+
     @staticmethod
     def verify_device_attr(attr, device):
         """
Signature · 1 line
1.8.3.1
#9 lore raw
In reply to Yishai Hadas
Quoted 72 lines · Yishai Hadas

When an application is not using RDMA CM and if it is using multiple RDMA devices with one or more RoCE ports, finding the right GID table entry is a long process.

For example, with two RoCE dual-port devices in a system, when IP failover is used between two RoCE ports, searching a suitable GID entry for a given source IP, matching netdevice of given RoCEv1/v2 type requires iterating over all 4 ports * 256 entry GID table.

Even though the best first match GID table for given criteria is used, when the matching entry is on the 4th port, it requires reading 3 ports * 256 entries * 3 files (GID, netdev, type) = 2304 files. The GID table needs to be referred on every QP creation during IP failover on other netdevice of an RDMA device.

We introduce this series of patches, which introduces an API to query the complete GID tables of an RDMA device, that returns all valid GID table entries.

This is done through single ioctl, eliminating 2304 read, 2304 open and 2304 close system calls to just a total of 2 calls (one for each device).

While at it, we also introduce an API to query an individual GID entry over ioctl interface, which provides all GID attributes information.

The APIs are based on the below RFC [1], the matching kernel part was sent to rdma-next.

PR was sent as well [2].

[1] https://www.spinics.net/lists/linux-rdma/msg91825.html
[2] https://github.com/linux-rdma/rdma-core/pull/828

Changelog:
v1:
* Move some GIDs stuff to cmd_device.c with better function's names.
* Cleanup in few places.

Avihai Horon (7):
verbs: Change the name of enum ibv_gid_type
verbs: Introduce a new query GID entry API
verbs: Implement ibv_query_gid and ibv_query_gid_type over ioctl
verbs: Optimize ibv_query_gid and ibv_query_gid_type
verbs: Introduce a new query GID table API
pyverbs: Add query_gid_table and query_gid_ex methods
tests: Add tests for ibv_query_gid_table and ibv_query_gid_ex

Yishai Hadas (1):
Update kernel headers

  debian/libibverbs1.symbols                |   3 +
  kernel-headers/rdma/ib_user_ioctl_cmds.h  |  16 ++
  kernel-headers/rdma/ib_user_ioctl_verbs.h |  14 ++
  kernel-headers/rdma/ib_user_verbs.h       |  11 ++
  kernel-headers/rdma/rdma_user_rxe.h       |   6 +-
  libibverbs/CMakeLists.txt                 |   2 +-
  libibverbs/cmd_device.c                   | 314 ++++++++++++++++++++++++++++++
  libibverbs/driver.h                       |  19 +-
  libibverbs/examples/devinfo.c             |  14 +-
  libibverbs/libibverbs.map.in              |   6 +
  libibverbs/man/CMakeLists.txt             |   2 +
  libibverbs/man/ibv_query_gid_ex.3.md      |  93 +++++++++
  libibverbs/man/ibv_query_gid_table.3.md   |  73 +++++++
  libibverbs/verbs.c                        | 105 ++++------
  libibverbs/verbs.h                        |  45 +++++
  providers/mlx5/verbs.c                    |   2 +-
  pyverbs/device.pxd                        |   3 +
  pyverbs/device.pyx                        | 108 +++++++++-
  pyverbs/libibverbs.pxd                    |  15 +-
  pyverbs/libibverbs_enums.pxd              |  11 +-
  tests/base.py                             |   3 +-
  tests/test_device.py                      |  32 +++
  22 files changed, 804 insertions(+), 93 deletions(-)
Quoted 3 lines

create mode 100644 libibverbs/man/ibv_query_gid_ex.3.md create mode 100644 libibverbs/man/ibv_query_gid_table.3.md

The PR was merged.

Thanks,
Yishai