cm_id, cq, mr, and pd resource tracking
Subsystems (4)
v4 cover cm_id, cq, mr, and pd resource tracking
This series enhances the iproute2 rdma tool to include dumping of connection manager id (cm_id), completion queue (cq), memory region (mr), and protection domain (pd) rdma resources. It is the user-space part of the kernel resource tracking series merged into rdma-next for 4.17 [1] and [2].
Changes since v3:
- replaced rdma_cma.h inclusion with UAPI rdma_user_cm.h
- display only device names instead of device/port for cq, mr, and pd since they are not associated with a specific port.
Changes since v2:
- pull in rdma-core:include/rdma/rdma_cma.h
- 80 column reformat
- add reviewed-by tags
Changes since v1/RFC:
- removed RFC tag
- initialize rd properly to avoid passing a garbage port number
- revert accidental change to qp_valid_filters
- removed cm_id dev/network/transport types
- cm_id ip addrs now passed up as __kernel_sockaddr_storage
- cm_id ip address ports printed as "address:port" strings
- only parse/display memory keys and iova if available
- filter on "users" for cqs and pds
- fixed memory leaks
- removed PD_FLAGS attribute
- filter on "mrlen" for mrs
- filter on "poll-ctx" for cqs
- don't require addrs or qp_type for parsing cm_ids
- only filter optional attrs if they are present
- remove PGSIZE MR attr to match kernel
[1] https://www.spinics.net/lists/linux-rdma/msg61720.html
[2] https://www.spinics.net/lists/linux-rdma/msg62979.html
https://www.spinics.net/lists/linux-rdma/msg62980.html
---
Steve Wise (7):
rdma: update rdma_netlink.h
rdma: add UAPI rdma_user_cm.h
rdma: initialize the rd struct
rdma: Add CM_ID resource tracking information
rdma: Add CQ resource tracking information
rdma: Add MR resource tracking information
rdma: Add PD resource tracking information
include/json_writer.h | 2 + lib/json_writer.c | 11 + rdma/include/uapi/rdma/rdma_netlink.h | 38 ++ rdma/include/uapi/rdma/rdma_user_cm.h | 324 +++++++++++++++++ rdma/rdma.c | 2 +- rdma/rdma.h | 2 + rdma/res.c | 633 +++++++++++++++++++++++++++++++++- rdma/utils.c | 16 + 8 files changed, 1026 insertions(+), 2 deletions(-) create mode 100644 rdma/include/uapi/rdma/rdma_user_cm.h
Signature · 1 line
1.8.3.1
PATCH v4 1/7 rdma: update rdma_netlink.h
From: Steve Wise <[email protected]>
Pull in the latest rdma_netlink.h which has support for the rdma nldev resource tracking objects being added with this patch series.
--- rdma/include/uapi/rdma/rdma_netlink.h | 38 +++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
rdma/include/uapi/rdma/rdma_netlink.h +38 -0
@@ -238,6 +238,14 @@ enum rdma_nldev_command {RDMA_NLDEV_CMD_RES_QP_GET ,/* can dump */ +RDMA_NLDEV_CMD_RES_CM_ID_GET ,/* can dump */ + +RDMA_NLDEV_CMD_RES_CQ_GET ,/* can dump */ + +RDMA_NLDEV_CMD_RES_MR_GET ,/* can dump */ + +RDMA_NLDEV_CMD_RES_PD_GET ,/* can dump */ + RDMA_NLDEV_NUM_OPS };
@@ -350,6 +358,36 @@ enum rdma_nldev_attr {*/ RDMA_NLDEV_ATTR_RES_KERN_NAME ,/* string */ +RDMA_NLDEV_ATTR_RES_CM_ID ,/* nested table */ +RDMA_NLDEV_ATTR_RES_CM_ID_ENTRY ,/* nested table */ +/* +* rdma_cm_id port space. +*/ +RDMA_NLDEV_ATTR_RES_PS ,/* u32 */ +/* +* Source and destination socket addresses +*/ +RDMA_NLDEV_ATTR_RES_SRC_ADDR ,/* __kernel_sockaddr_storage */ +RDMA_NLDEV_ATTR_RES_DST_ADDR ,/* __kernel_sockaddr_storage */ + +RDMA_NLDEV_ATTR_RES_CQ ,/* nested table */ +RDMA_NLDEV_ATTR_RES_CQ_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_RES_CQE ,/* u32 */ +RDMA_NLDEV_ATTR_RES_USECNT ,/* u64 */ +RDMA_NLDEV_ATTR_RES_POLL_CTX ,/* u8 */ + +RDMA_NLDEV_ATTR_RES_MR ,/* nested table */ +RDMA_NLDEV_ATTR_RES_MR_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_RES_RKEY ,/* u32 */ +RDMA_NLDEV_ATTR_RES_LKEY ,/* u32 */ +RDMA_NLDEV_ATTR_RES_IOVA ,/* u64 */ +RDMA_NLDEV_ATTR_RES_MRLEN ,/* u64 */ + +RDMA_NLDEV_ATTR_RES_PD ,/* nested table */ +RDMA_NLDEV_ATTR_RES_PD_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ,/* u32 */ +RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ,/* u32 */ +RDMA_NLDEV_ATTR_MAX };#endif /* _RDMA_NETLINK_H */
Signature · 1 line
1.8.3.1
Re: [PATCH v4 iproute2-next 1/7] rdma: update rdma_netlink.h
Quoted 11 lines · Steve Wise
From: Steve Wise <[email protected]>
Pull in the latest
rdma_netlink.hwhich has support for the rdma nldev resource tracking objects being added with this patch series.Signed-off-by Steve Wise <[email protected]>--- rdma/include/uapi/rdma/rdma_netlink.h | 38 +++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
Thanks,
PATCH v4 2/7 rdma: add UAPI rdma_user_cm.h
In reply to Steve WiseThis allows parsing rdma_cm_id UAPI values.
--- rdma/include/uapi/rdma/rdma_user_cm.h | 324 ++++++++++++++++++++++++++++++++++ 1 file changed, 324 insertions(+) create mode 100644 rdma/include/uapi/rdma/rdma_user_cm.h
rdma/include/uapi/rdma/rdma_user_cm.h new file +324 -0
@@ -0,0 +1,324 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) */ +/* +* Copyright (c) 2005-2006 Intel Corporation. All rights reserved. +* +* This software is available to you under a choice of one of two +* licenses. You may choose to be licensed under the terms of the GNU +* General Public License (GPL) Version 2, available from the file +* COPYING in the main directory of this source tree, or the +* OpenIB.org BSD license below: +* +* Redistribution and use in source and binary forms, with or +* without modification, are permitted provided that the following +* conditions are met: +* +* - Redistributions of source code must retain the above +* copyright notice, this list of conditions and the following +* disclaimer. +* +* - Redistributions in binary form must reproduce the above +* copyright notice, this list of conditions and the following +* disclaimer in the documentation and/or other materials +* provided with the distribution. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#ifndef _RDMA_USER_CM_H +#define _RDMA_USER_CM_H + +#include <linux/types.h> +#include <linux/socket.h> +#include <linux/in6.h> +#include <rdma/ib_user_verbs.h> +#include <rdma/ib_user_sa.h> + +#define RDMA_USER_CM_ABI_VERSION 4 + +#define RDMA_MAX_PRIVATE_DATA 256 + +enum { +RDMA_USER_CM_CMD_CREATE_ID , +RDMA_USER_CM_CMD_DESTROY_ID , +RDMA_USER_CM_CMD_BIND_IP , +RDMA_USER_CM_CMD_RESOLVE_IP , +RDMA_USER_CM_CMD_RESOLVE_ROUTE , +RDMA_USER_CM_CMD_QUERY_ROUTE , +RDMA_USER_CM_CMD_CONNECT , +RDMA_USER_CM_CMD_LISTEN , +RDMA_USER_CM_CMD_ACCEPT , +RDMA_USER_CM_CMD_REJECT , +RDMA_USER_CM_CMD_DISCONNECT , +RDMA_USER_CM_CMD_INIT_QP_ATTR , +RDMA_USER_CM_CMD_GET_EVENT , +RDMA_USER_CM_CMD_GET_OPTION , +RDMA_USER_CM_CMD_SET_OPTION , +RDMA_USER_CM_CMD_NOTIFY , +RDMA_USER_CM_CMD_JOIN_IP_MCAST , +RDMA_USER_CM_CMD_LEAVE_MCAST , +RDMA_USER_CM_CMD_MIGRATE_ID , +RDMA_USER_CM_CMD_QUERY , +RDMA_USER_CM_CMD_BIND , +RDMA_USER_CM_CMD_RESOLVE_ADDR , +RDMA_USER_CM_CMD_JOIN_MCAST +}; + +/* See IBTA Annex A11, servies ID bytes 4 & 5 */ +enum rdma_ucm_port_space { +RDMA_PS_IPOIB =0x0002 , +RDMA_PS_IB =0x013F , +RDMA_PS_TCP =0x0106 , +RDMA_PS_UDP =0x0111 , +}; + +/* +* command ABI structures. +*/ +struct rdma_ucm_cmd_hdr { +__u32 cmd; +__u16 in; +__u16 out; +}; + +struct rdma_ucm_create_id { +__aligned_u64 uid; +__aligned_u64 response; +__u16 ps;/* use enum rdma_ucm_port_space */ +__u8 qp_type; +__u8 reserved[5 ]; +}; + +struct rdma_ucm_create_id_resp { +__u32 id; +}; + +struct rdma_ucm_destroy_id { +__aligned_u64 response; +__u32 id; +__u32 reserved; +}; + +struct rdma_ucm_destroy_id_resp { +__u32 events_reported; +}; + +struct rdma_ucm_bind_ip { +__aligned_u64 response; +struct sockaddr_in6 addr; +__u32 id; +}; + +struct rdma_ucm_bind { +__u32 id; +__u16 addr_size; +__u16 reserved; +struct __kernel_sockaddr_storage addr; +}; + +struct rdma_ucm_resolve_ip { +struct sockaddr_in6 src_addr; +struct sockaddr_in6 dst_addr; +__u32 id; +__u32 timeout_ms; +}; + +struct rdma_ucm_resolve_addr { +__u32 id; +__u32 timeout_ms; +__u16 src_size; +__u16 dst_size; +__u32 reserved; +struct __kernel_sockaddr_storage src_addr; +struct __kernel_sockaddr_storage dst_addr; +}; + +struct rdma_ucm_resolve_route { +__u32 id; +__u32 timeout_ms; +}; + +enum { +RDMA_USER_CM_QUERY_ADDR , +RDMA_USER_CM_QUERY_PATH , +RDMA_USER_CM_QUERY_GID +}; + +struct rdma_ucm_query { +__aligned_u64 response; +__u32 id; +__u32 option; +}; + +struct rdma_ucm_query_route_resp { +__aligned_u64 node_guid; +struct ib_user_path_rec ib_route[2 ]; +struct sockaddr_in6 src_addr; +struct sockaddr_in6 dst_addr; +__u32 num_paths; +__u8 port_num; +__u8 reserved[3 ]; +}; + +struct rdma_ucm_query_addr_resp { +__aligned_u64 node_guid; +__u8 port_num; +__u8 reserved; +__u16 pkey; +__u16 src_size; +__u16 dst_size; +struct __kernel_sockaddr_storage src_addr; +struct __kernel_sockaddr_storage dst_addr; +}; + +struct rdma_ucm_query_path_resp { +__u32 num_paths; +__u32 reserved; +struct ib_path_rec_data path_data[0 ]; +}; + +struct rdma_ucm_conn_param { +__u32 qp_num; +__u32 qkey; +__u8 private_data[RDMA_MAX_PRIVATE_DATA ]; +__u8 private_data_len; +__u8 srq; +__u8 responder_resources; +__u8 initiator_depth; +__u8 flow_control; +__u8 retry_count; +__u8 rnr_retry_count; +__u8 valid; +}; + +struct rdma_ucm_ud_param { +__u32 qp_num; +__u32 qkey; +struct ib_uverbs_ah_attr ah_attr; +__u8 private_data[RDMA_MAX_PRIVATE_DATA ]; +__u8 private_data_len; +__u8 reserved[7 ]; +}; + +struct rdma_ucm_connect { +struct rdma_ucm_conn_param conn_param; +__u32 id; +__u32 reserved; +}; + +struct rdma_ucm_listen { +__u32 id; +__u32 backlog; +}; + +struct rdma_ucm_accept { +__aligned_u64 uid; +struct rdma_ucm_conn_param conn_param; +__u32 id; +__u32 reserved; +}; + +struct rdma_ucm_reject { +__u32 id; +__u8 private_data_len; +__u8 reserved[3 ]; +__u8 private_data[RDMA_MAX_PRIVATE_DATA ]; +}; + +struct rdma_ucm_disconnect { +__u32 id; +}; + +struct rdma_ucm_init_qp_attr { +__aligned_u64 response; +__u32 id; +__u32 qp_state; +}; + +struct rdma_ucm_notify { +__u32 id; +__u32 event; +}; + +struct rdma_ucm_join_ip_mcast { +__aligned_u64 response;/* rdma_ucm_create_id_resp */ +__aligned_u64 uid; +struct sockaddr_in6 addr; +__u32 id; +}; + +/* Multicast join flags */ +enum { +RDMA_MC_JOIN_FLAG_FULLMEMBER , +RDMA_MC_JOIN_FLAG_SENDONLY_FULLMEMBER , +RDMA_MC_JOIN_FLAG_RESERVED , +}; + +struct rdma_ucm_join_mcast { +__aligned_u64 response;/* rdma_ucma_create_id_resp */ +__aligned_u64 uid; +__u32 id; +__u16 addr_size; +__u16 join_flags; +struct __kernel_sockaddr_storage addr; +}; + +struct rdma_ucm_get_event { +__aligned_u64 response; +}; + +struct rdma_ucm_event_resp { +__aligned_u64 uid; +__u32 id; +__u32 event; +__u32 status; +/* +* NOTE: This union is not aligned to 8 bytes so none of the union +* members may contain a u64 or anything with higher alignment than 4. +*/ +union { +struct rdma_ucm_conn_param conn; +struct rdma_ucm_ud_param ud; + } param; +__u32 reserved; +}; + +/* Option levels */ +enum { +RDMA_OPTION_ID =0 , +RDMA_OPTION_IB =1 +}; + +/* Option details */ +enum { +RDMA_OPTION_ID_TOS =0 , +RDMA_OPTION_ID_REUSEADDR =1 , +RDMA_OPTION_ID_AFONLY =2 , +RDMA_OPTION_IB_PATH =1 +}; + +struct rdma_ucm_set_option { +__aligned_u64 optval; +__u32 id; +__u32 level; +__u32 optname; +__u32 optlen; +}; + +struct rdma_ucm_migrate_id { +__aligned_u64 response; +__u32 id; +__u32 fd; +}; + +struct rdma_ucm_migrate_resp { +__u32 events_reported; +}; + +#endif /* _RDMA_USER_CM_H */
Signature · 1 line
1.8.3.1
PATCH v4 3/7 rdma: initialize the rd struct
In reply to Steve WiseInitialize the rd struct so port_idx is 0 unless set otherwise. Otherwise, strict_port queries end up passing an uninitialized PORT nlattr.
--- rdma/rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
rdma/rdma.c +1 -1
@@ -135,7 +135,7 @@ int main(int argc, char **argv)bool json_output = false;bool force = false;char *filename; -struct rd rd; +struct rd rd = {};int opt;int err;
Signature · 1 line
1.8.3.1
PATCH v4 4/7 rdma: Add CM_ID resource tracking information
In reply to Steve WiseSample output:
# rdma resource 2: cxgb4_0: pd 5 cq 2 qp 2 cm_id 3 mr 7 3: mlx4_0: pd 7 cq 3 qp 3 cm_id 3 mr 7
# rdma resource show cm_id link cxgb4_0/- lqpn 0 qp-type RC state LISTEN ps TCP pid 30485 comm rping src-addr 0.0.0.0:7174 link cxgb4_0/2 lqpn 1048 qp-type RC state CONNECT ps TCP pid 30503 comm rping src-addr 172.16.2.1:7174 dst-addr 172.16.2.1:38246 link cxgb4_0/2 lqpn 1040 qp-type RC state CONNECT ps TCP pid 30498 comm rping src-addr 172.16.2.1:38246 dst-addr 172.16.2.1:7174 link mlx4_0/- lqpn 0 qp-type RC state LISTEN ps TCP pid 30485 comm rping src-addr 0.0.0.0:7174 link mlx4_0/1 lqpn 539 qp-type RC state CONNECT ps TCP pid 30494 comm rping src-addr 172.16.99.1:7174 dst-addr 172.16.99.1:43670 link mlx4_0/1 lqpn 538 qp-type RC state CONNECT ps TCP pid 30492 comm rping src-addr 172.16.99.1:43670 dst-addr 172.16.99.1:7174
# rdma resource show cm_id dst-port 7174 link cxgb4_0/2 lqpn 1040 qp-type RC state CONNECT ps TCP pid 30498 comm rping src-addr 172.16.2.1:38246 dst-addr 172.16.2.1:7174 link mlx4_0/1 lqpn 538 qp-type RC state CONNECT ps TCP pid 30492 comm rping src-addr 172.16.99.1:43670 dst-addr 172.16.99.1:7174
--- rdma/rdma.h | 2 + rdma/res.c | 262 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- rdma/utils.c | 5 ++ 3 files changed, 268 insertions(+), 1 deletion(-)
rdma/rdma.h +2 -0
@@ -17,7 +17,9 @@#include <getopt.h> #include <libmnl/libmnl.h> #include <rdma/rdma_netlink.h> +#include <rdma/rdma_user_cm.h> #include <time.h> +#include <net/if_arp.h> #include "list.h" #include "utils.h"
rdma/res.c +261 -1
287 lines not on this page · raw message
rdma/utils.c +5 -0
@@ -375,6 +375,11 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_RES_STATE] = MNL_TYPE_U8, [RDMA_NLDEV_ATTR_RES_PID] = MNL_TYPE_U32, [RDMA_NLDEV_ATTR_RES_KERN_NAME] = MNL_TYPE_NUL_STRING, + [RDMA_NLDEV_ATTR_RES_CM_ID] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_CM_ID_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_PS] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_SRC_ADDR] = MNL_TYPE_UNSPEC, + [RDMA_NLDEV_ATTR_RES_DST_ADDR] = MNL_TYPE_UNSPEC, };int rd_attr_cb (const struct nlattr *attr,void *data)
Signature · 1 line
1.8.3.1
PATCH v4 5/7 rdma: Add CQ resource tracking information
In reply to Steve WiseSample output:
# rdma resource show cq dev cxgb4_0 cqe 46 users 2 pid 30503 comm rping dev cxgb4_0 cqe 46 users 2 pid 30498 comm rping dev mlx4_0 cqe 63 users 2 pid 30494 comm rping dev mlx4_0 cqe 63 users 2 pid 30489 comm rping dev mlx4_0 cqe 1023 users 2 poll_ctx WORKQUEUE pid 0 comm [ib_core]
# rdma resource show cq pid 30489 dev mlx4_0 cqe 63 users 2 pid 30489 comm rping
--- rdma/res.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 5 ++ 2 files changed, 154 insertions(+)
rdma/res.c +149 -0
@@ -21,6 +21,8 @@ static int res_help(struct rd *rd)pr_out (" resource show qp link [DEV/PORT] [FILTER-NAME FILTER-VALUE]\n" );pr_out (" resource show cm_id link [DEV/PORT]\n" );pr_out (" resource show cm_id link [DEV/PORT] [FILTER-NAME FILTER-VALUE]\n" ); +pr_out (" resource show cq link [DEV/PORT]\n" ); +pr_out (" resource show cq link [DEV/PORT] [FILTER-NAME FILTER-VALUE]\n" );return 0 ; }
@@ -265,6 +267,16 @@ static void print_comm(struct rd *rd, const char *str,pr_out ("comm %s " , tmp); } +static void print_dev (struct rd *rd,uint32_t idx,const char *name) +{ +if (rd->json_output) { +jsonw_uint_field (rd->jw,"ifindex" , idx); +jsonw_string_field (rd->jw,"ifname" , name); + }else { +pr_out ("dev %s " , name); + } +} +static void print_link (struct rd *rd,uint32_t idx,const char *name,uint32_t port,struct nlattr **nla_line) {
@@ -674,6 +686,132 @@ static int res_cm_id_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_OK ; } +static void print_cqe (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_uint_field (rd->jw,"cqe" , val); +else +pr_out ("cqe %u " , val); +} + +static void print_users (struct rd *rd,uint64_t val) +{ +if (rd->json_output) +jsonw_uint_field (rd->jw,"users" , val); +else +pr_out ("users %" PRIu64" " , val); +} + +static const char *poll_ctx_to_str (uint8_t idx) +{ +static const char *const cm_id_states_str[] = { +"DIRECT" ,"SOFTIRQ" ,"WORKQUEUE" }; + +if (idx <ARRAY_SIZE (cm_id_states_str)) +return cm_id_states_str[idx]; +return "UNKNOWN" ; +} + +static void print_poll_ctx (struct rd *rd,uint8_t poll_ctx) +{ +if (rd->json_output) { +jsonw_string_field (rd->jw,"poll-ctx" , +poll_ctx_to_str (poll_ctx)); +return ; + } +pr_out ("poll-ctx %s " ,poll_ctx_to_str (poll_ctx)); +} + +static int res_cq_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +struct nlattr *nla_table, *nla_entry; +struct rd *rd = data; +const char *name; +uint32_t idx; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || + !tb[RDMA_NLDEV_ATTR_DEV_NAME ] || + !tb[RDMA_NLDEV_ATTR_RES_CQ ]) +return MNL_CB_ERROR ; + + name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); + nla_table = tb[RDMA_NLDEV_ATTR_RES_CQ ]; + +mnl_attr_for_each_nested (nla_entry, nla_table) { +struct nlattr *nla_line[RDMA_NLDEV_ATTR_MAX ] = {}; +char *comm =NULL ; +uint32_t pid =0 ; +uint8_t poll_ctx =0 ; +uint64_t users; +uint32_t cqe; +int err; + + err =mnl_attr_parse_nested (nla_entry, rd_attr_cb, nla_line); +if (err !=MNL_CB_OK ) +return MNL_CB_ERROR ; + +if (!nla_line[RDMA_NLDEV_ATTR_RES_CQE ] || + !nla_line[RDMA_NLDEV_ATTR_RES_USECNT ] || + (!nla_line[RDMA_NLDEV_ATTR_RES_PID ] && + !nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ])) { +return MNL_CB_ERROR ; + } + + cqe =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_CQE ]); + + users =mnl_attr_get_u64 (nla_line[RDMA_NLDEV_ATTR_RES_USECNT ]); +if (rd_check_is_filtered (rd,"users" , users)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_POLL_CTX ]) { + poll_ctx =mnl_attr_get_u8 ( + nla_line[RDMA_NLDEV_ATTR_RES_POLL_CTX ]); +if (rd_check_is_string_filtered (rd,"poll-ctx" , +poll_ctx_to_str (poll_ctx))) +continue ; + } + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) { + pid =mnl_attr_get_u32 ( + nla_line[RDMA_NLDEV_ATTR_RES_PID ]); + comm =get_task_name (pid); + } + +if (rd_check_is_filtered (rd,"pid" , pid)) { +free (comm); +continue ; + } + +if (nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]) +/* discard const from mnl_attr_get_str */ + comm = (char *)mnl_attr_get_str ( + nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]); + +if (rd->json_output) +jsonw_start_array (rd->jw); + +print_dev (rd, idx, name); +print_cqe (rd, cqe); +print_users (rd, users); +if (nla_line[RDMA_NLDEV_ATTR_RES_POLL_CTX ]) +print_poll_ctx (rd, poll_ctx); +print_pid (rd, pid); +print_comm (rd, comm, nla_line); + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) +free (comm); + +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); + } +return MNL_CB_OK ; +} +RES_FUNC (res_no_args,RDMA_NLDEV_CMD_RES_GET ,NULL , true); static const struct
@@ -716,12 +854,23 @@ struct filters cm_id_valid_filters[MAX_NUMBER_OF_FILTERS] = {RES_FUNC (res_cm_id,RDMA_NLDEV_CMD_RES_CM_ID_GET , cm_id_valid_filters, false); +static const +struct filters cq_valid_filters[MAX_NUMBER_OF_FILTERS ] = { + { .name ="dev" , .is_number = false }, + { .name ="users" , .is_number = true }, + { .name ="poll-ctx" , .is_number = false }, + { .name ="pid" , .is_number = true } +}; + +RES_FUNC (res_cq,RDMA_NLDEV_CMD_RES_CQ_GET , cq_valid_filters, true); +static int res_show (struct rd *rd) {const struct rd_cmd cmds[] = { {NULL , res_no_args }, {"qp" , res_qp }, {"cm_id" , res_cm_id }, + {"cq" , res_cq }, {0 } };
rdma/utils.c +5 -0
@@ -380,6 +380,11 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_RES_PS] = MNL_TYPE_U32, [RDMA_NLDEV_ATTR_RES_SRC_ADDR] = MNL_TYPE_UNSPEC, [RDMA_NLDEV_ATTR_RES_DST_ADDR] = MNL_TYPE_UNSPEC, + [RDMA_NLDEV_ATTR_RES_CQ] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_CQ_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_CQE] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_USECNT] = MNL_TYPE_U64, + [RDMA_NLDEV_ATTR_RES_POLL_CTX] = MNL_TYPE_U8, };int rd_attr_cb (const struct nlattr *attr,void *data)
Signature · 1 line
1.8.3.1
PATCH v4 6/7 rdma: Add MR resource tracking information
In reply to Steve WiseSample output:
Without CAP_NET_ADMIN:
$ rdma resource show mr mrlen 65536 dev mlx4_0 mrlen 65536 pid 0 comm [nvme_rdma] dev cxgb4_0 mrlen 65536 pid 0 comm [nvme_rdma]
With CAP_NET_ADMIN:
# rdma resource show mr mrlen 65536 dev mlx4_0 rkey 0x12702 lkey 0x12702 iova 0x85724a000 mrlen 65536 pid 0 comm [nvme_rdma] dev cxgb4_0 rkey 0x68fe4e9 lkey 0x68fe4e9 iova 0x835b91000 mrlen 65536 pid 0 comm [nvme_rdma]
--- include/json_writer.h | 2 + lib/json_writer.c | 11 +++++ rdma/res.c | 127 ++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 6 +++ 4 files changed, 146 insertions(+)
include/json_writer.h +2 -0
@@ -35,6 +35,7 @@ void jsonw_bool(json_writer_t *self, bool value);void jsonw_float (json_writer_t *self,double number);void jsonw_float_fmt (json_writer_t *self,const char *fmt,double num);void jsonw_uint (json_writer_t *self,uint64_t number); +void jsonw_xint (json_writer_t *self,uint64_t number);void jsonw_hu (json_writer_t *self,unsigned short number);void jsonw_int (json_writer_t *self,int64_t number);void jsonw_null (json_writer_t *self);
@@ -45,6 +46,7 @@ void jsonw_string_field(json_writer_t *self, const char *prop, const char *val);void jsonw_bool_field (json_writer_t *self,const char *prop,bool value);void jsonw_float_field (json_writer_t *self,const char *prop,double num);void jsonw_uint_field (json_writer_t *self,const char *prop,uint64_t num); +void jsonw_xint_field (json_writer_t *self,const char *prop,uint64_t num);void jsonw_hu_field (json_writer_t *self,const char *prop,unsigned short num);void jsonw_int_field (json_writer_t *self,const char *prop,int64_t num);void jsonw_null_field (json_writer_t *self,const char *prop);
lib/json_writer.c +11 -0
@@ -225,6 +225,11 @@ void jsonw_uint(json_writer_t *self, uint64_t num)jsonw_printf (self,"%" PRIu64, num); } +void jsonw_xint (json_writer_t *self,uint64_t num) +{ +jsonw_printf (self,"%" PRIx64, num); +} +void jsonw_lluint (json_writer_t *self,unsigned long long int num) {jsonw_printf (self,"%llu" , num);
@@ -269,6 +274,12 @@ void jsonw_uint_field(json_writer_t *self, const char *prop, uint64_t num)jsonw_uint (self, num); } +void jsonw_xint_field (json_writer_t *self,const char *prop,uint64_t num) +{ +jsonw_name (self, prop); +jsonw_xint (self, num); +} +void jsonw_hu_field (json_writer_t *self,const char *prop,unsigned short num) {jsonw_name (self, prop);
rdma/res.c +127 -0
145 lines not on this page · raw message
rdma/utils.c +6 -0
@@ -385,6 +385,12 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_RES_CQE] = MNL_TYPE_U32, [RDMA_NLDEV_ATTR_RES_USECNT] = MNL_TYPE_U64, [RDMA_NLDEV_ATTR_RES_POLL_CTX] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_RES_MR] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_MR_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_RKEY] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_LKEY] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_IOVA] = MNL_TYPE_U64, + [RDMA_NLDEV_ATTR_RES_MRLEN] = MNL_TYPE_U64, };int rd_attr_cb (const struct nlattr *attr,void *data)
Signature · 1 line
1.8.3.1
PATCH v4 7/7 rdma: Add PD resource tracking information
In reply to Steve WiseSample output:
Without CAP_NET_ADMIN capability:
dev mlx4_0 users 0 pid 0 comm [ib_srpt]
dev mlx4_0 users 0 pid 0 comm [ib_srp]
dev mlx4_0 users 1 pid 0 comm [ib_core]
dev cxgb4_0 users 0 pid 0 comm [ib_srp]
With CAP_NET_ADMIN capability:
dev mlx4_0 local_dma_lkey 0x8000 users 0 pid 0 comm [ib_srpt] dev mlx4_0 local_dma_lkey 0x8000 users 0 pid 0 comm [ib_srp] dev mlx4_0 local_dma_lkey 0x8000 users 1 pid 0 comm [ib_core] dev cxgb4_0 local_dma_lkey 0x0 users 0 pid 0 comm [ib_srp]
--- rdma/res.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
rdma/res.c +95 -0
@@ -927,6 +927,91 @@ static int res_mr_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_OK ; } +static int res_pd_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +struct nlattr *nla_table, *nla_entry; +struct rd *rd = data; +const char *name; +uint32_t idx; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || + !tb[RDMA_NLDEV_ATTR_DEV_NAME ] || + !tb[RDMA_NLDEV_ATTR_RES_PD ]) +return MNL_CB_ERROR ; + + name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); + nla_table = tb[RDMA_NLDEV_ATTR_RES_PD ]; + +mnl_attr_for_each_nested (nla_entry, nla_table) { +uint32_t local_dma_lkey =0 , unsafe_global_rkey =0 ; +struct nlattr *nla_line[RDMA_NLDEV_ATTR_MAX ] = {}; +char *comm =NULL ; +uint32_t pid =0 ; +uint64_t users; +int err; + + err =mnl_attr_parse_nested (nla_entry, rd_attr_cb, nla_line); +if (err !=MNL_CB_OK ) +return MNL_CB_ERROR ; + +if (!nla_line[RDMA_NLDEV_ATTR_RES_USECNT ] || + (!nla_line[RDMA_NLDEV_ATTR_RES_PID ] && + !nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ])) { +return MNL_CB_ERROR ; + } + +if (nla_line[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ]) + local_dma_lkey =mnl_attr_get_u32 ( + nla_line[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ]); + + users =mnl_attr_get_u64 (nla_line[RDMA_NLDEV_ATTR_RES_USECNT ]); +if (rd_check_is_filtered (rd,"users" , users)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ]) + unsafe_global_rkey =mnl_attr_get_u32 ( + nla_line[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ]); + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) { + pid =mnl_attr_get_u32 ( + nla_line[RDMA_NLDEV_ATTR_RES_PID ]); + comm =get_task_name (pid); + } + +if (rd_check_is_filtered (rd,"pid" , pid)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]) +/* discard const from mnl_attr_get_str */ + comm = (char *)mnl_attr_get_str ( + nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]); + +if (rd->json_output) +jsonw_start_array (rd->jw); + +print_dev (rd, idx, name); +if (nla_line[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ]) +print_key (rd,"local_dma_lkey" , local_dma_lkey); +print_users (rd, users); +if (nla_line[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ]) +print_key (rd,"unsafe_global_rkey" , unsafe_global_rkey); +print_pid (rd, pid); +print_comm (rd, comm, nla_line); + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) +free (comm); + +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); + } +return MNL_CB_OK ; +} +RES_FUNC (res_no_args,RDMA_NLDEV_CMD_RES_GET ,NULL , true); static const struct
@@ -990,6 +1075,15 @@ struct filters mr_valid_filters[MAX_NUMBER_OF_FILTERS] = {RES_FUNC (res_mr,RDMA_NLDEV_CMD_RES_MR_GET , mr_valid_filters, true); +static const +struct filters pd_valid_filters[MAX_NUMBER_OF_FILTERS ] = { + { .name ="dev" , .is_number = false }, + { .name ="users" , .is_number = true }, + { .name ="pid" , .is_number = true } +}; + +RES_FUNC (res_pd,RDMA_NLDEV_CMD_RES_PD_GET , pd_valid_filters, true); +static int res_show (struct rd *rd) {const struct rd_cmd cmds[] = {
@@ -998,6 +1092,7 @@ static int res_show(struct rd *rd) { "cm_id", res_cm_id }, { "cq", res_cq }, { "mr", res_mr }, + { "pd", res_pd }, { 0 } };
Signature · 1 line
1.8.3.1
David Ahern
AppliedQuoted 33 lines · Steve Wise
This series enhances the iproute2 rdma tool to include dumping of connection manager id (
cm_id), completion queue (cq), memory region (mr), and protection domain (pd) rdma resources. It is the user-space part of the kernel resource tracking series merged into rdma-next for 4.17 [1] and [2].Changes since v3:
- replacedrdma_cma.hinclusion with UAPIrdma_user_cm.h
- display only device names instead of device/port for cq, mr, and pd since they are not associated with a specific port.Changes since v2:
- pull in rdma-core:include/rdma/rdma_cma.h
- 80 column reformat
- add reviewed-by tagsChanges since v1/RFC:
- removed RFC tag
- initialize rd properly to avoid passing a garbage port number
- revert accidental change toqp_valid_filters
- removedcm_iddev/network/transport types
-cm_idip addrs now passed up as__kernel_sockaddr_storage
-cm_idip address ports printed as "address:port" strings
- only parse/display memory keys and iova if available
- filter on "users" for cqs and pds
- fixed memory leaks
- removedPD_FLAGSattribute
- filter on "mrlen" for mrs
- filter on "poll-ctx" for cqs
- don't require addrs orqp_typefor parsingcm_ids
- only filter optional attrs if they are present
- remove PGSIZE MR attr to match kernel
Quoted 6 lines
[1] https://www.spinics.net/lists/linux-rdma/msg61720.html
[2] https://www.spinics.net/lists/linux-rdma/msg62979.htmlhttps://www.spinics.net/lists/linux-rdma/msg62980.html---
applied to iproute2-next. Thanks,
Quoted 34 lines · David Ahern
Quoted 33 lines · Steve Wise
This series enhances the iproute2 rdma tool to include dumping of connection manager id (
cm_id), completion queue (cq), memory region (mr), and protection domain (pd) rdma resources. It is the user-space part of the kernel resource tracking series merged into rdma-next for 4.17 [1] and [2].Changes since v3:
- replacedrdma_cma.hinclusion with UAPIrdma_user_cm.h
- display only device names instead of device/port for cq, mr, and pd since they are not associated with a specific port.Changes since v2:
- pull in rdma-core:include/rdma/rdma_cma.h
- 80 column reformat
- add reviewed-by tagsChanges since v1/RFC:
- removed RFC tag
- initialize rd properly to avoid passing a garbage port number
- revert accidental change toqp_valid_filters
- removedcm_iddev/network/transport types
-cm_idip addrs now passed up as__kernel_sockaddr_storage
-cm_idip address ports printed as "address:port" strings
- only parse/display memory keys and iova if available
- filter on "users" for cqs and pds
- fixed memory leaks
- removedPD_FLAGSattribute
- filter on "mrlen" for mrs
- filter on "poll-ctx" for cqs
- don't require addrs orqp_typefor parsingcm_ids
- only filter optional attrs if they are present
- remove PGSIZE MR attr to match kernel
Quoted 8 lines
Quoted 6 lines
[1] https://www.spinics.net/lists/linux-rdma/msg61720.html
[2] https://www.spinics.net/lists/linux-rdma/msg62979.htmlhttps://www.spinics.net/lists/linux-rdma/msg62980.html---
applied to iproute2-next. Thanks,
Thanks David
Quoted 1 line
Quoted 7 lines · David Ahern
Quoted 3 lines · Steve Wise
Quoted 2 lines
This series enhances the iproute2 rdma tool to include dumping of connection manager id (
cm_id), completion queue (cq), memory region(mr),
Quoted 1 line
Quoted 1 line
and protection domain (pd) rdma resources. It is the user-space part
of
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
the kernel resource tracking series merged into rdma-next for 4.17 [1] and [2].
Changes since v3:
- replacedrdma_cma.hinclusion with UAPIrdma_user_cm.h
- display only device names instead of device/port for cq, mr, and pd since they are not associated with a specific port.Changes since v2:
- pull in rdma-core:include/rdma/rdma_cma.h
- 80 column reformat
- add reviewed-by tagsChanges since v1/RFC:
- removed RFC tag
- initialize rd properly to avoid passing a garbage port number
- revert accidental change toqp_valid_filters
- removedcm_iddev/network/transport types
-cm_idip addrs now passed up as__kernel_sockaddr_storage
-cm_idip address ports printed as "address:port" strings
- only parse/display memory keys and iova if available
- filter on "users" for cqs and pds
- fixed memory leaks
- removedPD_FLAGSattribute
- filter on "mrlen" for mrs
- filter on "poll-ctx" for cqs
- don't require addrs orqp_typefor parsingcm_ids
- only filter optional attrs if they are present
- remove PGSIZE MR attr to match kernel[1] https://www.spinics.net/lists/linux-rdma/msg61720.html
[2] https://www.spinics.net/lists/linux-rdma/msg62979.htmlhttps://www.spinics.net/lists/linux-rdma/msg62980.html
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
---
applied to iproute2-next. Thanks,
Thanks David
Yes, Thanks David and Leon.