RDMA resource tracking
Subsystems (3)
v2 cover RDMA resource tracking
Changelog:
v1 -> v2;
* Added checks for all occurrences of strdup failures and added patch with fix of already merged code.
* Sync with latest kernel code.
* Rewrote table representation to be similar to "ip route" output.
* Implemented string filters.
* Removed curr/max representation from the summary output.
v0 -> v1:
* Fixed subject title in patch #1: rdam -> rdma.
* Added newline between variable declaration and the code.
* Add check to failure in strdup() call in rd_check_is_string_filtered().
* Rewrote res_qp_parse_cb() to avoid long lines and extra indentation.
------------------------------------------------------------------------
David, Stephen,
The kernel code is not accepted yet, but the interface and functionality of iproute2 stable enough and ready for review.
iproute2 part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git/log/?h=topic/resource-tracking-3
kernel part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=rn/restrack-v5
Thanks
------------------------------------------------------------------------
Hi,
This is supplementary (user-space) part of RDMA resource tracking posted to the RDMA mailing list for the review [1].
The main goal of this new functionality in RDMAtool is to provide debug visibility of running applications in RDMA stack.
The current series adds new command object (resource) which provides short summary if it is called without arguments or more detailed information while it is called with request to present QPs.
1) Summary information:
$ rdma res 1: mlx5_0: pd 3 cq 5 qp 4 2: mlx5_1: pd 3 cq 5 qp 4 3: mlx5_2: pd 3 cq 5 qp 4 4: mlx5_3: pd 2 cq 3 qp 2 5: mlx5_4: pd 3 cq 5 qp 4
2) Summary information of specific device:
$ rdma res show mlx5_4 5: mlx5_4: pd 3 cq 5 qp 44
3) Get all QPs for the specific device:
$ rdma res show qp link mlx5_4 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
$ rdma res show qp link mlx5_4/ link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
4) Provide illegal port number (0 is illegal):
$ rdma res show qp link mlx5_4/0 Wrong device name
5) Get QPs of specific port:
$ rdma res show qp link mlx5_4/1 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
6) Get QPs which have not assigned port yet:
$ rdma res show qp link mlx5_3/- link mlx5_3/- lqpn 8 type UD state RTS sq-psn 0 pid 0 comm [ipoib-verbs]
7) Limit to specific Local QPNs:
$ rdma res show qp link mlx5_4/1 lqpn 1-3,7 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad]
8) Filter types (strings):
$ rdma res show qp link mlx5_4/1 type UD,gSi link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad]
Thanks
[1] https://www.spinics.net/lists/linux-rdma/msg59535.html
Leon Romanovsky (10):
rdma: Add option to provide "-" sign for the port number
rdma: Make visible the number of arguments
rdma: Add filtering infrastructure
rdma: Set pointer to device name position
rdma: Allow external usage of compare string routine
rdma: Update kernel header file
rdma: Add resource tracking summary
rdma: Add QP resource tracking information
rdma: Document resource tracking
rdma: Check return value of strdup call
include/uapi/rdma/rdma_netlink.h | 61 ++++- man/man8/rdma-resource.8 | 86 +++++++ rdma/Makefile | 2 +- rdma/link.c | 2 +- rdma/rdma.c | 4 +- rdma/rdma.h | 28 ++- rdma/res.c | 483 +++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 321 +++++++++++++++++++++++++- 8 files changed, 969 insertions(+), 18 deletions(-) create mode 100644 man/man8/rdma-resource.8 create mode 100644 rdma/res.c
PATCH v2 1/10 rdma: Add option to provide "-" sign for the port number
From: Leon Romanovsky <[email protected]>
According to the IBTA spec [1], the physical connected port is provided for the QP in RTR-to-INIT stage performed by modify_qp(). It causes to do not have port number for newly created QPs.
The following patch adds "-" sign to present absence of port, because QPs are going to be associated with rdmatool link object, which needs port number as an index.
[1] InfiniBand Architecture Release 1.3 -
"Table 96 QP State Transition Properties"
--- rdma/link.c | 2 +- rdma/rdma.h | 3 +-- rdma/utils.c | 50 ++++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 44 insertions(+), 11 deletions(-)
rdma/link.c +1 -1
@@ -285,7 +285,7 @@ static int link_one_show(struct rd *rd)static int link_show (struct rd *rd) { -return rd_exec_link (rd, link_one_show); +return rd_exec_link (rd, link_one_show, true); }int cmd_link (struct rd *rd)
rdma/rdma.h +1 -2
@@ -64,7 +64,6 @@ bool rd_no_arg(struct rd *rd);void rd_arg_inc (struct rd *rd);char *rd_argv (struct rd *rd); -uint32_t get_port_from_argv (struct rd *rd);/* * Commands interface
@@ -73,7 +72,7 @@ int cmd_dev(struct rd *rd);int cmd_link (struct rd *rd);int rd_exec_cmd (struct rd *rd,const struct rd_cmd *c,const char *str);int rd_exec_dev (struct rd *rd,int (*cb)(struct rd *rd)); -int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd)); +int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd),bool strict_port);void rd_free (struct rd *rd);/*
rdma/utils.c +42 -8
96 lines not on this page · raw message
Signature · 1 line
2.15.1
PATCH v2 2/10 rdma: Make visible the number of arguments
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
--- rdma/rdma.h | 1 + rdma/utils.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
rdma/rdma.h +1 -0
@@ -74,6 +74,7 @@ int rd_exec_cmd(struct rd *rd, const struct rd_cmd *c, const char *str);int rd_exec_dev (struct rd *rd,int (*cb)(struct rd *rd));int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd),bool strict_port);void rd_free (struct rd *rd); +int rd_argc (struct rd *rd);/* * Device manipulation
rdma/utils.c +1 -1
@@ -12,7 +12,7 @@#include "rdma.h" #include <ctype.h> -static int rd_argc (struct rd *rd) +int rd_argc (struct rd *rd) {return rd->argc; }
Signature · 1 line
2.15.1
PATCH v2 3/10 rdma: Add filtering infrastructure
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
This patch adds general infrastructure to RDMAtool to handle various filtering options needed for the downstream resource tracking patches.
The infrastructure is generic and stores filters in list of key<->value entries. There are three types of filters:
1. Numeric - the values are intended to be digits combined with '-' to mark range and ',' to mark multiple entries, e.g. pid 1-100,234,400-401 is perfectly legit filter to limit process ids.
2. String - the values are consist from strings and "," as a denominator.
3. Link - special case to allow '/' in string to provide link name, e.g. link mlx4_1/2.
--- rdma/rdma.c | 1 + rdma/rdma.h | 20 ++++++ rdma/utils.c | 229 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 250 insertions(+)
rdma/rdma.c +1 -0
@@ -47,6 +47,7 @@ static int rd_init(struct rd *rd, int argc, char **argv, char *filename) rd->argc = argc; rd->argv = argv;INIT_LIST_HEAD (&rd->dev_map_list); +INIT_LIST_HEAD (&rd->filter_list);if (rd->json_output) { rd->jw =jsonw_new (stdout);
rdma/rdma.h +20 -0
@@ -29,6 +29,18 @@#define RDMA_BITMAP_ENUM (name, bit_no) RDMA_BITMAP_##name = BIT(bit_no),#define RDMA_BITMAP_NAMES (name, bit_no) [bit_no] = #name, +#define MAX_NUMBER_OF_FILTERS 64 +struct filters { +char name[32 ]; +bool is_number; +}; + +struct filter_entry { +struct list_head list; +char *key; +char *value; +}; +struct dev_map {struct list_head list;char *dev_name;
@@ -50,6 +62,7 @@ struct rd {json_writer_t *jw;bool json_output;bool pretty_output; +struct list_head filter_list; };struct rd_cmd {
@@ -81,6 +94,13 @@ int rd_argc(struct rd *rd);*/ struct dev_map *dev_map_lookup (struct rd *rd,bool allow_port_index); +/* +* Filter manipulation +*/ +int rd_build_filter (struct rd *rd,const struct filters valid_filters[]); +bool rd_check_is_filtered (struct rd *rd,const char *key,uint32_t val); +bool rd_check_is_string_filtered (struct rd *rd,const char *key,const char *val); +bool rd_check_is_key_exist (struct rd *rd,const char *key);/* * Netlink */
rdma/utils.c +229 -0
241 lines not on this page · raw message
PATCH v2 4/10 rdma: Set pointer to device name position
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The dev and link execution callbacks expects that next command line argument is device or port name.
Set pointer to device or port name position prior calls to rd_exec_dev()/rd_exec_link().
--- rdma/rdma.h | 1 + rdma/utils.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+)
rdma/rdma.h +1 -0
@@ -87,6 +87,7 @@ int rd_exec_cmd(struct rd *rd, const struct rd_cmd *c, const char *str);int rd_exec_dev (struct rd *rd,int (*cb)(struct rd *rd));int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd),bool strict_port);void rd_free (struct rd *rd); +int rd_set_arg_to_devname (struct rd *rd);int rd_argc (struct rd *rd);/*
rdma/utils.c +19 -0
@@ -412,6 +412,25 @@ void rd_free(struct rd *rd)filters_cleanup (rd); } +int rd_set_arg_to_devname (struct rd *rd) +{ +int ret =0 ; + +while (!rd_no_arg (rd)) { +if (rd_argv_match (rd,"dev" ) ||rd_argv_match (rd,"link" )) { +rd_arg_inc (rd); +if (rd_no_arg (rd)) { +pr_err ("No device name was supplied\n" ); + ret = -EINVAL ; + } + goto out; + } +rd_arg_inc (rd); + } +out: +return ret; +} +int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd),bool strict_port) {struct dev_map *dev_map;
Signature · 1 line
2.15.1
PATCH v2 5/10 rdma: Allow external usage of compare string routine
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
--- rdma/rdma.h | 2 ++ rdma/utils.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
rdma/rdma.h +2 -0
@@ -90,6 +90,8 @@ void rd_free(struct rd *rd);int rd_set_arg_to_devname (struct rd *rd);int rd_argc (struct rd *rd); +int strcmpx (const char *str1,const char *str2); +/* * Device manipulation */
rdma/utils.c +1 -1
@@ -24,7 +24,7 @@ char *rd_argv(struct rd *rd)return *rd->argv; } -static int strcmpx (const char *str1,const char *str2) +int strcmpx (const char *str1,const char *str2) {if (strlen (str1) >strlen (str2))return -1 ;
Signature · 1 line
2.15.1
PATCH v2 6/10 rdma: Update kernel header file
In reply to a message not in the archiveFrom: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Synchronize iporute2 package with latest kernel
RDMA netlink header file.
--- include/uapi/rdma/rdma_netlink.h | 61 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 3 deletions(-)
include/uapi/rdma/rdma_netlink.h +58 -3
@@ -1,6 +1,6 @@/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _RDMA_NETLINK_H -#define _RDMA_NETLINK_H +#ifndef _UAPI_RDMA_NETLINK_H +#define _UAPI_RDMA_NETLINK_H#include <linux/types.h>
@@ -236,6 +236,16 @@ enum rdma_nldev_command {RDMA_NLDEV_CMD_PORT_NEW ,RDMA_NLDEV_CMD_PORT_DEL , +RDMA_NLDEV_CMD_RES_GET ,/* can dump */ +RDMA_NLDEV_CMD_RES_SET , +RDMA_NLDEV_CMD_RES_NEW , +RDMA_NLDEV_CMD_RES_DEL , + +RDMA_NLDEV_CMD_RES_QP_GET ,/* can dump */ +RDMA_NLDEV_CMD_RES_QP_SET , +RDMA_NLDEV_CMD_RES_QP_NEW , +RDMA_NLDEV_CMD_RES_QP_DEL , +RDMA_NLDEV_NUM_OPS };
@@ -303,6 +313,51 @@ enum rdma_nldev_attr {RDMA_NLDEV_ATTR_DEV_NODE_TYPE ,/* u8 */ +RDMA_NLDEV_ATTR_RES_SUMMARY ,/* nested table */ +RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_NAME ,/* string */ +RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR ,/* u64 */ + +RDMA_NLDEV_ATTR_RES_QP ,/* nested table */ +RDMA_NLDEV_ATTR_RES_QP_ENTRY ,/* nested table */ +/* +* Local QPN +*/ +RDMA_NLDEV_ATTR_RES_LQPN ,/* u32 */ +/* +* Remote QPN, +* Applicable for RC and UC only IBTA 11.2.5.3 QUERY QUEUE PAIR +*/ +RDMA_NLDEV_ATTR_RES_RQPN ,/* u32 */ +/* +* Receive Queue PSN, +* Applicable for RC and UC only 11.2.5.3 QUERY QUEUE PAIR +*/ +RDMA_NLDEV_ATTR_RES_RQ_PSN ,/* u32 */ +/* +* Send Queue PSN +*/ +RDMA_NLDEV_ATTR_RES_SQ_PSN ,/* u32 */ +RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE ,/* u8 */ +/* +* QP types as visible to RDMA/core, the reserved QPT +* are not exported through this interface. +*/ +RDMA_NLDEV_ATTR_RES_TYPE ,/* u8 */ +RDMA_NLDEV_ATTR_RES_STATE ,/* u8 */ +/* +* Process ID which created object, +* in case of kernel origin, PID won't exist. +*/ +RDMA_NLDEV_ATTR_RES_PID ,/* u32 */ +/* +* The name of process created following resource. +* It will exist only for kernel objects. +* For user created objects, the user is supposed +* to read /proc/PID/comm file. +*/ +RDMA_NLDEV_ATTR_RES_KERN_NAME ,/* string */ +RDMA_NLDEV_ATTR_MAX }; -#endif /* _RDMA_NETLINK_H */ +#endif /* _UAPI_RDMA_NETLINK_H */
Signature · 6 lines
2.15.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 7/10 rdma: Add resource tracking summary
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The global resource summary information. The object names, current utilization and maximum numbers are received as is from the kernel.
$ rdma res 1: mlx5_0: pd 3 cq 5 qp 4 2: mlx5_1: pd 3 cq 5 qp 4 3: mlx5_2: pd 3 cq 5 qp 4 4: mlx5_3: pd 2 cq 3 qp 2 5: mlx5_4: pd 3 cq 5 qp 4
$ rdma res show mlx5_4 5: mlx5_4: pd 3 cq 5 qp 44
--- rdma/Makefile | 2 +- rdma/rdma.c | 3 +- rdma/rdma.h | 1 + rdma/res.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 4 ++ 5 files changed, 166 insertions(+), 2 deletions(-) create mode 100644 rdma/res.c
rdma/Makefile +1 -1
@@ -3,7 +3,7 @@ include ../config.mk ifeq ($(HAVE_MNL),y) -RDMA_OBJ = rdma.o utils.o dev.o link.o +RDMA_OBJ = rdma.o utils.o dev.o link.o res.o TARGETS=rdma endif
rdma/rdma.c +2 -1
@@ -15,7 +15,7 @@static void help (char *name) {pr_out ("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n" -"where OBJECT := { dev | link | help }\n" +"where OBJECT := { dev | link | resource | help }\n" " OPTIONS := { -V[ersion] | -d[etails] | -j[son] | -p[retty]}\n" , name); }
@@ -32,6 +32,7 @@ static int rd_cmd(struct rd *rd) { "help", cmd_help }, { "dev", cmd_dev }, { "link", cmd_link }, + { "resource", cmd_res }, { 0 } };
rdma/rdma.h +1 -0
@@ -83,6 +83,7 @@ char *rd_argv(struct rd *rd);*/ int cmd_dev (struct rd *rd);int cmd_link (struct rd *rd); +int cmd_res (struct rd *rd);int rd_exec_cmd (struct rd *rd,const struct rd_cmd *c,const char *str);int rd_exec_dev (struct rd *rd,int (*cb)(struct rd *rd));int rd_exec_link (struct rd *rd,int (*cb)(struct rd *rd),bool strict_port);
rdma/res.c new file +158 -0
158 lines not on this page · raw message
rdma/utils.c +4 -0
@@ -356,6 +356,10 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_PORT_STATE] = MNL_TYPE_U8, [RDMA_NLDEV_ATTR_PORT_PHYS_STATE] = MNL_TYPE_U8, [RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_RES_SUMMARY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_NAME] = MNL_TYPE_NUL_STRING, + [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR] = MNL_TYPE_U64, };int rd_attr_cb (const struct nlattr *attr,void *data)
Signature · 1 line
2.15.1
PATCH v2 8/10 rdma: Add QP resource tracking information
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
This patch adds ss-similar interface to view various resource tracked objects. At this stage, only QP is presented.
1. Get all QPs for the specific device:
$ rdma res show qp link mlx5_4 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
$ rdma res show qp link mlx5_4/ link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
2. Provide illegal port number (0 is illegal):
$ rdma res show qp link mlx5_4/0 Wrong device name
3. Get QPs of specific port:
$ rdma res show qp link mlx5_4/1 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad] link mlx5_4/1 lqpn 0 type SMI state RTS sq-psn 0 pid 0 comm [rdma-mad]
4. Get QPs which have not assigned port yet:
$ rdma res show qp link mlx5_3/- link mlx5_3/- lqpn 8 type UD state RTS sq-psn 0 pid 0 comm [ipoib-verbs]
5. Limit to specific Local QPNs:
$ rdma res show qp link mlx5_4/1 lqpn 1-3,7 link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad]
6. Filter types (strings):
$ rdma res show qp link mlx5_4/1 type UD,gSi link mlx5_4/1 lqpn 7 type UD state RTS sq-psn 0 pid 0 comm [mlx5-gsi] link mlx5_4/1 lqpn 1 type GSI state RTS sq-psn 0 pid 0 comm [rdma-mad]
--- rdma/res.c | 325 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 11 ++ 2 files changed, 336 insertions(+)
rdma/res.c +325 -0
@@ -16,6 +16,9 @@ static int res_help(struct rd *rd) {pr_out ("Usage: %s resource\n" , rd->filename);pr_out (" resource show [DEV]\n" ); +pr_out (" resource show [qp]\n" ); +pr_out (" resource show qp link [DEV/PORT]\n" ); +pr_out (" resource show qp link [DEV/PORT] [FILTER-NAME FILTER-VALUE]\n" );return 0 ; }
@@ -126,12 +129,334 @@ static int _res_send_msg(struct rd *rd, uint32_t command, mnl_cb_t callback) return rd_exec_link(rd, _##name, strict_port); \ } +static const char *path_mig_to_str (uint8_t idx) +{ +static const char *const path_mig_str[] = {"MIGRATED" , +"REARM" ,"ARMED" }; + +if (idx <ARRAY_SIZE (path_mig_str)) +return path_mig_str[idx]; +return "UNKNOWN" ; +} + +static const char *qp_states_to_str (uint8_t idx) +{ +static const char *const qp_states_str[] = {"RESET" ,"INIT" , +"RTR" ,"RTS" ,"SQD" , +"SQE" ,"ERR" }; + +if (idx <ARRAY_SIZE (qp_states_str)) +return qp_states_str[idx]; +return "UNKNOWN" ; +} + +static const char *qp_types_to_str (uint8_t idx) +{ +static const char *const qp_types_str[] = {"SMI" ,"GSI" ,"RC" , +"UC" ,"UD" ,"RAW_IPV6" , +"RAW_ETHERTYPE" , +"UNKNOWN" ,"RAW_PACKET" , +"XRC_INI" ,"XRC_TGT" }; + +if (idx <ARRAY_SIZE (qp_types_str)) +return qp_types_str[idx]; +return "UNKNOWN" ; +} + +static void print_lqpn (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_uint_field (rd->jw,"lqpn" , val); +else +pr_out ("lqpn %u " , val); +} + +static void print_rqpn (struct rd *rd,uint32_t val,struct nlattr **nla_line) +{ +if (!nla_line[RDMA_NLDEV_ATTR_RES_RQPN ]) +return ; + +if (rd->json_output) +jsonw_uint_field (rd->jw,"rqpn" , val); +else +pr_out ("rqpn %u " , val); +} + +static void print_type (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_string_field (rd->jw,"type" , +qp_types_to_str (val)); +else +pr_out ("type %s " ,qp_types_to_str (val)); +} + +static void print_state (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_string_field (rd->jw,"state" , +qp_states_to_str (val)); +else +pr_out ("state %s " ,qp_states_to_str (val)); +} + +static void print_rqpsn (struct rd *rd,uint32_t val,struct nlattr **nla_line) +{ +if (!nla_line[RDMA_NLDEV_ATTR_RES_RQ_PSN ]) +return ; + +if (rd->json_output) +jsonw_uint_field (rd->jw,"rq-psn" , val); +else +pr_out ("rq-psn %u " , val); +} + +static void print_sqpsn (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_uint_field (rd->jw,"sq-psn" , val); +else +pr_out ("sq-psn %u " , val); +} + +static void print_pathmig (struct rd *rd,uint32_t val, +struct nlattr **nla_line) +{ +if (!nla_line[RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE ]) +return ; + +if (rd->json_output) +jsonw_string_field (rd->jw, +"path-mig-state" , +path_mig_to_str (val)); +else +pr_out ("path-mig-state %s " ,path_mig_to_str (val)); +} + +static void print_pid (struct rd *rd,uint32_t val) +{ +if (rd->json_output) +jsonw_uint_field (rd->jw,"pid" , val); +else +pr_out ("pid %u " , val); +} + +static void print_comm (struct rd *rd,const char *str, +struct nlattr **nla_line) +{ +char tmp[18 ]; + +if (rd->json_output) { +/* Don't beatify output in JSON format */ +jsonw_string_field (rd->jw,"comm" , str); +return ; + } + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) +snprintf (tmp,sizeof (tmp),"%s" , str); +else +snprintf (tmp,sizeof (tmp),"[%s]" , str); + +pr_out ("comm %s " , tmp); +} + +static void print_link (struct rd *rd,uint32_t idx,const char *name, +uint32_t port,struct nlattr **nla_line) +{ +if (rd->json_output) { +jsonw_uint_field (rd->jw,"ifindex" , idx); + +if (nla_line[RDMA_NLDEV_ATTR_PORT_INDEX ]) +jsonw_uint_field (rd->jw,"port" , port); + +jsonw_string_field (rd->jw,"ifname" , name); + }else { +if (nla_line[RDMA_NLDEV_ATTR_PORT_INDEX ]) +pr_out ("link %s/%u " , name, port); +else +pr_out ("link %s/- " , name); + } +} + +static char *get_task_name (uint32_t pid) +{ +char *comm; +FILE *f; + +if (asprintf (&comm,"/proc/%d/comm" , pid) <0 ) +return NULL ; + + f =fopen (comm,"r" ); +free (comm); +if (!f) +return NULL ; + +if (fscanf (f,"%ms\n" , &comm) !=1 ) + comm =NULL ; + +fclose (f); + +return comm; +} + +static int res_qp_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +struct nlattr *nla_table, *nla_entry; +struct rd *rd = data; +const char *name; +uint32_t idx; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || + !tb[RDMA_NLDEV_ATTR_DEV_NAME ] || + !tb[RDMA_NLDEV_ATTR_RES_QP ]) +return MNL_CB_ERROR ; + + name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); + nla_table = tb[RDMA_NLDEV_ATTR_RES_QP ]; + +mnl_attr_for_each_nested (nla_entry, nla_table) { +struct nlattr *nla_line[RDMA_NLDEV_ATTR_MAX ] = {}; +uint32_t lqpn, rqpn =0 , rq_psn =0 , sq_psn; +uint8_t type, state, path_mig_state =0 ; +uint32_t port =0 , pid =0 ; +char *comm =NULL ; +int err; + + err =mnl_attr_parse_nested (nla_entry, rd_attr_cb, nla_line); +if (err !=MNL_CB_OK ) +return MNL_CB_ERROR ; + +if (!nla_line[RDMA_NLDEV_ATTR_RES_LQPN ] || + !nla_line[RDMA_NLDEV_ATTR_RES_SQ_PSN ] || + !nla_line[RDMA_NLDEV_ATTR_RES_TYPE ] || + !nla_line[RDMA_NLDEV_ATTR_RES_STATE ] || + (!nla_line[RDMA_NLDEV_ATTR_RES_PID ] && + !nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ])) { +return MNL_CB_ERROR ; + } + +if (nla_line[RDMA_NLDEV_ATTR_PORT_INDEX ]) + port =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_PORT_INDEX ]); + +if (port != rd->port_idx) +continue ; + + lqpn =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_LQPN ]); +if (rd_check_is_filtered (rd,"lqpn" , lqpn)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_RQPN ]) { + rqpn =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_RQPN ]); +if (rd_check_is_filtered (rd,"rqpn" , rqpn)) +continue ; + }else { +if (rd_check_is_key_exist (rd,"rqpn" )) +continue ; + } + +if (nla_line[RDMA_NLDEV_ATTR_RES_RQ_PSN ]) { + rq_psn =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_RQ_PSN ]); +if (rd_check_is_filtered (rd,"rq-psn" , rq_psn)) +continue ; + }else { +if (rd_check_is_key_exist (rd,"rq-psn" )) +continue ; + } + + sq_psn =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_SQ_PSN ]); +if (rd_check_is_filtered (rd,"sq-psn" , sq_psn)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE ]) { + path_mig_state =mnl_attr_get_u8 (nla_line[RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE ]); +if (rd_check_is_string_filtered (rd,"path-mig-state" ,path_mig_to_str (path_mig_state))) +continue ; + }else { +if (rd_check_is_key_exist (rd,"path-mig-state" )) +continue ; + } + + type =mnl_attr_get_u8 (nla_line[RDMA_NLDEV_ATTR_RES_TYPE ]); +if (rd_check_is_string_filtered (rd,"type" ,qp_types_to_str (type))) +continue ; + + state =mnl_attr_get_u8 (nla_line[RDMA_NLDEV_ATTR_RES_STATE ]); +if (rd_check_is_string_filtered (rd,"state" ,qp_states_to_str (state))) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) { + pid =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_PID ]); + comm =get_task_name (pid); + } + +if (rd_check_is_filtered (rd,"pid" , pid)) +continue ; + +if (nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]) +/* discard const from mnl_attr_get_str */ + comm = (char *)mnl_attr_get_str (nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME ]); + +if (rd->json_output) +jsonw_start_array (rd->jw); + +print_link (rd, idx, name, port, nla_line); + +print_lqpn (rd, lqpn); +print_rqpn (rd, rqpn, nla_line); + +print_type (rd, type); +print_state (rd, state); + +print_rqpsn (rd, rq_psn, nla_line); +print_sqpsn (rd, sq_psn); + +print_pathmig (rd, path_mig_state, nla_line); +print_pid (rd, pid); +print_comm (rd, comm, nla_line); + +if (nla_line[RDMA_NLDEV_ATTR_RES_PID ]) +free (comm); + +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); + } +return MNL_CB_OK ; +} +RES_FUNC (res_no_args,RDMA_NLDEV_CMD_RES_GET ,NULL , true); +static const struct +filters qp_valid_filters[MAX_NUMBER_OF_FILTERS ] = {{ .name ="link" , + .is_number = false }, + { .name ="lqpn" , + .is_number = true }, + { .name ="rqpn" , + .is_number = true }, + { .name ="pid" , + .is_number = true }, + { .name ="sq-psn" , + .is_number = true }, + { .name ="rq-psn" , + .is_number = true }, + { .name ="type" , + .is_number = false }, + { .name ="path-mig-state" , + .is_number = false }, + { .name ="state" , + .is_number = false } }; + +RES_FUNC (res_qp,RDMA_NLDEV_CMD_RES_QP_GET , qp_valid_filters, false); +static int res_show (struct rd *rd) {const struct rd_cmd cmds[] = { {NULL , res_no_args }, + {"qp" , res_qp }, {0 } };
rdma/utils.c +11 -0
@@ -360,6 +360,17 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY] = MNL_TYPE_NESTED, [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_NAME] = MNL_TYPE_NUL_STRING, [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR] = MNL_TYPE_U64, + [RDMA_NLDEV_ATTR_RES_QP] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_QP_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_RES_LQPN] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_RQPN] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_RQ_PSN] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_SQ_PSN] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_RES_TYPE] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_RES_STATE] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_RES_PID] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_RES_KERN_NAME] = MNL_TYPE_NUL_STRING, };int rd_attr_cb (const struct nlattr *attr,void *data)
Signature · 1 line
2.15.1
PATCH v2 9/10 rdma: Document resource tracking
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Spartan version of resource tracking documentation.
--- man/man8/rdma-resource.8 | 86 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 man/man8/rdma-resource.8
man/man8/rdma-resource.8 new file +86 -0
@@ -0,0 +1,86 @@ +.TH RDMA\-RESOURCE 8 "26 Dec 2017" "iproute2" "Linux" +.SH NAME +rdma-resource \- rdma resource configuration +.SH SYNOPSIS +.sp +.ad l +.in +8 +.ti -8 +.B rdma +.RI "[ " OPTIONS " ]" +.B resource +.RI " { " COMMAND " | " +.BR help " }" +.sp + +.ti -8 +.IR OPTIONS " := { " +\fB\-j\fR[\fIson\fR] | +\fB\-d\fR[\fIetails\fR] } + +.ti -8 +.B rdma resource show +.RI "[ " DEV/PORT_INDEX " ]" + +.ti -8 +.B rdma resource help + +.SH "DESCRIPTION" +.SS rdma resource show - display rdma resource tracking information + +.PP +.I "DEV/PORT_INDEX" +- specifies the RDMA link to show. +If this argument is omitted all links are listed. + +.SH "EXAMPLES" +.PP +rdma resource show +.RS 4 +Shows summary for all devices on the system. +.RE +.PP +rdma resource show mlx5_2 +.RS 4 +Shows the state of specified rdma device. +.RE +.PP +rdma res show qp link mlx5_4 +.RS 4 +Get all QPs for the specific device. +.RE +.PP +rdma res show qp link mlx5_4/1 +.RS 4 +Get QPs of specific port. +.RE +.PP +rdma res show qp link mlx5_4/0 +.RS 4 +Provide illegal port number (0 is illegal). +.RE +.PP +rdma res show qp link mlx5_4/- +.RS 4 +Get QPs which have not assigned port yet. +.RE +.PP +rdma res show qp link mlx5_4/- -d +.RS 4 +Detailed view. +.RE +.PP +rdma res show qp link mlx5_4/1 lqpn 0-6 +.RS 4 +Limit to specific Local QPNs. +.RE +.PP + +.SH SEE ALSO +.BR rdma (8), +.BR rdma-dev (8), +.BR rdma-link (8), +.br + +.SH AUTHOR +Leon Romanovsky <[email protected]>
Signature · 1 line
2.15.1
PATCH v2 10/10 rdma: Check return value of strdup call
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
--- rdma/utils.c | 4 ++++ 1 file changed, 4 insertions(+)
rdma/utils.c +4 -0
@@ -98,6 +98,10 @@ static struct dev_map *dev_map_alloc(const char *dev_name)if (!dev_map)return NULL ; dev_map->dev_name =strdup (dev_name); +if (!dev_map->dev_name) { +free (dev_map); +return NULL ; + }return dev_map; }
Signature · 1 line
2.15.1
Quoted 11 lines · Leon Romanovsky
David, Stephen,
The kernel code is not accepted yet, but the interface and functionality of iproute2 stable enough and ready for review.
iproute2 part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git/log/?h=topic/resource-tracking-3kernel part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=rn/restrack-v5
BTW, I just remembered rdma tool is using libmnl and it has the same problem devlink had -- old commands may not work as expected on newer kernels as more RDMA_NLDEV_ATTR_ are added.
Specifically, this check in rd_attr_cb:
if (mnl_attr_type_valid(attr, RDMA_NLDEV_ATTR_MAX) < 0) return MNL_CB_ERROR;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 13 lines · David Ahern
Quoted 11 lines · Leon Romanovsky
David, Stephen,
The kernel code is not accepted yet, but the interface and functionality of iproute2 stable enough and ready for review.
iproute2 part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git/log/?h=topic/resource-tracking-3kernel part:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=rn/restrack-v5
Quoted 7 lines
BTW, I just remembered rdma tool is using libmnl and it has the same problem devlink had -- old commands may not work as expected on newer kernels as more
RDMA_NLDEV_ATTR_are added.Specifically, this check in
rd_attr_cb:if (mnl_attr_type_valid(attr, RDMA_NLDEV_ATTR_MAX) < 0) return MNL_CB_ERROR;
Thanks, I'll take a look on the devlink implementation.