RDMA tool provider resource tracking
Subsystems (3)
cover RDMA tool provider resource tracking
Hello,
This series enhances the iproute2 rdma tool to include displaying provider-specific resource attributes. It is the user-space part of the kernel provider resource tracking series currently under review [1].
This is an RFC and should not be merged yet. Once [1] is in the linux-rdma for-next branch (and all reviewing is complete), I'll post a final version and request merging.
Thanks,
Steve.
[1] https://www.spinics.net/lists/linux-rdma/msg64013.html
Steve Wise (2):
rdma: update rdma_netlink.h to get provider attrs
rdma: print provider resource attributes
rdma/include/uapi/rdma/rdma_netlink.h | 37 ++++++- rdma/rdma.c | 7 +- rdma/rdma.h | 11 ++ rdma/res.c | 30 ++---- rdma/utils.c | 194 ++++++++++++++++++++++++++++++++++ 5 files changed, 257 insertions(+), 22 deletions(-)
Signature · 1 line
1.8.3.1
RFC PATCH 1/2 rdma: update rdma_netlink.h to get provider attrs
--- rdma/include/uapi/rdma/rdma_netlink.h | 37 ++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
rdma/include/uapi/rdma/rdma_netlink.h +36 -1
@@ -249,10 +249,22 @@ enum rdma_nldev_command {RDMA_NLDEV_NUM_OPS }; +enum { +RDMA_NLDEV_ATTR_ENTRY_STRLEN =16 , +}; + +enum rdma_nldev_print_type { +RDMA_NLDEV_PRINT_TYPE_UNSPEC , +RDMA_NLDEV_PRINT_TYPE_HEX , +}; +enum rdma_nldev_attr {/* don't change the order or add anything between, this is ABI! */ RDMA_NLDEV_ATTR_UNSPEC , +/* Pad attribute for 64b alignment */ +RDMA_NLDEV_ATTR_PAD =RDMA_NLDEV_ATTR_UNSPEC , +/* Identifier for ib_device */ RDMA_NLDEV_ATTR_DEV_INDEX ,/* u32 */
@@ -387,8 +399,31 @@ enum rdma_nldev_attr {RDMA_NLDEV_ATTR_RES_PD_ENTRY ,/* nested table */ RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ,/* u32 */ RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ,/* u32 */ +/* +* provider-specific attributes. +*/ +RDMA_NLDEV_ATTR_PROVIDER ,/* nested table */ +RDMA_NLDEV_ATTR_PROVIDER_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_PROVIDER_STRING ,/* string */ +/* +* u8 values from enum rdma_nldev_print_type +*/ +RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE ,/* u8 */ +RDMA_NLDEV_ATTR_PROVIDER_S32 ,/* s32 */ +RDMA_NLDEV_ATTR_PROVIDER_U32 ,/* u32 */ +RDMA_NLDEV_ATTR_PROVIDER_S64 ,/* s64 */ +RDMA_NLDEV_ATTR_PROVIDER_U64 ,/* u64 */ -/* Netdev information for relevant protocols, like RoCE and iWARP */ +/* +* Provides logical name and index of netdevice which is +* connected to physical port. This information is relevant +* for RoCE and iWARP. +* +* The netdevices which are associated with containers are +* supposed to be exported together with GID table once it +* will be exposed through the netlink. Because the +* associated netdevices are properties of GIDs. +*/ RDMA_NLDEV_ATTR_NDEV_INDEX ,/* u32 */ RDMA_NLDEV_ATTR_NDEV_NAME, /* string */
Signature · 1 line
1.8.3.1
Re: [PATCH RFC iproute2-next 1/2] rdma: update rdma_netlink.h to get provider attrs
Quoted 18 lines · Steve Wise
Signed-off-by Steve Wise <[email protected]>--- rdma/include/uapi/rdma/rdma_netlink.h | 37 ++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)rdma/include/uapi/rdma/rdma_netlink.h +5
-0@@ -249,10 +249,22 @@ enum rdma_nldev_command {RDMA_NLDEV_NUM_OPS }; +enum { +RDMA_NLDEV_ATTR_ENTRY_STRLEN =16 , +}; + +enum rdma_nldev_print_type {
rdma/include/uapi/rdma/rdma_netlink.h enum rdma_nldev_command { · line 266
Quoted 10 lines
rdma/include/uapi/rdma/rdma_netlink.h +6
-0+RDMA_NLDEV_PRINT_TYPE_UNSPEC , + RDMA_NLDEV_PRINT_TYPE_HEX, +}; +enum rdma_nldev_attr {/* don't change the order or add anything between, this is ABI! */ RDMA_NLDEV_ATTR_UNSPEC , +/* Pad attribute for 64b alignment */ + RDMA_NLDEV_ATTR_PAD = RDMA_NLDEV_ATTR_UNSPEC,
are you really inserting a new value here?
Quoted 12 lines
rdma/include/uapi/rdma/rdma_netlink.h +5
-0+/* Identifier for ib_device */ RDMA_NLDEV_ATTR_DEV_INDEX, /* u32 */@@ -387,8 +399,31 @@ enum rdma_nldev_attr {RDMA_NLDEV_ATTR_RES_PD_ENTRY ,/* nested table */ RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY ,/* u32 */ RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY ,/* u32 */ +/* +* provider-specific attributes. +*/ + RDMA_NLDEV_ATTR_PROVIDER, /* nested table */
rdma/include/uapi/rdma/rdma_netlink.h enum rdma_nldev_attr { · line 415
Quoted 10 lines
rdma/include/uapi/rdma/rdma_netlink.h +10
-0+RDMA_NLDEV_ATTR_PROVIDER_ENTRY ,/* nested table */ +RDMA_NLDEV_ATTR_PROVIDER_STRING ,/* string */ +/* +* u8 values from enum rdma_nldev_print_type +*/ +RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE ,/* u8 */ +RDMA_NLDEV_ATTR_PROVIDER_S32 ,/* s32 */ +RDMA_NLDEV_ATTR_PROVIDER_U32 ,/* u32 */ +RDMA_NLDEV_ATTR_PROVIDER_S64 ,/* s64 */ + RDMA_NLDEV_ATTR_PROVIDER_U64, /* u64 */
and this sequence too.
Quoted 16 lines
rdma/include/uapi/rdma/rdma_netlink.h +10
-1-/* Netdev information for relevant protocols, like RoCE and iWARP */ +/* +* Provides logical name and index of netdevice which is +* connected to physical port. This information is relevant +* for RoCE and iWARP. +* +* The netdevices which are associated with containers are +* supposed to be exported together with GID table once it +* will be exposed through the netlink. Because the +* associated netdevices are properties of GIDs. +*/ RDMA_NLDEV_ATTR_NDEV_INDEX ,/* u32 */ RDMA_NLDEV_ATTR_NDEV_NAME, /* string */
RFC PATCH 2/2 rdma: print provider resource attributes
In reply to Steve WiseThis enhancement allows printing rdma device-specific state, if provided by the kernel. This is done in a generic manner, so rdma tool doesn't need to know about the details of every type of rdma device.
Provider attributes for a rdma resource are in the form of <key, [print_type], value> tuples, where the key is a string and the value can be any supported provider attribute. The print_type attribute, if present, provides a print format to use vs the standard print format for the type. For example, the default print type for a PROVIDER_S32 value is "%d ", but "0x%x " if the print_type of PRINT_TYPE_HEX is included inthe tuple.
Provider resources are only printed when the -dd flag is present. If -p is present, then the output is formatted to not exceed 80 columns, otherwise it is printed as a single row to be grep/awk friendly.
Example output:
# rdma resource show qp lqpn 1028 -dd -p
link cxgb4_0/- lqpn 1028 rqpn 0 type RC state RTS rq-psn 0 sq-psn 0 path-mig-state MIGRATED pid 0 comm [nvme_rdma]
sqid 1028 flushed 0 memsize 123968 cidx 85 pidx 85 wq_pidx 106 flush_cidx 85 in_use 0
size 386 flags 0x0 rqid 1029 memsize 16768 cidx 43 pidx 41 wq_pidx 171 msn 44 rqt_hwaddr 0x2a8a5d00
rqt_size 256 in_use 128 size 130 idx 43 wr_id 0xffff881057c03408 idx 40 wr_id 0xffff881057c033f0--- rdma/rdma.c | 7 ++- rdma/rdma.h | 11 ++++ rdma/res.c | 30 +++------ rdma/utils.c | 194 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 221 insertions(+), 21 deletions(-)
rdma/rdma.c +6 -1
@@ -132,6 +132,7 @@ int main(int argc, char **argv)const char *batch_file =NULL ;bool pretty_output = false;bool show_details = false; +bool show_provider_details = false;bool json_output = false;bool force = false;char *filename;
@@ -152,7 +153,10 @@ int main(int argc, char **argv) pretty_output = true;break ;case 'd' : - show_details = true; +if (show_details) + show_provider_details = true; +else + show_details = true;break ;case 'j' : json_output = true;
@@ -180,6 +184,7 @@ int main(int argc, char **argv) argv += optind; rd.show_details = show_details; + rd.show_provider_details = show_provider_details; rd.json_output = json_output; rd.pretty_output = pretty_output;
rdma/rdma.h +11 -0
@@ -55,6 +55,7 @@ struct rd {char **argv;char *filename;bool show_details; +bool show_provider_details;struct list_head dev_map_list;uint32_t dev_idx;uint32_t port_idx;
@@ -115,4 +116,14 @@ int rd_recv_msg(struct rd *rd, mnl_cb_t callback, void *data, uint32_t seq);void rd_prepare_msg (struct rd *rd,uint32_t cmd,uint32_t *seq,uint16_t flags);int rd_dev_init_cb (const struct nlmsghdr *nlh,void *data);int rd_attr_cb (const struct nlattr *attr,void *data); +int rd_attr_check (const struct nlattr *attr,int *typep); + +/* +* Print helpers +*/ +void print_provider_table (struct rd *rd,struct nlattr *tb); +void newline (struct rd *rd); +void newline_indent (struct rd *rd); +#define MAX_LINE_LENGTH 80 +#endif /* _RDMA_TOOL_H_ */
rdma/res.c +10 -20
@@ -439,10 +439,8 @@ static int res_qp_parse_cb(const struct nlmsghdr *nlh, void *data)if (nla_line[RDMA_NLDEV_ATTR_RES_PID ])free (comm); -if (rd->json_output) -jsonw_end_array (rd->jw); -else -pr_out ("\n" ); +print_provider_table (rd, nla_line[RDMA_NLDEV_ATTR_PROVIDER ]); +newline (rd); }return MNL_CB_OK ; }
@@ -678,10 +676,8 @@ static int res_cm_id_parse_cb(const struct nlmsghdr *nlh, void *data)if (nla_line[RDMA_NLDEV_ATTR_RES_PID ])free (comm); -if (rd->json_output) -jsonw_end_array (rd->jw); -else -pr_out ("\n" ); +print_provider_table (rd, nla_line[RDMA_NLDEV_ATTR_PROVIDER ]); +newline (rd); }return MNL_CB_OK ; }
@@ -804,10 +800,8 @@ static int res_cq_parse_cb(const struct nlmsghdr *nlh, void *data)if (nla_line[RDMA_NLDEV_ATTR_RES_PID ])free (comm); -if (rd->json_output) -jsonw_end_array (rd->jw); -else -pr_out ("\n" ); +print_provider_table (rd, nla_line[RDMA_NLDEV_ATTR_PROVIDER ]); +newline (rd); }return MNL_CB_OK ; }
@@ -919,10 +913,8 @@ static int res_mr_parse_cb(const struct nlmsghdr *nlh, void *data)if (nla_line[RDMA_NLDEV_ATTR_RES_PID ])free (comm); -if (rd->json_output) -jsonw_end_array (rd->jw); -else -pr_out ("\n" ); +print_provider_table (rd, nla_line[RDMA_NLDEV_ATTR_PROVIDER ]); +newline (rd); }return MNL_CB_OK ; }
@@ -1004,10 +996,8 @@ static int res_pd_parse_cb(const struct nlmsghdr *nlh, void *data)if (nla_line[RDMA_NLDEV_ATTR_RES_PID ])free (comm); -if (rd->json_output) -jsonw_end_array (rd->jw); -else -pr_out ("\n" ); +print_provider_table (rd, nla_line[RDMA_NLDEV_ATTR_PROVIDER ]); +newline (rd); }return MNL_CB_OK ; }
rdma/utils.c +194 -0
211 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH RFC iproute2-next 2/2] rdma: print provider resource attributes
Quoted 35 lines · On Mon, 30 Apr 2018 07:36:18 -0700 Steve Wise <[email protected]>…
rdma/utils.c +35
-0+#define nla_type (attr) ((attr)->nla_type & NLA_TYPE_MASK) + +void newline (struct rd *rd) +{ +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); +} + +void newline_indent (struct rd *rd) +{ +newline (rd); +if (!rd->json_output) +pr_out (" " ); +} + +static int print_provider_string (struct rd *rd,const char *key_str, +const char *val_str) +{ +if (rd->json_output) { +jsonw_string_field (rd->jw, key_str, val_str); +return 0 ; + }else { +return pr_out ("%s %s " , key_str, val_str); + } +} + +static int print_provider_s32 (struct rd *rd,const char *key_str,int32_t val, +enum rdma_nldev_print_type print_type) +{ +if (rd->json_output) { +jsonw_int_field (rd->jw, key_str, val); +return 0 ; + }
rdma/utils.c struct dev_map *dev_map_lookup(struct rd *rd, bool allow_port_index) · line 733
Quoted 10 lines
rdma/utils.c +10
-0+switch (print_type) { +case RDMA_NLDEV_PRINT_TYPE_UNSPEC : +return pr_out ("%s %d " , key_str, val); +case RDMA_NLDEV_PRINT_TYPE_HEX : +return pr_out ("%s 0x%x " , key_str, val); +default : +return -EINVAL ; + } +} +
This code should get converted to json_print library that handles the different output modes; rather than rolling it's own equivalent functionality.
Re: [PATCH RFC iproute2-next 2/2] rdma: print provider resource attributes
Quoted 43 lines · Stephen Hemminger
Quoted 40 lines · On Mon, 30 Apr 2018 07:36:18 -0700 Steve Wise <[email protected]>…
rdma/utils.c +40
-0+#define nla_type (attr) ((attr)->nla_type & NLA_TYPE_MASK) + +void newline (struct rd *rd) +{ +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); +} + +void newline_indent (struct rd *rd) +{ +newline (rd); +if (!rd->json_output) +pr_out (" " ); +} + +static int print_provider_string (struct rd *rd,const char *key_str, +const char *val_str) +{ +if (rd->json_output) { +jsonw_string_field (rd->jw, key_str, val_str); +return 0 ; + }else { +return pr_out ("%s %s " , key_str, val_str); + } +} + +static int print_provider_s32 (struct rd *rd,const char *key_str,int32_t val, +enum rdma_nldev_print_type print_type) +{ +if (rd->json_output) { +jsonw_int_field (rd->jw, key_str, val); +return 0 ; + } +switch (print_type) { +case RDMA_NLDEV_PRINT_TYPE_UNSPEC : +return pr_out ("%s %d " , key_str, val); +case RDMA_NLDEV_PRINT_TYPE_HEX : +return pr_out ("%s 0x%x " , key_str, val);
Quoted 8 lines
rdma/utils.c struct dev_map *dev_map_lookup(struct rd *rd, bool allow_port_index) · line 733
Quoted 5 lines
rdma/utils.c +5
-0+default : +return -EINVAL ; + } +} +This code should get converted to
json_printlibrary that handles the different output modes; rather than rolling it's own equivalent functionality.
Can it be done after this patch is merged? It will simplify review and testing because current code is implemented to be in the same format as the rest of the tool.
Thanks
Re: [PATCH RFC iproute2-next 2/2] rdma: print provider resource attributes
Quoted 53 lines · On Wed, 2 May 2018 16:38:52 +0300 Leon Romanovsky <[email protected]> wrote:
Quoted 43 lines · Stephen Hemminger
Quoted 40 lines · On Mon, 30 Apr 2018 07:36:18 -0700 Steve Wise <[email protected]>…
rdma/utils.c +40
-0+#define nla_type (attr) ((attr)->nla_type & NLA_TYPE_MASK) + +void newline (struct rd *rd) +{ +if (rd->json_output) +jsonw_end_array (rd->jw); +else +pr_out ("\n" ); +} + +void newline_indent (struct rd *rd) +{ +newline (rd); +if (!rd->json_output) +pr_out (" " ); +} + +static int print_provider_string (struct rd *rd,const char *key_str, +const char *val_str) +{ +if (rd->json_output) { +jsonw_string_field (rd->jw, key_str, val_str); +return 0 ; + }else { +return pr_out ("%s %s " , key_str, val_str); + } +} + +static int print_provider_s32 (struct rd *rd,const char *key_str,int32_t val, +enum rdma_nldev_print_type print_type) +{ +if (rd->json_output) { +jsonw_int_field (rd->jw, key_str, val); +return 0 ; + } +switch (print_type) { +case RDMA_NLDEV_PRINT_TYPE_UNSPEC : +return pr_out ("%s %d " , key_str, val); +case RDMA_NLDEV_PRINT_TYPE_HEX : +return pr_out ("%s 0x%x " , key_str, val);Quoted 8 lines
rdma/utils.c struct dev_map *dev_map_lookup(struct rd *rd, bool allow_port_index) · line 733
Quoted 5 lines
rdma/utils.c +5
-0+default : +return -EINVAL ; + } +} +This code should get converted to
json_printlibrary that handles the different output modes; rather than rolling it's own equivalent functionality.
Quoted 5 lines
Can it be done after this patch is merged? It will simplify review and testing because current code is implemented to be in the same format as the rest of the tool.
Thanks
Sure, I was just encouraging future direction.