Statistics counter support
Subsystems (3)
cover Statistics counter support
From: Leon Romanovsky <[email protected]>
Changelog v0->v1:
* Fixed typo in manual page (Gal)
* Rebased on top of d035cc1b "ip tunnel: warn when changing IPv6 tunnel without tunnel name"
* Dropped update header file because it was already merged.
---------------------------------------------------------------------------------------
Hi,
This is supplementary part of accepted to rdma-next kernel series, that kernel series provided an option to get various counters: global and per-objects.
Currently, all counters are printed in format similar to other device/link properties, while "-p" option will print them in table like format.
[leonro@server ~]$ rdma stat show
link mlx5_0/1 rx_write_requests 0 rx_read_requests 0 rx_atomic_requests 0 out_of_buffer 0 duplicate_request 0 rnr_nak_retry_err 0 packet_seq_err 0 implied_nak_seq_err 0 local_ack_timeout_err 0 resp_local_length_error 0 resp_cqe_error 0 req_cqe_error 0 req_remote_invalid_request 0 req_remote_access_errors 0 resp_remote_access_errors 0 resp_cqe_flush_error 0 req_cqe_flush_error 0 rp_cnp_ignored 0 rp_cnp_handled 0 np_ecn_marked_roce_packets 0 np_cnp_sent 0
[leonro@server ~]$ rdma stat show -p
link mlx5_0/1
rx_write_requests 0
rx_read_requests 0
rx_atomic_requests 0
out_of_buffer 0
duplicate_request 0
rnr_nak_retry_err 0
packet_seq_err 0
implied_nak_seq_err 0
local_ack_timeout_err 0
resp_local_length_error 0
resp_cqe_error 0
req_cqe_error 0
req_remote_invalid_request 0
req_remote_access_errors 0
resp_remote_access_errors 0
resp_cqe_flush_error 0
req_cqe_flush_error 0
rp_cnp_ignored 0
rp_cnp_handled 0
np_ecn_marked_roce_packets 0
np_cnp_sent 0
Thanks
Mark Zhang (7):
rdma: Add "stat qp show" support
rdma: Add get per-port counter mode support
rdma: Add rdma statistic counter per-port auto mode support
rdma: Make get_port_from_argv() returns valid port in strict port mode
rdma: Add stat manual mode support
rdma: Add default counter show support
rdma: Document counter statistic
man/man8/rdma-dev.8 | 1 + man/man8/rdma-link.8 | 1 + man/man8/rdma-resource.8 | 1 + man/man8/rdma-statistic.8 | 167 +++++++++ man/man8/rdma.8 | 7 +- rdma/Makefile | 2 +- rdma/rdma.c | 3 +- rdma/rdma.h | 1 + rdma/stat.c | 759 ++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 17 +- 10 files changed, 954 insertions(+), 5 deletions(-) create mode 100644 man/man8/rdma-statistic.8 create mode 100644 rdma/stat.c
--
2.20.1
PATCH 1/7 rdma: Add "stat qp show" support
From: Mark Zhang <[email protected]>
This patch presents link, id, task name, lqpn, as well as all sub counters of a QP counter.
A QP counter is a dynamically allocated statistic counter that is bound with one or more QPs. It has several sub-counters, each is used for a different purpose.
Examples:
$ rdma stat qp show
link mlx5_2/1 cntn 5 pid 31609 comm client.1 rx_write_requests 0
rx_read_requests 0 rx_atomic_requests 0 out_of_buffer 0 out_of_sequence 0
duplicate_request 0 rnr_nak_retry_err 0 packet_seq_err 0
implied_nak_seq_err 0 local_ack_timeout_err 0 resp_local_length_error 0
resp_cqe_error 0 req_cqe_error 0 req_remote_invalid_request 0
req_remote_access_errors 0 resp_remote_access_errors 0
resp_cqe_flush_error 0 req_cqe_flush_error 0
LQPN: <178>
$ rdma stat show link rocep1s0f5/1
link rocep1s0f5/1 rx_write_requests 0 rx_read_requests 0 rx_atomic_requests 0 out_of_buffer 0 duplicate_request 0
rnr_nak_retry_err 0 packet_seq_err 0 implied_nak_seq_err 0 local_ack_timeout_err 0 resp_local_length_error 0 resp_cqe_error 0
req_cqe_error 0 req_remote_invalid_request 0 req_remote_access_errors 0 resp_remote_access_errors 0 resp_cqe_flush_error 0
req_cqe_flush_error 0 rp_cnp_ignored 0 rp_cnp_handled 0 np_ecn_marked_roce_packets 0 np_cnp_sent 0
$ rdma stat show link rocep1s0f5/1 -p
link rocep1s0f5/1
rx_write_requests 0
rx_read_requests 0
rx_atomic_requests 0
out_of_buffer 0
duplicate_request 0
rnr_nak_retry_err 0
packet_seq_err 0
implied_nak_seq_err 0
local_ack_timeout_err 0
resp_local_length_error 0
resp_cqe_error 0
req_cqe_error 0
req_remote_invalid_request 0
req_remote_access_errors 0
resp_remote_access_errors 0
resp_cqe_flush_error 0
req_cqe_flush_error 0
rp_cnp_ignored 0
rp_cnp_handled 0
np_ecn_marked_roce_packets 0
np_cnp_sent 0--- rdma/Makefile | 2 +- rdma/rdma.c | 3 +- rdma/rdma.h | 1 + rdma/stat.c | 268 ++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 7 ++ 5 files changed, 279 insertions(+), 2 deletions(-) create mode 100644 rdma/stat.c
rdma/Makefile +1 -1
@@ -7,7 +7,7 @@ ifeq ($(HAVE_MNL),y) CFLAGS += -I./include/uapi/ RDMA_OBJ = rdma.o utils.o dev.o link.o res.o res-pd.o res-mr.o res-cq.o \ - res-cmid.o res-qp.o sys.o + res-cmid.o res-qp.o sys.o stat.o TARGETS += rdma endif
rdma/rdma.c +2 -1
@@ -11,7 +11,7 @@ static void help(char *name) {pr_out ("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n" " %s [ -f[orce] ] -b[atch] filename\n" -"where OBJECT := { dev | link | resource | system | help }\n" +"where OBJECT := { dev | link | resource | system | statistic | help }\n" " OPTIONS := { -V[ersion] | -d[etails] | -j[son] | -p[retty]}\n" , name, name); }
@@ -30,6 +30,7 @@ static int rd_cmd(struct rd *rd, int argc, char **argv) { "link", cmd_link }, { "resource", cmd_res }, { "system", cmd_sys }, + { "statistic", cmd_stat }, { 0 } };
rdma/rdma.h +1 -0
@@ -94,6 +94,7 @@ int cmd_dev(struct rd *rd);int cmd_link (struct rd *rd);int cmd_res (struct rd *rd);int cmd_sys (struct rd *rd); +int cmd_stat (struct rd *rd);int rd_exec_cmd (struct rd *rd,const struct rd_cmd *c,const char *str);int rd_exec_dev (struct rd *rd,int (*cb)(struct rd *rd));int rd_exec_require_dev (struct rd *rd,int (*cb)(struct rd *rd));
rdma/stat.c new file +268 -0
268 lines not on this page · raw message
rdma/utils.c +7 -0
@@ -436,6 +436,13 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_DRIVER_S64] = MNL_TYPE_U64, [RDMA_NLDEV_ATTR_DRIVER_U64] = MNL_TYPE_U64, [RDMA_NLDEV_SYS_ATTR_NETNS_MODE] = MNL_TYPE_U8, + [RDMA_NLDEV_ATTR_STAT_COUNTER] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_STAT_COUNTER_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_STAT_COUNTER_ID] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_STAT_HWCOUNTERS] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY] = MNL_TYPE_NESTED, + [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY_NAME] = MNL_TYPE_NUL_STRING, + [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY_VALUE] = MNL_TYPE_U64, };int rd_attr_check (const struct nlattr *attr,int *typep)
Signature · 1 line
2.20.1
PATCH 2/7 rdma: Add get per-port counter mode support
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
Add an interface to show which mode is active. Two modes are supported:
- "auto": In this mode all QPs belong to one category are bind automatically to a single counter set. Currently only "qp type" is supported;
- "manual": In this mode QPs are bound to a counter manually.
Examples:
$ rdma statistic qp mode 0/1: mlx5_0/1: qp auto off 1/1: mlx5_1/1: qp auto off 2/1: mlx5_2/1: qp auto type on 3/1: mlx5_3/1: qp auto off
$ rdma statistic qp mode link mlx5_0 0/1: mlx5_0/1: qp auto off
--- rdma/stat.c | 140 +++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 2 + 2 files changed, 142 insertions(+)
rdma/stat.c +140 -0
@@ -13,13 +13,152 @@ static int stat_help(struct rd *rd)pr_out ("Usage: %s [ OPTIONS ] statistic { COMMAND | help }\n" , rd->filename);pr_out (" %s statistic OBJECT show\n" , rd->filename);pr_out (" %s statistic OBJECT show link [ DEV/PORT_INDEX ] [ FILTER-NAME FILTER-VALUE ]\n" , rd->filename); +pr_out (" %s statistic OBJECT mode\n" , rd->filename); +pr_out ("where OBJECT: = { qp }\n" );pr_out ("Examples:\n" );pr_out (" %s statistic qp show\n" , rd->filename);pr_out (" %s statistic qp show link mlx5_2/1\n" , rd->filename); +pr_out (" %s statistic qp mode\n" , rd->filename); +pr_out (" %s statistic qp mode link mlx5_0\n" , rd->filename);return 0 ; } +struct counter_param { +char *name; +uint32_t attr; +}; + +static struct counter_param auto_params[] = { + {"type" ,RDMA_COUNTER_MASK_QP_TYPE , }, + {NULL }, +}; + +static int prepare_auto_mode_str (struct nlattr **tb,uint32_t mask, +char *output,int len) +{ +char s[] ="qp auto" ; +int i, outlen =strlen (s); + +memset (output,0 , len); +snprintf (output, len,"%s" , s); + +if (mask) { +for (i =0 ; auto_params[i].name !=NULL ; i++) { +if (mask & auto_params[i].attr) { + outlen +=strlen (auto_params[i].name) +1 ; +if (outlen >= len) +return -EINVAL ; +strcat (output," " ); +strcat (output, auto_params[i].name); + } + } + +if (outlen +strlen (" on" ) >= len) +return -EINVAL ; +strcat (output," on" ); + }else { +if (outlen +strlen (" off" ) >= len) +return -EINVAL ; +strcat (output," off" ); + } + +return 0 ; +} + +static int qp_link_get_mode_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +uint32_t mode =0 , mask =0 ; +char output[128 ] = {}; +struct rd *rd = data; +uint32_t idx, port; +const char *name; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || !tb[RDMA_NLDEV_ATTR_DEV_NAME ]) +return MNL_CB_ERROR ; + +if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX ]) { +pr_err ("This tool doesn't support switches yet\n" ); +return MNL_CB_ERROR ; + } + + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); + port =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]); + name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); +if (tb[RDMA_NLDEV_ATTR_STAT_MODE ]) + mode =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_STAT_MODE ]); + +if (mode ==RDMA_COUNTER_MODE_AUTO ) { +if (!tb[RDMA_NLDEV_ATTR_STAT_AUTO_MODE_MASK ]) +return MNL_CB_ERROR ; + mask =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_STAT_AUTO_MODE_MASK ]); +prepare_auto_mode_str (tb, mask, output,sizeof (output)); + }else { +snprintf (output,sizeof (output),"qp auto off" ); + } + +if (rd->json_output) { +jsonw_uint_field (rd->jw,"ifindex" , idx); +jsonw_uint_field (rd->jw,"port" , port); +jsonw_string_field (rd->jw,"mode" , output); + }else { +pr_out ("%u/%u: %s/%u: %s\n" , idx, port, name, port, output); + } + +return MNL_CB_OK ; +} + +static int stat_one_qp_link_get_mode (struct rd *rd) +{ +uint32_t seq; +int ret; + +if (!rd->port_idx) +return 0 ; + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_GET , + &seq, (NLM_F_REQUEST |NLM_F_ACK )); + +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); +/* Make RDMA_NLDEV_ATTR_STAT_MODE valid so that kernel knows +* return only mode instead of all counters +*/ +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_MODE , +RDMA_COUNTER_MODE_MANUAL ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_RES ,RDMA_NLDEV_ATTR_RES_QP ); + ret =rd_send_msg (rd); +if (ret) +return ret; + +if (rd->json_output) +jsonw_start_object (rd->jw); + ret =rd_recv_msg (rd, qp_link_get_mode_parse_cb, rd, seq); +if (rd->json_output) +jsonw_end_object (rd->jw); + +return ret; +} + +static int stat_qp_link_get_mode (struct rd *rd) +{ +return rd_exec_link (rd, stat_one_qp_link_get_mode, false); +} + +static int stat_qp_get_mode (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_qp_link_get_mode }, + {"link" , stat_qp_link_get_mode }, + {"help" , stat_help }, + {0 } + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} +static int res_get_hwcounters (struct rd *rd,struct nlattr *hwc_table,bool print) {struct nlattr *nla_entry;
@@ -248,6 +387,7 @@ static int stat_qp(struct rd *rd) { NULL, stat_qp_show }, { "show", stat_qp_show }, { "list", stat_qp_show }, + { "mode", stat_qp_get_mode }, { "help", stat_help }, { 0 } };
rdma/utils.c +2 -0
@@ -443,6 +443,8 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY] = MNL_TYPE_NESTED, [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY_NAME] = MNL_TYPE_NUL_STRING, [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY_VALUE] = MNL_TYPE_U64, + [RDMA_NLDEV_ATTR_STAT_MODE] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_STAT_RES] = MNL_TYPE_U32, };int rd_attr_check (const struct nlattr *attr,int *typep)
Signature · 1 line
2.20.1
PATCH 3/7 rdma: Add rdma statistic counter per-port auto mode support
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
With per-QP statistic counter support, a user is allowed to monitor specific QPs categories, which are bound to/unbound from counters dynamically allocated/deallocated.
In per-port "auto" mode, QPs are bound to counters automatically according to common criteria. For example a per "type"(qp type) scheme, where in each process all QPs have same qp type are bind automatically to a single counter.
Currently only "type" (qp type) is supported. Examples:
$ rdma statistic qp set link mlx5_2/1 auto type on $ rdma statistic qp set link mlx5_2/1 auto off
--- rdma/stat.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 1 + 2 files changed, 88 insertions(+)
rdma/stat.c +87 -0
@@ -14,12 +14,17 @@ static int stat_help(struct rd *rd)pr_out (" %s statistic OBJECT show\n" , rd->filename);pr_out (" %s statistic OBJECT show link [ DEV/PORT_INDEX ] [ FILTER-NAME FILTER-VALUE ]\n" , rd->filename);pr_out (" %s statistic OBJECT mode\n" , rd->filename); +pr_out (" %s statistic OBJECT set COUNTER_SCOPE [DEV/PORT_INDEX] auto {CRITERIA | off}\n" , rd->filename);pr_out ("where OBJECT: = { qp }\n" ); +pr_out (" CRITERIA : = { type }\n" ); +pr_out (" COUNTER_SCOPE: = { link | dev }\n" );pr_out ("Examples:\n" );pr_out (" %s statistic qp show\n" , rd->filename);pr_out (" %s statistic qp show link mlx5_2/1\n" , rd->filename);pr_out (" %s statistic qp mode\n" , rd->filename);pr_out (" %s statistic qp mode link mlx5_0\n" , rd->filename); +pr_out (" %s statistic qp set link mlx5_2/1 auto type on\n" , rd->filename); +pr_out (" %s statistic qp set link mlx5_2/1 auto off\n" , rd->filename);return 0 ; }
@@ -381,6 +386,87 @@ static int stat_qp_show(struct rd *rd)return rd_exec_cmd (rd, cmds,"parameter" ); } +static int stat_qp_set_link_auto_sendmsg (struct rd *rd,uint32_t mask) +{ +uint32_t seq; + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_SET , + &seq, (NLM_F_REQUEST |NLM_F_ACK )); + +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_RES ,RDMA_NLDEV_ATTR_RES_QP ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_MODE , +RDMA_COUNTER_MODE_AUTO ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_AUTO_MODE_MASK , mask); + +return rd_sendrecv_msg (rd, seq); +} + +static int stat_one_qp_set_link_auto_off (struct rd *rd) +{ +return stat_qp_set_link_auto_sendmsg (rd,0 ); +} + +static int stat_one_qp_set_auto_type_on (struct rd *rd) +{ +return stat_qp_set_link_auto_sendmsg (rd,RDMA_COUNTER_MASK_QP_TYPE ); +} + +static int stat_one_qp_set_link_auto_type (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_help }, + {"on" , stat_one_qp_set_auto_type_on }, + {0 } + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} + +static int stat_one_qp_set_link_auto (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_one_qp_link_get_mode }, + {"off" , stat_one_qp_set_link_auto_off }, + {"type" , stat_one_qp_set_link_auto_type }, + {0 } + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} + +static int stat_one_qp_set_link (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_one_qp_link_get_mode }, + {"auto" , stat_one_qp_set_link_auto }, + {0 } + }; + +if (!rd->port_idx) +return 0 ; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} + +static int stat_qp_set_link (struct rd *rd) +{ +return rd_exec_link (rd, stat_one_qp_set_link, false); +} + +static int stat_qp_set (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_help }, + {"link" , stat_qp_set_link }, + {"help" , stat_help }, + {0 } + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} +static int stat_qp (struct rd *rd) {const struct rd_cmd cmds[] = {
@@ -388,6 +474,7 @@ static int stat_qp(struct rd *rd) { "show", stat_qp_show }, { "list", stat_qp_show }, { "mode", stat_qp_get_mode }, + { "set", stat_qp_set }, { "help", stat_help }, { 0 } };
rdma/utils.c +1 -0
@@ -445,6 +445,7 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = { [RDMA_NLDEV_ATTR_STAT_HWCOUNTER_ENTRY_VALUE] = MNL_TYPE_U64, [RDMA_NLDEV_ATTR_STAT_MODE] = MNL_TYPE_U32, [RDMA_NLDEV_ATTR_STAT_RES] = MNL_TYPE_U32, + [RDMA_NLDEV_ATTR_STAT_AUTO_MODE_MASK] = MNL_TYPE_U32, };int rd_attr_check (const struct nlattr *attr,int *typep)
Signature · 1 line
2.20.1
PATCH 4/7 rdma: Make get_port_from_argv() returns valid port in strict port mode
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
When strict_port is set, make get_port_from_argv() returns failure if no valid port is specified.
--- rdma/utils.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
rdma/utils.c +5 -2
@@ -56,7 +56,7 @@ bool rd_no_arg(struct rd *rd)* mlx5_1/1 | 1 | false * mlx5_1/- | 0 | false * -* In strict mode, /- will return error. +* In strict port mode, a non-0 port must be provided */ static int get_port_from_argv (struct rd *rd,uint32_t *port,bool *is_dump_all,bool strict_port)
@@ -64,7 +64,7 @@ static int get_port_from_argv(struct rd *rd, uint32_t *port,char *slash; *port =0 ; - *is_dump_all = true; + *is_dump_all = strict_port ? false : true; slash =strchr (rd_argv (rd),'/' );/* if no port found, return 0 */
@@ -83,6 +83,9 @@ static int get_port_from_argv(struct rd *rd, uint32_t *port,if (!*port &&strlen (slash))return -EINVAL ; } +if (strict_port && (*port ==0 )) +return -EINVAL ; +return 0 ; }
Signature · 1 line
2.20.1
PATCH 5/7 rdma: Add stat manual mode support
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
In manual mode a QP can be manually bound to a counter. If the counter id(cntn) is not specified that kernel will allocate one. After a successful bind, the cntn can be seen through "rdma statistic qp show". And in unbind if lqpn is not specified then all QPs on this counter will be unbound.
The manual and auto mode are mutual-exclusive.
Examples:
$ rdma statistic qp bind link mlx5_2/1 lqpn 178 $ rdma statistic qp bind link mlx5_2/1 lqpn 178 cntn 4 $ rdma statistic qp unbind link mlx5_2/1 cntn 4 $ rdma statistic qp unbind link mlx5_2/1 cntn 4 lqpn 178
--- rdma/stat.c | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 192 insertions(+)
rdma/stat.c +192 -0
@@ -15,6 +15,8 @@ static int stat_help(struct rd *rd)pr_out (" %s statistic OBJECT show link [ DEV/PORT_INDEX ] [ FILTER-NAME FILTER-VALUE ]\n" , rd->filename);pr_out (" %s statistic OBJECT mode\n" , rd->filename);pr_out (" %s statistic OBJECT set COUNTER_SCOPE [DEV/PORT_INDEX] auto {CRITERIA | off}\n" , rd->filename); +pr_out (" %s statistic OBJECT bind COUNTER_SCOPE [DEV/PORT_INDEX] [OBJECT-ID] [COUNTER-ID]\n" , rd->filename); +pr_out (" %s statistic OBJECT unbind COUNTER_SCOPE [DEV/PORT_INDEX] [COUNTER-ID]\n" , rd->filename);pr_out ("where OBJECT: = { qp }\n" );pr_out (" CRITERIA : = { type }\n" );pr_out (" COUNTER_SCOPE: = { link | dev }\n" );
@@ -25,6 +27,10 @@ static int stat_help(struct rd *rd)pr_out (" %s statistic qp mode link mlx5_0\n" , rd->filename);pr_out (" %s statistic qp set link mlx5_2/1 auto type on\n" , rd->filename);pr_out (" %s statistic qp set link mlx5_2/1 auto off\n" , rd->filename); +pr_out (" %s statistic qp bind link mlx5_2/1 lqpn 178\n" , rd->filename); +pr_out (" %s statistic qp bind link mlx5_2/1 lqpn 178 cntn 4\n" , rd->filename); +pr_out (" %s statistic qp unbind link mlx5_2/1 cntn 4\n" , rd->filename); +pr_out (" %s statistic qp unbind link mlx5_2/1 cntn 4 lqpn 178\n" , rd->filename);return 0 ; }
@@ -467,6 +473,190 @@ static int stat_qp_set(struct rd *rd)return rd_exec_cmd (rd, cmds,"parameter" ); } +static int stat_get_arg (struct rd *rd,const char *arg) +{ +int value =0 ; +char *endp; + +if (strcmpx (rd_argv (rd), arg) !=0 ) +return -EINVAL ; + +rd_arg_inc (rd); + value =strtol (rd_argv (rd), &endp,10 ); +rd_arg_inc (rd); + +return value; +} + +static int stat_one_qp_bind (struct rd *rd) +{ +int lqpn =0 , cntn =0 , ret; +uint32_t seq; + +if (rd_no_arg (rd)) { +stat_help (rd); +return -EINVAL ; + } + + ret =rd_build_filter (rd, stat_valid_filters); +if (ret) +return ret; + + lqpn =stat_get_arg (rd,"lqpn" ); + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_SET , + &seq, (NLM_F_REQUEST |NLM_F_ACK )); + +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_MODE , +RDMA_COUNTER_MODE_MANUAL ); + +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_RES ,RDMA_NLDEV_ATTR_RES_QP ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_RES_LQPN , lqpn); + +if (rd_argc (rd)) { + cntn =stat_get_arg (rd,"cntn" ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_COUNTER_ID , + cntn); + } + +return rd_sendrecv_msg (rd, seq); +} + +static int do_stat_qp_unbind_lqpn (struct rd *rd,uint32_t cntn,uint32_t lqpn) +{ +uint32_t seq; + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_DEL , + &seq, (NLM_F_REQUEST |NLM_F_ACK )); + +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_MODE , +RDMA_COUNTER_MODE_MANUAL ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_RES ,RDMA_NLDEV_ATTR_RES_QP ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_COUNTER_ID , cntn); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_RES_LQPN , lqpn); + +return rd_sendrecv_msg (rd, seq); +} + +static int stat_get_counter_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +struct nlattr *nla_table, *nla_entry; +struct rd *rd = data; +uint32_t lqpn, cntn; +int err; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); + +if (!tb[RDMA_NLDEV_ATTR_STAT_COUNTER_ID ]) +return MNL_CB_ERROR ; + cntn =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_STAT_COUNTER_ID ]); + + nla_table = tb[RDMA_NLDEV_ATTR_RES_QP ]; +if (!nla_table) +return MNL_CB_ERROR ; + +mnl_attr_for_each_nested (nla_entry, nla_table) { +struct nlattr *nla_line[RDMA_NLDEV_ATTR_MAX ] = {}; + + err =mnl_attr_parse_nested (nla_entry, rd_attr_cb, nla_line); +if (err !=MNL_CB_OK ) +return -EINVAL ; + +if (!nla_line[RDMA_NLDEV_ATTR_RES_LQPN ]) +return -EINVAL ; + + lqpn =mnl_attr_get_u32 (nla_line[RDMA_NLDEV_ATTR_RES_LQPN ]); + err =do_stat_qp_unbind_lqpn (rd, cntn, lqpn); +if (err) +return MNL_CB_ERROR ; + } + +return MNL_CB_OK ; +} + +static int stat_one_qp_unbind (struct rd *rd) +{ +int flags =NLM_F_REQUEST |NLM_F_ACK , ret; +char buf[MNL_SOCKET_BUFFER_SIZE ]; +int lqpn =0 , cntn =0 ; +unsigned int portid; +uint32_t seq; + + ret =rd_build_filter (rd, stat_valid_filters); +if (ret) +return ret; + + cntn =stat_get_arg (rd,"cntn" ); +if (rd_argc (rd)) { + lqpn =stat_get_arg (rd,"lqpn" ); +return do_stat_qp_unbind_lqpn (rd, cntn, lqpn); + } + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_GET , &seq, flags); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_RES ,RDMA_NLDEV_ATTR_RES_QP ); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_STAT_COUNTER_ID , cntn); + ret =rd_send_msg (rd); +if (ret) +return ret; + + +/* Can't use rd_recv_msg() since the callback also calls it (recursively), +* then rd_recv_msg() always return -1 here +*/ + portid =mnl_socket_get_portid (rd->nl); + ret =mnl_socket_recvfrom (rd->nl, buf,sizeof (buf)); +if (ret <=0 ) +return ret; + + ret =mnl_cb_run (buf, ret, seq, portid, stat_get_counter_parse_cb, rd); +mnl_socket_close (rd->nl); +if (ret !=MNL_CB_OK ) +return ret; + +return 0 ; +} + +static int stat_qp_bind_link (struct rd *rd) +{ +return rd_exec_link (rd, stat_one_qp_bind, true); +} + +static int stat_qp_bind (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_help }, + {"link" , stat_qp_bind_link }, + {"help" , stat_help }, + {0 }, + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} + +static int stat_qp_unbind_link (struct rd *rd) +{ +return rd_exec_link (rd, stat_one_qp_unbind, true); +} + +static int stat_qp_unbind (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_help }, + {"link" , stat_qp_unbind_link }, + {"help" , stat_help }, + {0 }, + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} +static int stat_qp (struct rd *rd) {const struct rd_cmd cmds[] = {
@@ -475,6 +665,8 @@ static int stat_qp(struct rd *rd) { "list", stat_qp_show }, { "mode", stat_qp_get_mode }, { "set", stat_qp_set }, + { "bind", stat_qp_bind }, + { "unbind", stat_qp_unbind }, { "help", stat_help }, { 0 } };
Signature · 1 line
2.20.1
PATCH 6/7 rdma: Add default counter show support
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
Show default counter statistics, which are same through the sysfs
interface: /sys/class/infiniband/<dev>/ports/<port>/hw_counters/
Example:
$ rdma stat show link mlx5_2/1 link mlx5_2/1 rx_write_requests 8 rx_read_requests 4 rx_atomic_requests 0 out_of_buffer 0 out_of_sequence 0 duplicate_request 0 rnr_nak_retry_err 0 packet_seq_err 0 implied_nak_seq_err 0 local_ack_timeout_err 0 resp_local_length_error 0 resp_cqe_error 0 req_cqe_error 0 req_remote_invalid_request 0 req_remote_access_errors 0 resp_remote_access_errors 0 resp_cqe_flush_error 0 req_cqe_flush_error 0 rp_cnp_ignored 0 rp_cnp_handled 0 np_ecn_marked_roce_packets 0 np_cnp_sent 0 rx_icrc_encapsulated 0
--- rdma/stat.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-)
rdma/stat.c +73 -1
@@ -17,6 +17,8 @@ static int stat_help(struct rd *rd)pr_out (" %s statistic OBJECT set COUNTER_SCOPE [DEV/PORT_INDEX] auto {CRITERIA | off}\n" , rd->filename);pr_out (" %s statistic OBJECT bind COUNTER_SCOPE [DEV/PORT_INDEX] [OBJECT-ID] [COUNTER-ID]\n" , rd->filename);pr_out (" %s statistic OBJECT unbind COUNTER_SCOPE [DEV/PORT_INDEX] [COUNTER-ID]\n" , rd->filename); +pr_out (" %s statistic show\n" , rd->filename); +pr_out (" %s statistic show link [ DEV/PORT_INDEX ]\n" , rd->filename);pr_out ("where OBJECT: = { qp }\n" );pr_out (" CRITERIA : = { type }\n" );pr_out (" COUNTER_SCOPE: = { link | dev }\n" );
@@ -31,6 +33,8 @@ static int stat_help(struct rd *rd)pr_out (" %s statistic qp bind link mlx5_2/1 lqpn 178 cntn 4\n" , rd->filename);pr_out (" %s statistic qp unbind link mlx5_2/1 cntn 4\n" , rd->filename);pr_out (" %s statistic qp unbind link mlx5_2/1 cntn 4 lqpn 178\n" , rd->filename); +pr_out (" %s statistic show\n" , rd->filename); +pr_out (" %s statistic show link mlx5_2/1\n" , rd->filename);return 0 ; }
@@ -674,10 +678,78 @@ static int stat_qp(struct rd *rd)return rd_exec_cmd (rd, cmds,"parameter" ); } +static int stat_show_parse_cb (const struct nlmsghdr *nlh,void *data) +{ +struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ] = {}; +struct rd *rd = data; +const char *name; +uint32_t port; +int ret; + +mnl_attr_parse (nlh,0 , rd_attr_cb, tb); +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || !tb[RDMA_NLDEV_ATTR_DEV_NAME ] || + !tb[RDMA_NLDEV_ATTR_PORT_INDEX ] || + !tb[RDMA_NLDEV_ATTR_STAT_HWCOUNTERS ]) +return MNL_CB_ERROR ; + + name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); + port =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]); +if (rd->json_output) { +jsonw_string_field (rd->jw,"ifname" , name); +jsonw_uint_field (rd->jw,"port" , port); + }else { +pr_out ("link %s/%u " , name, port); + } + + ret =res_get_hwcounters (rd, tb[RDMA_NLDEV_ATTR_STAT_HWCOUNTERS ], true); + +if (!rd->json_output) +pr_out ("\n" ); +return ret; +} + +static int stat_show_one_link (struct rd *rd) +{ +int flags =NLM_F_REQUEST |NLM_F_ACK ; +uint32_t seq; +int ret; + +if (!rd->port_idx) +return 0 ; + +rd_prepare_msg (rd,RDMA_NLDEV_CMD_STAT_GET , &seq, flags); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_DEV_INDEX , rd->dev_idx); +mnl_attr_put_u32 (rd->nlh,RDMA_NLDEV_ATTR_PORT_INDEX , rd->port_idx); + ret =rd_send_msg (rd); +if (ret) +return ret; + +return rd_recv_msg (rd, stat_show_parse_cb, rd, seq); +} + +static int stat_show_link (struct rd *rd) +{ +return rd_exec_link (rd, stat_show_one_link, false); +} + +static int stat_show (struct rd *rd) +{ +const struct rd_cmd cmds[] = { + {NULL , stat_show_link }, + {"link" , stat_show_link }, + {"help" , stat_help }, + {0 } + }; + +return rd_exec_cmd (rd, cmds,"parameter" ); +} +int cmd_stat (struct rd *rd) {const struct rd_cmd cmds[] = { - {NULL , stat_help }, + {NULL , stat_show }, + {"show" , stat_show }, + {"list" , stat_show }, {"help" , stat_help }, {"qp" , stat_qp }, {0 }
Signature · 1 line
2.20.1
PATCH 7/7 rdma: Document counter statistic
In reply to Leon RomanovskyFrom: Mark Zhang <[email protected]>
Add document of accessing the QP counter, including bind/unbind a QP to a counter manually or automatically, and dump counter statistics.
--- man/man8/rdma-dev.8 | 1 + man/man8/rdma-link.8 | 1 + man/man8/rdma-resource.8 | 1 + man/man8/rdma-statistic.8 | 167 ++++++++++++++++++++++++++++++++++++++ man/man8/rdma.8 | 7 +- 5 files changed, 176 insertions(+), 1 deletion(-) create mode 100644 man/man8/rdma-statistic.8
man/man8/rdma-dev.8 +1 -0
@@ -77,6 +77,7 @@ previously created using iproute2 ip command. .BR rdma-link (8), .BR rdma-resource (8), .BR rdma-system (8), +.BR rdma-statistic (8), .br .SH AUTHOR
man/man8/rdma-link.8 +1 -0
@@ -97,6 +97,7 @@ Removes RXE link rxe_eth0 .BR rdma (8), .BR rdma-dev (8), .BR rdma-resource (8), +.BR rdma-statistic (8), .br .SH AUTHOR
man/man8/rdma-resource.8 +1 -0
@@ -103,6 +103,7 @@ Show CQs belonging to pid 30489 .BR rdma (8), .BR rdma-dev (8), .BR rdma-link (8), +.BR rdma-statistic (8), .br .SH AUTHOR
man/man8/rdma-statistic.8 new file +167 -0
167 lines not on this page · raw message
man/man8/rdma.8 +6 -1
@@ -19,7 +19,7 @@ rdma \- RDMA tool .ti -8 .IR OBJECT " := { " -.BR dev " | " link " | " system " }" +.BR dev " | " link " | " system " | " statistic " }" .sp .ti -8
@@ -74,6 +74,10 @@ Generate JSON output. .B sys - RDMA subsystem related. +.TP +.B statistic +- RDMA counter statistic related. + .PP The names of all objects may be written in full or abbreviated form, for example
@@ -112,6 +116,7 @@ Exit status is 0 if command was successful or a positive integer upon failure. .BR rdma-link (8), .BR rdma-resource (8), .BR rdma-system (8), +.BR rdma-statistic (8), .br .SH REPORTING BUGS
Signature · 1 line
2.20.1
Quoted 77 lines · On Wed, 17 Jul 2019 17:31:49 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Changelog
v0->v1:
* Fixed typo in manual page (Gal)
* Rebased on top ofd035cc1b"ip tunnel: warn when changing IPv6 tunnel without tunnel name"
* Dropped update header file because it was already merged.---------------------------------------------------------------------------------------
Hi,
This is supplementary part of accepted to rdma-next kernel series, that kernel series provided an option to get various counters: global and per-objects.
Currently, all counters are printed in format similar to other device/link properties, while "-p" option will print them in table like format.
[leonro@server ~]$ rdma stat show
link mlx5_0/1rx_write_requests0rx_read_requests0rx_atomic_requests0out_of_buffer0duplicate_request0rnr_nak_retry_err0packet_seq_err0implied_nak_seq_err0local_ack_timeout_err0resp_local_length_error0resp_cqe_error0req_cqe_error0req_remote_invalid_request0req_remote_access_errors0resp_remote_access_errors0resp_cqe_flush_error0req_cqe_flush_error0rp_cnp_ignored0rp_cnp_handled0np_ecn_marked_roce_packets0np_cnp_sent0[leonro@server ~]$ rdma stat show -p
link mlx5_0/1
rx_write_requests0
rx_read_requests0
rx_atomic_requests0
out_of_buffer0
duplicate_request0
rnr_nak_retry_err0
packet_seq_err0
implied_nak_seq_err0
local_ack_timeout_err0
resp_local_length_error0
resp_cqe_error0
req_cqe_error0
req_remote_invalid_request0
req_remote_access_errors0
resp_remote_access_errors0
resp_cqe_flush_error0
req_cqe_flush_error0
rp_cnp_ignored0
rp_cnp_handled0
np_ecn_marked_roce_packets0
np_cnp_sent0Thanks
Mark Zhang (7):
rdma: Add "stat qp show" support
rdma: Add get per-port counter mode support
rdma: Add rdma statistic counter per-port auto mode support
rdma: Makeget_port_from_argv()returns valid port in strict port mode
rdma: Add stat manual mode support
rdma: Add default counter show support
rdma: Document counter statisticman/man8/rdma-dev.8 | 1 + man/man8/rdma-link.8 | 1 + man/man8/rdma-resource.8 | 1 + man/man8/rdma-statistic.8 | 167 +++++++++ man/man8/rdma.8 | 7 +- rdma/Makefile | 2 +- rdma/rdma.c | 3 +- rdma/rdma.h | 1 + rdma/stat.c | 759 ++++++++++++++++++++++++++++++++++++++ rdma/utils.c | 17 +- 10 files changed, 954 insertions(+), 5 deletions(-) create mode 100644 man/man8/rdma-statistic.8 create mode 100644 rdma/stat.c
Quoted 3 lines
--
2.20.1
Applied now, thanks