IB/mlx5: Support congestion related counters
Subsystems (3)
PATCH 1/1 IB/mlx5: Support congestion related counters
From: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>
This patch adds support to query the congestion related hardware counters through new command and links them with other hw counters being available in hw_counters sysfs location.
In order to reuse existing infrastructure it renames related q_counter data structures to more generic counters to reflect q_counters and congestion counters and maybe some other counters in the future.
New hardware counters:
* rp_cnp_handled - CNP packets handled by the reaction point
* rp_cnp_ignored - CNP packets ignored by the reaction point
* np_cnp_sent - CNP packets sent by notification point to respond to
CE marked RoCE packets
* np_ecn_marked_roce_packets - CE marked RoCE packets received by notification point
It also avoids returning ENOSYS which is specific for invalid system call and produces the following checkpatch.pl warning.
WARNING: ENOSYS means 'invalid syscall nr' and nothing else
+ return -ENOSYS;
---
Changes from v0:
* fix overflow issue in access to congestion counters
--- drivers/infiniband/hw/mlx5/cmd.c | 11 +++ drivers/infiniband/hw/mlx5/cmd.h | 2 + drivers/infiniband/hw/mlx5/main.c | 172 ++++++++++++++++++++++++----------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- drivers/infiniband/hw/mlx5/qp.c | 7 +- include/linux/mlx5/mlx5_ifc.h | 18 ++-- 6 files changed, 150 insertions(+), 67 deletions(-)
drivers/infiniband/hw/mlx5/cmd.c +11 -0
@@ -46,3 +46,14 @@ int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey) null_mkey);return err; } + +int mlx5_cmd_query_cong_counter (struct mlx5_core_dev *dev, +bool reset,void *out,int out_size) +{ +u32 in[MLX5_ST_SZ_DW (query_cong_statistics_in)] = { }; + +MLX5_SET (query_cong_statistics_in, in, opcode, +MLX5_CMD_OP_QUERY_CONG_STATISTICS ); +MLX5_SET (query_cong_statistics_in, in, clear, reset); +return mlx5_cmd_exec (dev, in,sizeof (in), out, out_size); +}
drivers/infiniband/hw/mlx5/cmd.h +2 -0
@@ -37,4 +37,6 @@#include <linux/mlx5/driver.h> int mlx5_cmd_null_mkey (struct mlx5_core_dev *dev,u32 *null_mkey); +int mlx5_cmd_query_cong_counter (struct mlx5_core_dev *dev, +bool reset,void *out,int out_size);#endif /* MLX5_IB_CMD_H */
drivers/infiniband/hw/mlx5/main.c +120 -52
312 lines not on this page · raw message
drivers/infiniband/hw/mlx5/mlx5_ib.h +4 -3
@@ -595,15 +595,16 @@ struct mlx5_ib_resources {struct mutex mutex; }; -struct mlx5_ib_q_counters { +struct mlx5_ib_counters {const char **names;size_t *offsets; -u32 num_counters; +u32 num_q_counters; +u32 num_cong_counters;u16 set_id; };struct mlx5_ib_port { -struct mlx5_ib_q_counters q_cnts; +struct mlx5_ib_counters cnts; };struct mlx5_roce {
drivers/infiniband/hw/mlx5/qp.c +4 -3
@@ -2798,7 +2798,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, qp->port) - 1; mibport = &dev->port[port_num]; context->qp_counter_set_usr_page |= -cpu_to_be32 ((u32)(mibport->q_cnts.set_id) <<24 ); +cpu_to_be32 ((u32)(mibport->cnts.set_id) <<24 ); }if (!ibqp->uobject && cur_state ==IB_QPS_RESET && new_state ==IB_QPS_INIT )
@@ -2826,7 +2826,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, raw_qp_param.operation = op;if (cur_state ==IB_QPS_RESET && new_state ==IB_QPS_INIT ) { - raw_qp_param.rq_q_ctr_id = mibport->q_cnts.set_id; + raw_qp_param.rq_q_ctr_id = mibport->cnts.set_id; raw_qp_param.set_mask |=MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID ; }
@@ -4964,7 +4964,8 @@ int mlx5_ib_modify_wq(struct ib_wq *wq, struct ib_wq_attr *wq_attr,if (MLX5_CAP_GEN (dev->mdev, modify_rq_counter_set_id)) {MLX5_SET64 (modify_rq_in, in, modify_bitmask,MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_RQ_COUNTER_SET_ID ); -MLX5_SET (rqc, rqc, counter_set_id, dev->port->q_cnts.set_id); +MLX5_SET (rqc, rqc, counter_set_id, + dev->port->cnts.set_id); }else pr_info_once ("%s: Receive WQ counters are not supported on current FW\n" , dev->ib_dev.name);
include/linux/mlx5/mlx5_ifc.h +9 -9
@@ -4623,17 +4623,17 @@ struct mlx5_ifc_query_cong_statistics_out_bits {u8 reserved_at_40[0x40 ]; -u8 cur_flows[0x20 ]; +u8 rp_cur_flows[0x20 ];u8 sum_flows[0x20 ]; -u8 cnp_ignored_high[0x20 ]; +u8 rp_cnp_ignored_high[0x20 ]; -u8 cnp_ignored_low[0x20 ]; +u8 rp_cnp_ignored_low[0x20 ]; -u8 cnp_handled_high[0x20 ]; +u8 rp_cnp_handled_high[0x20 ]; -u8 cnp_handled_low[0x20 ]; +u8 rp_cnp_handled_low[0x20 ];u8 reserved_at_140[0x100 ];
@@ -4643,13 +4643,13 @@ struct mlx5_ifc_query_cong_statistics_out_bits {u8 accumulators_period[0x20 ]; -u8 ecn_marked_roce_packets_high[0x20 ]; +u8 np_ecn_marked_roce_packets_high[0x20 ]; -u8 ecn_marked_roce_packets_low[0x20 ]; +u8 np_ecn_marked_roce_packets_low[0x20 ]; -u8 cnps_sent_high[0x20 ]; +u8 np_cnps_sent_high[0x20 ]; -u8 cnps_sent_low[0x20 ]; +u8 np_cnps_sent_low[0x20 ];u8 reserved_at_320[0x560 ]; };
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170413134810.13159-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
kbuild test robot
build botreproduce:
Hi Parav,
[auto build test ERROR on rdma/master]
[also build test ERROR on v4.11-rc6 next-20170413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Leon-Romanovsky/IB-mlx5-Support-congestion-related-counters/20170413-231953 base: https://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git master config: x86_64-randconfig-x018-201715 (attached as .config) compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
All error/warnings (new ones prefixed by >>):
In file included from include/linux/mlx5/driver.h:48:0,
from include/linux/mlx5/port.h:36,
from drivers/infiniband/hw/mlx5/main.c:50:
>> include/linux/mlx5/device.h:51:80: error: 'struct mlx5_ifc_query_cong_statistics_out_bits' has no member named 'np_cnp_sent_high'; did you mean 'np_cnps_sent_high'?
#define __mlx5_bit_off(typ, fld) ((unsigned)(unsigned long)(&(__mlx5_nullp(typ)->fld)))
^
>> include/linux/mlx5/device.h:65:34: note: in expansion of macro '__mlx5_bit_off'
#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)
^~~~~~~~~~~~~~
>> drivers/infiniband/hw/mlx5/main.c:3166:3: note: in expansion of macro 'MLX5_BYTE_OFF'
MLX5_BYTE_OFF(query_cong_statistics_out, _name ## _high)}
^~~~~~~~~~~~~
>> drivers/infiniband/hw/mlx5/main.c:3172:2: note: in expansion of macro 'INIT_CONG_COUNTER'
INIT_CONG_COUNTER(np_cnp_sent),
^~~~~~~~~~~~~~~~~
--
In file included from include/linux/mlx5/driver.h:48:0,
from include/linux/mlx5/port.h:36,
from drivers/infiniband//hw/mlx5/main.c:50:
>> include/linux/mlx5/device.h:51:80: error: 'struct mlx5_ifc_query_cong_statistics_out_bits' has no member named 'np_cnp_sent_high'; did you mean 'np_cnps_sent_high'?
#define __mlx5_bit_off(typ, fld) ((unsigned)(unsigned long)(&(__mlx5_nullp(typ)->fld)))
^
>> include/linux/mlx5/device.h:65:34: note: in expansion of macro '__mlx5_bit_off'
#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)
^~~~~~~~~~~~~~
drivers/infiniband//hw/mlx5/main.c:3166:3: note: in expansion of macro 'MLX5_BYTE_OFF'
MLX5_BYTE_OFF(query_cong_statistics_out, _name ## _high)}
^~~~~~~~~~~~~
drivers/infiniband//hw/mlx5/main.c:3172:2: note: in expansion of macro 'INIT_CONG_COUNTER'
INIT_CONG_COUNTER(np_cnp_sent),
^~~~~~~~~~~~~~~~~vim +51 include/linux/mlx5/device.h
e126ba97 Eli Cohen 2013-07-07 45 #error Host endianness not defined e126ba97 Eli Cohen 2013-07-07 46 #endif e126ba97 Eli Cohen 2013-07-07 47 d29b796a Eli Cohen 2014-10-02 48 /* helper macros */ d29b796a Eli Cohen 2014-10-02 49 #define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0) d29b796a Eli Cohen 2014-10-02 50 #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld) d29b796a Eli Cohen 2014-10-02 @51 #define __mlx5_bit_off(typ, fld) ((unsigned)(unsigned long)(&(__mlx5_nullp(typ)->fld))) d29b796a Eli Cohen 2014-10-02 52 #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32) d29b796a Eli Cohen 2014-10-02 53 #define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64) d29b796a Eli Cohen 2014-10-02 54 #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f)) d29b796a Eli Cohen 2014-10-02 55 #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) d29b796a Eli Cohen 2014-10-02 56 #define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld)) d29b796a Eli Cohen 2014-10-02 57 #define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) d29b796a Eli Cohen 2014-10-02 58 d29b796a Eli Cohen 2014-10-02 59 #define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) d29b796a Eli Cohen 2014-10-02 60 #define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) d29b796a Eli Cohen 2014-10-02 61 #define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) 9218b44d Gal Pressman 2016-04-24 62 #define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) 938fe83c Saeed Mahameed 2015-05-28 63 #define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8) 938fe83c Saeed Mahameed 2015-05-28 64 #define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32) d29b796a Eli Cohen 2014-10-02 @65 #define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8) d29b796a Eli Cohen 2014-10-02 66 #define MLX5_ADDR_OF(typ, p, fld) ((char *)(p) + MLX5_BYTE_OFF(typ, fld)) d29b796a Eli Cohen 2014-10-02 67 d29b796a Eli Cohen 2014-10-02 68 /* insert a value to a struct */
:::::: The code at line 51 was first introduced by commit
:::::: d29b796adada8780db3512c4a34b339f9aeef1ae net/mlx5_core: Use hardware registers description header file
:::::: TO: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]> :::::: CC: David S. Miller <davem-fT/PcQaiUtIeIZ0/[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botreproduce:
Hi Parav,
[auto build test ERROR on rdma/master]
[also build test ERROR on v4.11-rc6 next-20170413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Leon-Romanovsky/IB-mlx5-Support-congestion-related-counters/20170413-231953 base: https://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git master config: i386-allyesconfig (attached as .config) compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce: # save the attached .config to linux build tree make ARCH=i386
All errors (new ones prefixed by >>):
>> drivers/infiniband//hw/mlx5/main.c:3172:120: error: 'struct mlx5_ifc_query_cong_statistics_out_bits' has no member named 'np_cnp_sent_high'; did you mean 'np_cnps_sent_high'?
INIT_CONG_COUNTER(np_cnp_sent),
^ vim +3172 drivers/infiniband//hw/mlx5/main.c
3166 MLX5_BYTE_OFF(query_cong_statistics_out, _name ## _high)}
3167
3168 static const struct mlx5_ib_counter cong_cnts[] = {
3169 INIT_CONG_COUNTER(rp_cnp_ignored),
3170 INIT_CONG_COUNTER(rp_cnp_handled),
3171 INIT_CONG_COUNTER(np_ecn_marked_roce_packets),
> 3172 INIT_CONG_COUNTER(np_cnp_sent),
3173 };
3174
3175 static void mlx5_ib_dealloc_counters(struct mlx5_ib_dev *dev)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
A message that is not in the archive (sent off-list or not yet received): 201704140302.3OiAKJBn%[email protected]
Quoted 6 lines · kbuild test robot
Hi Parav,
[auto build test ERROR on rdma/master]
[also build test ERROR on v4.11-rc6 next-20170413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
It looks like that I sent wrong patch again
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=testing/queue-next&id=5d94819cda09641954c002b8c11d50965cd230c6
I'll do my homework to find what broke in my scripts and flow and resubmit.
Thanks