ODP information and statistics
Subsystems (2)
v3 cover ODP information and statistics
From: Leon Romanovsky <[email protected]>
Changelog:
v3:
* Implement batch count of "invalidataions".
* "prefetched" ounter is dropped, will be separated to two separate counters (RQ/SQ) later once Jason's ODP rework will be finished.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Since umems can disappear during rereg flow and the fact that we
are not locking its during our counter usage (uverbs prevents this
by holding the uobject write lock), expose possible race bugs. Move
the counters into mlx5_ib_mr to avoid racing bugs (related patches
- #1, #3 & #4).
* Fix page invalidation counting (Patch #1).
* Make the code more elegant by defining fill_function type and use it within res_common_{dumpit, doit} (Patch #2).
* Put an ODP implicit indicator within mlx5 reg MR operation,
indicating when a given MR is ODP implicit registered and use
its indication when dumping ODP type.
* Since the counters has been moved to mlx5_ib_mr, the ODP stats are now filled with internal to mlx5 driver function. Remove the fill_odp_stats device operation from the reason mentioned above.
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Dropped umem patch, because it doesn't follow our IB model, where
UMEM is driver object and ib_core object (Jason).
* Removed the ODP type indicator from ib_umem_odp not needed after
commit fd7dbf035edc ("RDMA/odp: Make it clearer when a umem is an implicit ODP umem")
* Since umems are not part of core MR (from the reason mentioned
above) there is no way to access the odp type as was previously done via nldev (old patch #3). Instead, patch #4 is adding mlx5 implementation for fill_res_entry and dumping ODP type as part of the driver table entry, as its driver details.
* Counter types are now atomic64_t instead of u64.
v0: https://lore.kernel.org/linux-rdma/[email protected]
-----------------------------------------------------------------------------
Hi,
This series from Erez refactors exposes ODP type information (explicit, implicit) and statistics through netlink interface.
Thanks
Erez Alfasi (4):
IB/mlx5: Introduce ODP diagnostic counters RDMA/nldev: Allow different fill function per resource RDMA/mlx5: Return ODP type per MR
RDMA/nldev: Provide MR statistics
drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/nldev.c | 98 ++++++++++++++++++++------- drivers/infiniband/hw/mlx5/Makefile | 2 +- drivers/infiniband/hw/mlx5/main.c | 3 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +++ drivers/infiniband/hw/mlx5/odp.c | 17 +++++ drivers/infiniband/hw/mlx5/restrack.c | 90 ++++++++++++++++++++++++ include/rdma/ib_verbs.h | 12 ++++ include/rdma/restrack.h | 6 ++ 9 files changed, 212 insertions(+), 26 deletions(-) create mode 100644 drivers/infiniband/hw/mlx5/restrack.c
--
2.20.1
PATCH v3 2/4 RDMA/nldev: Allow different fill function per resource
From: Erez Alfasi <[email protected]>
So far res_get_common_{dumpit, doit} was using the default resource fill function which was defined as part of the nldev_fill_res_entry fill_entries.
Add a fill function pointer as an argument allows us to use different fill function in case we want to dump different values then 'rdma resource' flow do, but still use the same existing general resources dumping flow.
--- drivers/infiniband/core/nldev.c | 44 +++++++++++++++++---------------- 1 file changed, 23 insertions(+), 21 deletions(-)
drivers/infiniband/core/nldev.c +23 -21
@@ -42,6 +42,9 @@#include "cma_priv.h" #include "restrack.h" +typedef int (*res_fill_func_t )(struct sk_buff *,bool , +struct rdma_restrack_entry *,uint32_t ); +/* * Sort array elements by the netlink attribute name */
@@ -1128,8 +1131,6 @@ static int nldev_res_get_dumpit(struct sk_buff *skb, }struct nldev_fill_res_entry { -int (*fill_res_func)(struct sk_buff *msg,bool has_cap_net_admin, -struct rdma_restrack_entry *res,u32 port);enum rdma_nldev_attr nldev_attr;enum rdma_nldev_command nldev_cmd;u8 flags;
@@ -1143,21 +1144,18 @@ enum nldev_res_flags {static const struct nldev_fill_res_entry fill_entries[RDMA_RESTRACK_MAX ] = { [RDMA_RESTRACK_QP ] = { - .fill_res_func = fill_res_qp_entry, .nldev_cmd =RDMA_NLDEV_CMD_RES_QP_GET , .nldev_attr =RDMA_NLDEV_ATTR_RES_QP , .entry =RDMA_NLDEV_ATTR_RES_QP_ENTRY , .id =RDMA_NLDEV_ATTR_RES_LQPN , }, [RDMA_RESTRACK_CM_ID ] = { - .fill_res_func = fill_res_cm_id_entry, .nldev_cmd =RDMA_NLDEV_CMD_RES_CM_ID_GET , .nldev_attr =RDMA_NLDEV_ATTR_RES_CM_ID , .entry =RDMA_NLDEV_ATTR_RES_CM_ID_ENTRY , .id =RDMA_NLDEV_ATTR_RES_CM_IDN , }, [RDMA_RESTRACK_CQ ] = { - .fill_res_func = fill_res_cq_entry, .nldev_cmd =RDMA_NLDEV_CMD_RES_CQ_GET , .nldev_attr =RDMA_NLDEV_ATTR_RES_CQ , .flags =NLDEV_PER_DEV ,
@@ -1165,7 +1163,6 @@ static const struct nldev_fill_res_entry fill_entries[RDMA_RESTRACK_MAX] = { .id = RDMA_NLDEV_ATTR_RES_CQN, }, [RDMA_RESTRACK_MR] = { - .fill_res_func = fill_res_mr_entry, .nldev_cmd = RDMA_NLDEV_CMD_RES_MR_GET, .nldev_attr = RDMA_NLDEV_ATTR_RES_MR, .flags = NLDEV_PER_DEV,
@@ -1173,7 +1170,6 @@ static const struct nldev_fill_res_entry fill_entries[RDMA_RESTRACK_MAX] = { .id = RDMA_NLDEV_ATTR_RES_MRN, }, [RDMA_RESTRACK_PD] = { - .fill_res_func = fill_res_pd_entry, .nldev_cmd = RDMA_NLDEV_CMD_RES_PD_GET, .nldev_attr = RDMA_NLDEV_ATTR_RES_PD, .flags = NLDEV_PER_DEV,
@@ -1181,7 +1177,6 @@ static const struct nldev_fill_res_entry fill_entries[RDMA_RESTRACK_MAX] = { .id = RDMA_NLDEV_ATTR_RES_PDN, }, [RDMA_RESTRACK_COUNTER] = { - .fill_res_func = fill_res_counter_entry, .nldev_cmd = RDMA_NLDEV_CMD_STAT_GET, .nldev_attr = RDMA_NLDEV_ATTR_STAT_COUNTER, .entry = RDMA_NLDEV_ATTR_STAT_COUNTER_ENTRY,
@@ -1191,7 +1186,8 @@ static const struct nldev_fill_res_entry fill_entries[RDMA_RESTRACK_MAX] = {static int res_get_common_doit (struct sk_buff *skb,struct nlmsghdr *nlh,struct netlink_ext_ack *extack, -enum rdma_restrack_type res_type) +enum rdma_restrack_type res_type, +res_fill_func_t fill_func) {const struct nldev_fill_res_entry *fe = &fill_entries[res_type];struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ];
@@ -1249,7 +1245,9 @@ static int res_get_common_doit(struct sk_buff *skb, struct nlmsghdr *nlh, } has_cap_net_admin =netlink_capable (skb,CAP_NET_ADMIN ); - ret = fe->fill_res_func (msg, has_cap_net_admin, res, port); + + ret =fill_func (msg, has_cap_net_admin, res, port); +rdma_restrack_put (res);if (ret) goto err_free;
@@ -1269,7 +1267,8 @@ static int res_get_common_doit(struct sk_buff *skb, struct nlmsghdr *nlh,static int res_get_common_dumpit (struct sk_buff *skb,struct netlink_callback *cb, -enum rdma_restrack_type res_type) +enum rdma_restrack_type res_type, +res_fill_func_t fill_func) {const struct nldev_fill_res_entry *fe = &fill_entries[res_type];struct nlattr *tb[RDMA_NLDEV_ATTR_MAX ];
@@ -1354,7 +1353,8 @@ static int res_get_common_dumpit(struct sk_buff *skb, goto msg_full; } - ret = fe->fill_res_func (skb, has_cap_net_admin, res, port); + ret =fill_func (skb, has_cap_net_admin, res, port); +rdma_restrack_put (res);if (ret) {
@@ -1397,17 +1397,19 @@ next: idx++;return ret; } -#define RES_GET_FUNCS (name, type) \ - static int nldev_res_get_##name##_dumpit(struct sk_buff *skb, \ +#define RES_GET_FUNCS (name, type) \ + static int nldev_res_get_##name##_dumpit(struct sk_buff *skb, \ struct netlink_callback *cb) \ - { \ - return res_get_common_dumpit(skb, cb, type); \ - } \ - static int nldev_res_get_##name##_doit(struct sk_buff *skb, \ - struct nlmsghdr *nlh, \ + { \ + return res_get_common_dumpit(skb, cb, type, \ + fill_res_##name##_entry); \ + } \ + static int nldev_res_get_##name##_doit(struct sk_buff *skb, \ + struct nlmsghdr *nlh, \ struct netlink_ext_ack *extack) \ - { \ - return res_get_common_doit(skb, nlh, extack, type); \ + { \ + return res_get_common_doit(skb, nlh, extack, type, \ + fill_res_##name##_entry); \ }RES_GET_FUNCS (qp,RDMA_RESTRACK_QP );
--
2.20.1
PATCH v3 3/4 RDMA/mlx5: Return ODP type per MR
In reply to Leon RomanovskyFrom: Erez Alfasi <[email protected]>
Provide an ODP explicit/implicit type as part
of 'rdma -dd resource show mr' dump.
For example:
~$: rdma -dd resource show mr
dev mlx5_0 mrn 1 rkey 0xa99a lkey 0xa99a mrlen 50000000 pdn 9 pid 7372 comm ibv_rc_pingpong drv_odp explicit
For non-ODP MRs, we won't print "drv_odp ..." at all.
--- drivers/infiniband/core/nldev.c | 13 ++++++++ drivers/infiniband/hw/mlx5/Makefile | 2 +- drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 ++ drivers/infiniband/hw/mlx5/odp.c | 2 ++ drivers/infiniband/hw/mlx5/restrack.c | 48 +++++++++++++++++++++++++++ include/rdma/restrack.h | 3 ++ 7 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/hw/mlx5/restrack.c
drivers/infiniband/core/nldev.c +13 -0
@@ -183,6 +183,19 @@ static int _rdma_nl_put_driver_u64(struct sk_buff *msg, const char *name,return 0 ; } +int rdma_nl_put_driver_string (struct sk_buff *msg,const char *name, +const char *str) +{ +if (put_driver_name_print_type (msg, name, +RDMA_NLDEV_PRINT_TYPE_UNSPEC )) +return -EMSGSIZE ; +if (nla_put_string (msg,RDMA_NLDEV_ATTR_DRIVER_STRING , str)) +return -EMSGSIZE ; + +return 0 ; +} +EXPORT_SYMBOL (rdma_nl_put_driver_string); +int rdma_nl_put_driver_u32 (struct sk_buff *msg,const char *name,u32 value) { return _rdma_nl_put_driver_u32(msg, name, RDMA_NLDEV_PRINT_TYPE_UNSPEC,
drivers/infiniband/hw/mlx5/Makefile +1 -1
@@ -3,7 +3,7 @@ obj-$(CONFIG_MLX5_INFINIBAND) += mlx5_ib.o mlx5_ib-y := main.o cq.o doorbell.o qp.o mem.o srq_cmd.o \ srq.o mr.o ah.o mad.o gsi.o ib_virt.o cmd.o \ - cong.o + cong.o restrack.o mlx5_ib-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += odp.o mlx5_ib-$(CONFIG_MLX5_ESWITCH) += ib_rep.o mlx5_ib-$(CONFIG_INFINIBAND_USER_ACCESS) += devx.o
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -6269,6 +6269,7 @@ static const struct ib_device_ops mlx5_ib_dev_ops = { .disassociate_ucontext = mlx5_ib_disassociate_ucontext, .drain_rq = mlx5_ib_drain_rq, .drain_sq = mlx5_ib_drain_sq, + .fill_res_entry = mlx5_ib_fill_res_entry, .get_dev_fw_str = get_dev_fw_str, .get_dma_mr = mlx5_ib_get_dma_mr, .get_link_layer = mlx5_ib_port_link_layer,
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -0
@@ -626,6 +626,7 @@ struct mlx5_ib_mr {struct mlx5_async_work cb_work;atomic_t num_pending_prefetch;struct ib_odp_counters odp_stats; +bool is_odp_implicit; };static inline bool is_odp_mr (struct mlx5_ib_mr *mr)
@@ -1339,6 +1340,8 @@ struct mlx5_core_dev *mlx5_ib_get_native_port_mdev(struct mlx5_ib_dev *dev, u8 *native_port_num);void mlx5_ib_put_native_port_mdev (struct mlx5_ib_dev *dev,u8 port_num); +int mlx5_ib_fill_res_entry (struct sk_buff *msg, +struct rdma_restrack_entry *res);#if IS_ENABLED (CONFIG_INFINIBAND_USER_ACCESS )int mlx5_ib_devx_create (struct mlx5_ib_dev *dev,bool is_user);
drivers/infiniband/hw/mlx5/odp.c +2 -0
@@ -543,6 +543,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,atomic_set (&imr->num_leaf_free,0 );atomic_set (&imr->num_pending_prefetch,0 ); + imr->is_odp_implicit = true; +return imr; }
drivers/infiniband/hw/mlx5/restrack.c new file +48 -0
48 lines not on this page · raw message
include/rdma/restrack.h +3 -0
@@ -156,6 +156,9 @@ int rdma_nl_put_driver_u32_hex(struct sk_buff *msg, const char *name,int rdma_nl_put_driver_u64 (struct sk_buff *msg,const char *name,u64 value);int rdma_nl_put_driver_u64_hex (struct sk_buff *msg,const char *name,u64 value); +int rdma_nl_put_driver_string (struct sk_buff *msg,const char *name, +const char *str); +struct rdma_restrack_entry *rdma_restrack_get_byid (struct ib_device *dev,enum rdma_restrack_type type,u32 id);
--
2.20.1
PATCH v3 4/4 RDMA/nldev: Provide MR statistics
In reply to Leon RomanovskyFrom: Erez Alfasi <[email protected]>
Add RDMA nldev netlink interface for dumping MR
statistics information.
Output example:
ereza@dev~$: ./ibv_rc_pingpong -o -P -s 500000000
local address: LID 0x0001, QPN 0x00008a, PSN 0xf81096, GID ::
ereza@dev~$: rdma stat show mr
dev mlx5_0 mrn 2 page_faults 122071 page_invalidations 0
--- drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/nldev.c | 41 +++++++++++++++++++++++--- drivers/infiniband/hw/mlx5/main.c | 2 ++ drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 ++ drivers/infiniband/hw/mlx5/restrack.c | 42 +++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 7 +++++ include/rdma/restrack.h | 3 ++ 7 files changed, 94 insertions(+), 4 deletions(-)
drivers/infiniband/core/device.c +1 -0
@@ -2606,6 +2606,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, drain_sq);SET_DEVICE_OP (dev_ops, enable_driver);SET_DEVICE_OP (dev_ops, fill_res_entry); +SET_DEVICE_OP (dev_ops, fill_stat_entry);SET_DEVICE_OP (dev_ops, get_dev_fw_str);SET_DEVICE_OP (dev_ops, get_dma_mr);SET_DEVICE_OP (dev_ops, get_hw_stats);
drivers/infiniband/core/nldev.c +37 -4
71 lines not on this page · raw message
drivers/infiniband/hw/mlx5/main.c +2 -0
@@ -67,6 +67,7 @@#include <rdma/uverbs_std_types.h> #include <rdma/mlx5_user_ioctl_verbs.h> #include <rdma/mlx5_user_ioctl_cmds.h> +#include <rdma/ib_umem_odp.h> #define UVERBS_MODULE_NAME mlx5_ib#include <rdma/uverbs_named_ioctl.h>
@@ -6270,6 +6271,7 @@ static const struct ib_device_ops mlx5_ib_dev_ops = { .drain_rq = mlx5_ib_drain_rq, .drain_sq = mlx5_ib_drain_sq, .fill_res_entry = mlx5_ib_fill_res_entry, + .fill_stat_entry = mlx5_ib_fill_stat_entry, .get_dev_fw_str = get_dev_fw_str, .get_dma_mr = mlx5_ib_get_dma_mr, .get_link_layer = mlx5_ib_port_link_layer,
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
@@ -1342,6 +1342,8 @@ void mlx5_ib_put_native_port_mdev(struct mlx5_ib_dev *dev, u8 port_num);int mlx5_ib_fill_res_entry (struct sk_buff *msg,struct rdma_restrack_entry *res); +int mlx5_ib_fill_stat_entry (struct sk_buff *msg, +struct rdma_restrack_entry *res);#if IS_ENABLED (CONFIG_INFINIBAND_USER_ACCESS )int mlx5_ib_devx_create (struct mlx5_ib_dev *dev,bool is_user);
drivers/infiniband/hw/mlx5/restrack.c +42 -0
51 lines not on this page · raw message
include/rdma/ib_verbs.h +7 -0
@@ -2568,6 +2568,13 @@ struct ib_device_ops {*/ int (*counter_update_stats)(struct rdma_counter *counter); +/** +* Allows rdma drivers to add their own restrack attributes +* dumped via 'rdma stat' iproute2 command. +*/ +int (*fill_stat_entry)(struct sk_buff *msg, +struct rdma_restrack_entry *entry); +DECLARE_RDMA_OBJ_SIZE (ib_ah);DECLARE_RDMA_OBJ_SIZE (ib_cq);DECLARE_RDMA_OBJ_SIZE (ib_pd);
include/rdma/restrack.h +3 -0
@@ -162,4 +162,7 @@ int rdma_nl_put_driver_string(struct sk_buff *msg, const char *name,struct rdma_restrack_entry *rdma_restrack_get_byid (struct ib_device *dev,enum rdma_restrack_type type,u32 id); +int fill_stat_hwcounter_entry (struct sk_buff *msg, +const char *name,u64 value); +#endif /* _RDMA_RESTRACK_H_ */
--
2.20.1
PATCH v3 1/4 IB/mlx5: Introduce ODP diagnostic counters
In reply to Leon RomanovskyFrom: Erez Alfasi <[email protected]>
Introduce ODP diagnostic counters and count the following per MR within IB/mlx5 driver:
1) Page faults:
Total number of faulted pages.
2) Page invalidations:
Total number of pages invalidated by the OS during all invalidation events. The translations can be no longer valid due to either non-present pages or mapping changes.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 4 ++++ drivers/infiniband/hw/mlx5/odp.c | 15 +++++++++++++++ include/rdma/ib_verbs.h | 5 +++++ 3 files changed, 24 insertions(+)
drivers/infiniband/hw/mlx5/mlx5_ib.h +4 -0
@@ -585,6 +585,9 @@ struct mlx5_ib_dm { IB_ACCESS_REMOTE_READ |\ IB_ZERO_BASED) +#define mlx5_update_odp_stats (mr, counter_name, value) \ + atomic64_add(value, &((mr)->odp_stats.counter_name)) +struct mlx5_ib_mr {struct ib_mr ibmr;void *descs;
@@ -622,6 +625,7 @@ struct mlx5_ib_mr {wait_queue_head_t q_leaf_free;struct mlx5_async_work cb_work;atomic_t num_pending_prefetch; +struct ib_odp_counters odp_stats; };static inline bool is_odp_mr (struct mlx5_ib_mr *mr)
drivers/infiniband/hw/mlx5/odp.c +15 -0
@@ -224,6 +224,7 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start,const u64 umr_block_mask = (MLX5_UMR_MTT_ALIGNMENT /sizeof (struct mlx5_mtt )) -1 ;u64 idx =0 , blk_start_idx =0 ; +u64 invalidations =0 ;int in_block =0 ;u64 addr;
@@ -261,6 +262,9 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start, blk_start_idx = idx; in_block =1 ; } + +/* Count page invalidations */ + invalidations += idx - blk_start_idx +1 ; }else {u64 umr_offset = idx & umr_block_mask;
@@ -279,6 +283,9 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start,MLX5_IB_UPD_XLT_ZAP | MLX5_IB_UPD_XLT_ATOMIC);mutex_unlock (&umem_odp->umem_mutex); + +mlx5_update_odp_stats (mr, invalidations, invalidations); +/* * We are now sure that the device will not access the * memory. We can safely unmap it, and mark it as dirty if
@@ -287,6 +294,7 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start,ib_umem_odp_unmap_dma_pages (umem_odp, start, end); +if (unlikely (!umem_odp->npages && mr->parent && !umem_odp->dying)) {WRITE_ONCE (umem_odp->dying,1 );
@@ -801,6 +809,13 @@ static int pagefault_single_data_segment(struct mlx5_ib_dev *dev,if (ret <0 ) goto srcu_unlock; +/* +* When prefetching a page, page fault is generated +* in order to bring the page to the main memory. +* In the current flow, page faults are being counted. +*/ +mlx5_update_odp_stats (mr, faults, ret); + npages += ret; ret =0 ;break ;
include/rdma/ib_verbs.h +5 -0
@@ -2218,6 +2218,11 @@ struct rdma_netdev_alloc_params { struct net_device *netdev, void *param); }; +struct ib_odp_counters { +atomic64_t faults; +atomic64_t invalidations; +}; +struct ib_counters {struct ib_device *device;struct ib_uobject *uobject;
--
2.20.1
Re: [PATCH rdma-next v3 4/4] RDMA/nldev: Provide MR statistics
In reply to Leon Romanovskydrivers/infiniband/core/nldev.c static int fill_stat_hwcounter_entry(struct sk_buff *msg, · line 784
Quoted 5 lines · Leon Romanovsky
drivers/infiniband/core/nldev.c +1
-0@@ -773,6 +781,25 @@ static int fill_stat_hwcounter_entry(struct sk_buff *msg,nla_nest_cancel (msg, entry_attr);return -EMSGSIZE ; } +EXPORT_SYMBOL (fill_stat_hwcounter_entry);
This is not a good name for a global symbol, I changed it to rdma_nl_stat_hwcounter_entry
Jason
Jason Gunthorpe
AppliedQuoted 5 lines · Leon Romanovsky
Erez Alfasi (4):
IB/mlx5: Introduce ODP diagnostic counters RDMA/nldev: Allow different fill function per resource RDMA/mlx5: Return ODP type per MR
RDMA/nldev: Provide MR statistics
applied to for-next, thanks
Jason