Query GID table API
Subsystems (3)
PATCH 3/4 RDMA/core: Introduce new GID table query API
In reply to Leon RomanovskyFrom: Avihai Horon <[email protected]>
Introduce rdma_query_gid_table which enables querying all the GID tables of a given device and copying the attributes of all valid GID entries to a provided buffer.
This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
--- drivers/infiniband/core/cache.c | 93 +++++++++++++++++++++++++ include/rdma/ib_cache.h | 5 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 8 +++ 3 files changed, 106 insertions(+)
drivers/infiniband/core/cache.c +93 -0
@@ -1247,6 +1247,99 @@ rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) }EXPORT_SYMBOL (rdma_get_gid_attr); +/** +* rdma_get_ndev_ifindex - Reads ndev ifindex of the given gid attr. +* @gid_attr: Pointer to the GID attribute. +* @ndev_ifindex: Pointer through which the ndev ifindex is returned. +* +* Returns 0 on success or appropriate error code. The netdevice must be in UP +* state. +*/ +int rdma_get_ndev_ifindex (const struct ib_gid_attr *gid_attr,u32 *ndev_ifindex) +{ +struct net_device *ndev; +int ret =0 ; + +if (rdma_protocol_ib (gid_attr->device, gid_attr->port_num)) { + *ndev_ifindex =0 ; +return 0 ; + } + +rcu_read_lock (); + ndev =rcu_dereference (gid_attr->ndev); +if (!ndev || (READ_ONCE (ndev->flags) &IFF_UP ) ==0 ) { + ret = -ENODEV ; + goto out; + } + + *ndev_ifindex = ndev->ifindex; +out: +rcu_read_unlock (); +return ret; +} +EXPORT_SYMBOL (rdma_get_ndev_ifindex); + +/** +* rdma_query_gid_table - Reads GID table entries of all the ports of a device up to max_entries. +* @device: The device to query. +* @entries: Entries where GID entries are returned. +* @max_entries: Maximum number of entries that can be returned. +* Entries array must be allocated to hold max_entries number of entries. +* @num_entries: Updated to the number of entries that were successfully read. +* +* Returns 0 on success or appropriate error code. +*/ +int rdma_query_gid_table (struct ib_device *device, +struct ib_uverbs_gid_entry *entries, +size_t max_entries,size_t *num_entries) +{ +const struct ib_gid_attr *gid_attr; +struct ib_gid_table *table; +unsigned int port_num; +unsigned long flags; +int ret; +int i; + + *num_entries =0 ; +rdma_for_each_port (device, port_num) { +if (!rdma_ib_or_roce (device, port_num)) +continue ; + + table =rdma_gid_table (device, port_num); +read_lock_irqsave (&table->rwlock, flags); +for (i =0 ; i < table->sz; i++) { +if (!is_gid_entry_valid (table->data_vec[i])) +continue ; +if (*num_entries >= max_entries) { + ret = -EINVAL ; + goto err; + } + + gid_attr = &table->data_vec[i]->attr; + +memcpy (&entries->gid, &gid_attr->gid, +sizeof (gid_attr->gid)); + entries->gid_index = gid_attr->index; + entries->port_num = gid_attr->port_num; + entries->gid_type = gid_attr->gid_type; + ret =rdma_get_ndev_ifindex (gid_attr, + &entries->netdev_ifindex); +if (ret) + goto err; + + (*num_entries)++; + entries++; + } +read_unlock_irqrestore (&table->rwlock, flags); + } + +return 0 ; +err: +read_unlock_irqrestore (&table->rwlock, flags); +return ret; +} +EXPORT_SYMBOL (rdma_query_gid_table); +/** * rdma_put_gid_attr - Release reference to the GID attribute * @attr: Pointer to the GID attribute whose reference
include/rdma/ib_cache.h +5 -0
@@ -110,5 +110,10 @@ const struct ib_gid_attr *rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index);void rdma_put_gid_attr (const struct ib_gid_attr *attr);void rdma_hold_gid_attr (const struct ib_gid_attr *attr); +int rdma_get_ndev_ifindex (const struct ib_gid_attr *gid_attr, +u32 *ndev_ifindex); +int rdma_query_gid_table (struct ib_device *device, +struct ib_uverbs_gid_entry *entries, +size_t max_entries,size_t *num_entries);#endif /* _IB_CACHE_H */
include/uapi/rdma/ib_user_ioctl_verbs.h +8 -0
@@ -250,4 +250,12 @@ enum rdma_driver_id { RDMA_DRIVER_SIW, }; +struct ib_uverbs_gid_entry { +__aligned_u64 gid[2 ]; +__u32 gid_index; +__u32 port_num; +__u32 gid_type; +__u32 netdev_ifindex;/* It is 0 if there is no netdev associated with it */ +}; +#endif
Signature · 1 line
2.26.2
cover Query GID table API
From: Leon Romanovsky <[email protected]>
From Avihai,
When an application is not using RDMA CM and if it is using multiple RDMA devices with one or more RoCE ports, finding the right GID table entry is a long process.
For example, with two RoCE dual-port devices in a system, when IP failover is used between two RoCE ports, searching a suitable GID entry for a given source IP, matching netdevice of given RoCEv1/v2 type requires iterating over all 4 ports * 256 entry GID table.
Even though the best first match GID table for given criteria is used, when the matching entry is on the 4th port, it requires reading 3 ports * 256 entries * 3 files (GID, netdev, type) = 2304 files.
The GID table needs to be referred on every QP creation during IP failover on other netdevice of an RDMA device.
In an alternative approach, a GID cache may be maintained and updated on GID change event was reported by the kernel. However, it comes with below two limitations:
(a) Maintain a thread per application process instance to listen and update the cache.
(b) Without the thread, on cache miss event, query the GID table. Even in this approach, if multiple processes are used, a GID cache needs to be maintained on a per-process basis. With a large number of processes, this method doesn't scale.
Hence, we introduce this series of patches, which introduces an API to query the complete GID tables of an RDMA device, that returns all valid GID table entries.
This is done through single ioctl, eliminating 2304 read, 2304 open and 2304 close system calls to just a total of 2 calls (one for each device).
While at it, we also introduce an API to query an individual GID entry over ioctl interface, which provides all GID attributes information.
Thanks
Avihai Horon (4):
RDMA/core: Change rdma_get_gid_attr returned error code RDMA/core: Modify enum ib_gid_type and enum rdma_network_type RDMA/core: Introduce new GID table query API RDMA/uverbs: Expose the new GID query API to user space
drivers/infiniband/core/cache.c | 99 +++++++++- drivers/infiniband/core/cma.c | 10 - drivers/infiniband/core/cma_configfs.c | 13 +- drivers/infiniband/core/cma_priv.h | 10 + drivers/infiniband/core/sysfs.c | 3 +- .../infiniband/core/uverbs_std_types_device.c | 180 +++++++++++++++++- drivers/infiniband/core/verbs.c | 2 +- drivers/infiniband/hw/mlx5/cq.c | 2 +- drivers/infiniband/hw/mlx5/main.c | 4 +- drivers/infiniband/hw/qedr/verbs.c | 4 +- include/rdma/ib_cache.h | 5 + include/rdma/ib_verbs.h | 19 +- include/uapi/rdma/ib_user_ioctl_cmds.h | 16 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 14 ++ 14 files changed, 351 insertions(+), 30 deletions(-)
--
2.26.2
PATCH 2/4 RDMA/core: Modify enum ib_gid_type and enum rdma_network_type
From: Avihai Horon <[email protected]>
Separate IB_GID_TYPE_IB and IB_GID_TYPE_ROCE to two different values, so enum ib_gid_type will match the gid types of the new query GID table API which will be introduced in the following patches.
This change in enum ib_gid_type requires to change also enum rdma_network_type by separating RDMA_NETWORK_IB and RDMA_NETWORK_ROCE_V1 values.
--- drivers/infiniband/core/cache.c | 4 ++++ drivers/infiniband/core/cma.c | 10 ---------- drivers/infiniband/core/cma_configfs.c | 13 +++++++++---- drivers/infiniband/core/cma_priv.h | 10 ++++++++++ drivers/infiniband/core/verbs.c | 2 +- drivers/infiniband/hw/mlx5/cq.c | 2 +- drivers/infiniband/hw/mlx5/main.c | 4 ++-- drivers/infiniband/hw/qedr/verbs.c | 4 +++- include/rdma/ib_verbs.h | 17 ++++++++++------- 9 files changed, 40 insertions(+), 26 deletions(-)
drivers/infiniband/core/cache.c +4 -0
@@ -133,7 +133,11 @@ static void dispatch_gid_change_event(struct ib_device *ib_dev, u8 port) }static const char *const gid_type_str[] = { +/* IB/RoCE v1 value is set for IB_GID_TYPE_IB and IB_GID_TYPE_ROCE for +* user space compatibility reasons. +*/ [IB_GID_TYPE_IB ] ="IB/RoCE v1" , + [IB_GID_TYPE_ROCE ] ="IB/RoCE v1" , [IB_GID_TYPE_ROCE_UDP_ENCAP ] ="RoCE v2" , };
drivers/infiniband/core/cma.c +0 -10
@@ -203,16 +203,6 @@ struct xarray *cma_pernet_xa(struct net *net, enum rdma_ucm_port_space ps) } } -struct cma_device { -struct list_head list; -struct ib_device *device; -struct completion comp; -refcount_t refcount; -struct list_head id_list; -enum ib_gid_type *default_gid_type; -u8 *default_roce_tos; -}; -struct rdma_bind_list {enum rdma_ucm_port_space ps;struct hlist_head owners;
drivers/infiniband/core/cma_configfs.c +9 -4
@@ -123,16 +123,21 @@ static ssize_t default_roce_mode_store(struct config_item *item, {struct cma_device *cma_dev;struct cma_dev_port_group *group; -int gid_type =ib_cache_gid_parse_type_str (buf); +int gid_type;ssize_t ret; -if (gid_type <0 ) -return -EINVAL ; - ret =cma_configfs_params_get (item, &cma_dev, &group);if (ret)return ret; + gid_type =ib_cache_gid_parse_type_str (buf); +if (gid_type <0 ) +return -EINVAL ; + +if (gid_type ==IB_GID_TYPE_IB && +rdma_protocol_roce_eth_encap (cma_dev->device, group->port_num)) + gid_type =IB_GID_TYPE_ROCE ; + ret =cma_set_default_gid_type (cma_dev, group->port_num, gid_type);cma_configfs_params_put (cma_dev);
drivers/infiniband/core/cma_priv.h +10 -0
@@ -50,6 +50,16 @@ enum rdma_cm_state {RDMA_CM_DESTROYING }; +struct cma_device { +struct list_head list; +struct ib_device *device; +struct completion comp; +refcount_t refcount; +struct list_head id_list; +enum ib_gid_type *default_gid_type; +u8 *default_roce_tos; +}; +struct rdma_id_private {struct rdma_cm_id id;
drivers/infiniband/core/verbs.c +1 -1
@@ -740,7 +740,7 @@ int ib_get_gids_from_rdma_hdr(const union rdma_network_hdr *hdr, (struct in6_addr *)dgid);return 0 ; }else if (net_type ==RDMA_NETWORK_IPV6 || - net_type ==RDMA_NETWORK_IB ) { + net_type ==RDMA_NETWORK_IB ||RDMA_NETWORK_ROCE_V1 ) { *dgid = hdr->ibgrh.dgid; *sgid = hdr->ibgrh.sgid;return 0 ;
drivers/infiniband/hw/mlx5/cq.c +1 -1
@@ -255,7 +255,7 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe,switch (roce_packet_type) {case MLX5_CQE_ROCE_L3_HEADER_TYPE_GRH : - wc->network_hdr_type =RDMA_NETWORK_IB ; + wc->network_hdr_type =RDMA_NETWORK_ROCE_V1 ;break ;case MLX5_CQE_ROCE_L3_HEADER_TYPE_IPV6 : wc->network_hdr_type =RDMA_NETWORK_IPV6 ;
drivers/infiniband/hw/mlx5/main.c +2 -2
@@ -546,7 +546,7 @@ static int set_roce_addr(struct mlx5_ib_dev *dev, u8 port_num, unsigned int index, const union ib_gid *gid, const struct ib_gid_attr *attr) { - enum ib_gid_type gid_type = IB_GID_TYPE_IB; + enum ib_gid_type gid_type = IB_GID_TYPE_ROCE;u16 vlan_id =0xffff ;u8 roce_version =0 ;u8 roce_l3_type =0 ;
@@ -561,7 +561,7 @@ static int set_roce_addr(struct mlx5_ib_dev *dev, u8 port_num, }switch (gid_type) { -case IB_GID_TYPE_IB : +case IB_GID_TYPE_ROCE : roce_version =MLX5_ROCE_VERSION_1 ;break ;case IB_GID_TYPE_ROCE_UDP_ENCAP :
drivers/infiniband/hw/qedr/verbs.c +3 -1
@@ -1138,7 +1138,7 @@ static inline int get_gid_info_from_table(struct ib_qp *ibqp,SET_FIELD (qp_params->modify_flags,QED_ROCE_MODIFY_QP_VALID_ROCE_MODE ,1 );break ; -case RDMA_NETWORK_IB : +case RDMA_NETWORK_ROCE_V1 :memcpy (&qp_params->sgid.bytes[0 ], &gid_attr->gid.raw[0 ],sizeof (qp_params->sgid)); memcpy(&qp_params->dgid.bytes[0],
@@ -1158,6 +1158,8 @@ static inline int get_gid_info_from_table(struct ib_qp *ibqp, QED_ROCE_MODIFY_QP_VALID_ROCE_MODE, 1); qp_params->roce_mode =ROCE_V2_IPV4 ;break ; +default : +return -EINVAL ; }for (i =0 ; i <4 ; i++) {
include/rdma/ib_verbs.h +10 -7
@@ -138,10 +138,9 @@ union ib_gid {extern union ib_gid zgid;enum ib_gid_type { -/* If link layer is Ethernet, this is RoCE V1 */ IB_GID_TYPE_IB =0 , -IB_GID_TYPE_ROCE =0 , -IB_GID_TYPE_ROCE_UDP_ENCAP =1 , +IB_GID_TYPE_ROCE =1 , +IB_GID_TYPE_ROCE_UDP_ENCAP =2 ,IB_GID_TYPE_SIZE };
@@ -180,7 +179,7 @@ rdma_node_get_transport(unsigned int node_type);enum rdma_network_type {RDMA_NETWORK_IB , -RDMA_NETWORK_ROCE_V1 =RDMA_NETWORK_IB , +RDMA_NETWORK_ROCE_V1 ,RDMA_NETWORK_IPV4 ,RDMA_NETWORK_IPV6 };
@@ -190,9 +189,10 @@ static inline enum ib_gid_type ib_network_to_gid_type(enum rdma_network_type netif (network_type ==RDMA_NETWORK_IPV4 || network_type ==RDMA_NETWORK_IPV6 )return IB_GID_TYPE_ROCE_UDP_ENCAP ; - -/* IB_GID_TYPE_IB same as RDMA_NETWORK_ROCE_V1 */ -return IB_GID_TYPE_IB ; +else if (network_type ==RDMA_NETWORK_ROCE_V1 ) +return IB_GID_TYPE_ROCE ; +else +return IB_GID_TYPE_IB ; } static inline enum rdma_network_type
@@ -201,6 +201,9 @@ rdma_gid_attr_network_type(const struct ib_gid_attr *attr)if (attr->gid_type ==IB_GID_TYPE_IB )return RDMA_NETWORK_IB ; +if (attr->gid_type ==IB_GID_TYPE_ROCE ) +return RDMA_NETWORK_ROCE_V1 ; +if (ipv6_addr_v4mapped ((struct in6_addr *)&attr->gid))return RDMA_NETWORK_IPV4 ;else
Signature · 1 line
2.26.2
PATCH 4/4 RDMA/uverbs: Expose the new GID query API to user space
In reply to Leon RomanovskyFrom: Avihai Horon <[email protected]>
Expose the query GID table and entry API to user space by adding two new methods and method handlers to the device object.
This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
--- .../infiniband/core/uverbs_std_types_device.c | 180 +++++++++++++++++- include/rdma/ib_verbs.h | 6 +- include/uapi/rdma/ib_user_ioctl_cmds.h | 16 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 6 + 4 files changed, 204 insertions(+), 4 deletions(-)
drivers/infiniband/core/uverbs_std_types_device.c +179 -1
@@ -8,6 +8,7 @@#include "uverbs.h" #include <rdma/uverbs_ioctl.h> #include <rdma/opa_addr.h> +#include <rdma/ib_cache.h> /* * This ioctl method allows calling any defined write or write_ex
@@ -266,6 +267,157 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)(return ucontext->device->ops.query_ucontext (ucontext, attrs); } +static int copy_gid_entries_to_user (struct uverbs_attr_bundle *attrs, +struct ib_uverbs_gid_entry *entries, +size_t num_entries,size_t user_entry_size) +{ +const struct uverbs_attr *attr; +void __user *user_entries; +size_t copy_len; +int ret; +int i; + +if (user_entry_size ==sizeof (*entries)) { + ret =uverbs_copy_to (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , + entries,sizeof (*entries) * num_entries); +return ret; + } + + copy_len =min_t (size_t, user_entry_size,sizeof (*entries)); + attr =uverbs_attr_get (attrs,UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +if (IS_ERR (attr)) +return PTR_ERR (attr); + + user_entries =u64_to_user_ptr (attr->ptr_attr.data); +for (i =0 ; i < num_entries; i++) { +if (copy_to_user (user_entries, entries, copy_len)) +return -EFAULT ; + +if (user_entry_size >sizeof (*entries)) { +if (clear_user (user_entries +sizeof (*entries), + user_entry_size -sizeof (*entries))) +return -EFAULT ; + } + + entries++; + user_entries += user_entry_size; + } + +return uverbs_output_written (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +} + +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_TABLE )( +struct uverbs_attr_bundle *attrs) +{ +struct ib_uverbs_gid_entry *entries; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +size_t user_entry_size; +size_t max_entries; +size_t num_entries; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&user_entry_size, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE ); +if (ret) +return ret; + + max_entries =uverbs_attr_ptr_get_array_size ( + attrs,UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , + user_entry_size); +if (max_entries <=0 ) +return -EINVAL ; + + ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device; + + entries =uverbs_zalloc (attrs, max_entries *sizeof (*entries)); +if (!entries) +return -ENOMEM ; + + ret =rdma_query_gid_table (ib_dev, entries, max_entries, &num_entries); +if (ret) +return ret; + + ret =copy_gid_entries_to_user (attrs, entries, num_entries, + user_entry_size); +if (ret) +return ret; + + ret =uverbs_copy_to (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_NUM_ENTRIES , + &num_entries,sizeof (num_entries)); +return ret; +} + +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_ENTRY )( +struct uverbs_attr_bundle *attrs) +{ +const struct ib_gid_attr *gid_attr; +struct ib_uverbs_gid_entry entry; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +u32 gid_index; +u32 port_num; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&port_num, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_PORT ); +if (ret) +return ret; + + ret =uverbs_get_const (&gid_index, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX ); +if (ret) +return ret; + + ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device; + +if (!rdma_is_port_valid (ib_dev, port_num)) +return -EINVAL ; + +if (!rdma_ib_or_roce (ib_dev, port_num)) +return -EINVAL ; + + gid_attr =rdma_get_gid_attr (ib_dev, port_num, gid_index); +if (IS_ERR (gid_attr)) +return PTR_ERR (gid_attr); + +memcpy (&entry.gid, &gid_attr->gid,sizeof (gid_attr->gid)); + entry.gid_index = gid_attr->index; + entry.port_num = gid_attr->port_num; + entry.gid_type = gid_attr->gid_type; + ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex); +if (ret) + goto out; + + ret =uverbs_copy_to_struct_or_zero ( + attrs,UVERBS_ATTR_QUERY_GID_ENTRY_RESP_ENTRY , &entry, +sizeof (entry)); +out: +rdma_put_gid_attr (gid_attr); +return ret; +} + DECLARE_UVERBS_NAMED_METHOD( UVERBS_METHOD_GET_CONTEXT, UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS,
@@ -300,12 +452,38 @@ DECLARE_UVERBS_NAMED_METHOD( reserved), UA_MANDATORY)); +DECLARE_UVERBS_NAMED_METHOD ( +UVERBS_METHOD_QUERY_GID_TABLE , +UVERBS_ATTR_CONST_IN (UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE , u64, +UA_MANDATORY ), +UVERBS_ATTR_FLAGS_IN (UVERBS_ATTR_QUERY_GID_TABLE_FLAGS , u32, +UA_OPTIONAL ), +UVERBS_ATTR_PTR_OUT (UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , +UVERBS_ATTR_MIN_SIZE (0 ),UA_MANDATORY ), +UVERBS_ATTR_PTR_OUT (UVERBS_ATTR_QUERY_GID_TABLE_RESP_NUM_ENTRIES , +UVERBS_ATTR_TYPE (u64),UA_MANDATORY )); + +DECLARE_UVERBS_NAMED_METHOD ( +UVERBS_METHOD_QUERY_GID_ENTRY , +UVERBS_ATTR_CONST_IN (UVERBS_ATTR_QUERY_GID_ENTRY_PORT , u32, +UA_MANDATORY ), +UVERBS_ATTR_CONST_IN (UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX , u32, +UA_MANDATORY ), +UVERBS_ATTR_FLAGS_IN (UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS , u32, +UA_MANDATORY ), +UVERBS_ATTR_PTR_OUT (UVERBS_ATTR_QUERY_GID_ENTRY_RESP_ENTRY , + UVERBS_ATTR_STRUCT(struct ib_uverbs_gid_entry, + netdev_ifindex), +UA_MANDATORY )); +DECLARE_UVERBS_GLOBAL_METHODS (UVERBS_OBJECT_DEVICE , &UVERBS_METHOD (UVERBS_METHOD_GET_CONTEXT ), &UVERBS_METHOD (UVERBS_METHOD_INVOKE_WRITE ), &UVERBS_METHOD (UVERBS_METHOD_INFO_HANDLES ), &UVERBS_METHOD (UVERBS_METHOD_QUERY_PORT ), - &UVERBS_METHOD (UVERBS_METHOD_QUERY_CONTEXT )); + &UVERBS_METHOD (UVERBS_METHOD_QUERY_CONTEXT ), + &UVERBS_METHOD (UVERBS_METHOD_QUERY_GID_TABLE ), + &UVERBS_METHOD (UVERBS_METHOD_QUERY_GID_ENTRY ));const struct uapi_definition uverbs_def_obj_device[] = {UAPI_DEF_CHAIN_OBJ_TREE_NAMED (UVERBS_OBJECT_DEVICE ),
include/rdma/ib_verbs.h +3 -3
@@ -138,9 +138,9 @@ union ib_gid {extern union ib_gid zgid;enum ib_gid_type { -IB_GID_TYPE_IB =0 , -IB_GID_TYPE_ROCE =1 , -IB_GID_TYPE_ROCE_UDP_ENCAP =2 , +IB_GID_TYPE_IB =IB_UVERBS_GID_TYPE_IB , +IB_GID_TYPE_ROCE =IB_UVERBS_GID_TYPE_ROCE_V1 , +IB_GID_TYPE_ROCE_UDP_ENCAP =IB_UVERBS_GID_TYPE_ROCE_V2 ,IB_GID_TYPE_SIZE };
include/uapi/rdma/ib_user_ioctl_cmds.h +16 -0
@@ -70,6 +70,8 @@ enum uverbs_methods_device {UVERBS_METHOD_QUERY_PORT ,UVERBS_METHOD_GET_CONTEXT ,UVERBS_METHOD_QUERY_CONTEXT , +UVERBS_METHOD_QUERY_GID_TABLE , + UVERBS_METHOD_QUERY_GID_ENTRY, }; enum uverbs_attrs_invoke_write_cmd_attr_ids {
@@ -352,4 +354,18 @@ enum uverbs_attrs_async_event_create { UVERBS_ATTR_ASYNC_EVENT_ALLOC_FD_HANDLE, }; +enum uverbs_attrs_query_gid_table_cmd_attr_ids { +UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE , +UVERBS_ATTR_QUERY_GID_TABLE_FLAGS , +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , +UVERBS_ATTR_QUERY_GID_TABLE_RESP_NUM_ENTRIES , +}; + +enum uverbs_attrs_query_gid_entry_cmd_attr_ids { +UVERBS_ATTR_QUERY_GID_ENTRY_PORT , +UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX , +UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS , +UVERBS_ATTR_QUERY_GID_ENTRY_RESP_ENTRY , +}; +#endif
include/uapi/rdma/ib_user_ioctl_verbs.h +6 -0
@@ -250,6 +250,12 @@ enum rdma_driver_id { RDMA_DRIVER_SIW, }; +enum ib_uverbs_gid_type { +IB_UVERBS_GID_TYPE_IB , +IB_UVERBS_GID_TYPE_ROCE_V1 , +IB_UVERBS_GID_TYPE_ROCE_V2 , +}; +struct ib_uverbs_gid_entry {__aligned_u64 gid[2 ];__u32 gid_index;
Signature · 1 line
2.26.2
PATCH 1/4 RDMA/core: Change rdma_get_gid_attr returned error code
In reply to Leon RomanovskyFrom: Avihai Horon <[email protected]>
Change the error code returned from rdma_get_gid_attr when the GID entry is invalid but the GID index is in the gid table size range to -ENODATA instead of -EINVAL.
This change is done in order to provide a more accurate error reporting to be used by the new GID query API in user space. Nevertheless, -EINVAL is still returned from sysfs in the aforementioned case to maintain compatibility with user space that expects -EINVAL.
--- drivers/infiniband/core/cache.c | 2 +- drivers/infiniband/core/sysfs.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/infiniband/core/cache.c +1 -1
@@ -1220,7 +1220,7 @@ EXPORT_SYMBOL(ib_get_cached_port_state);const struct ib_gid_attr *rdma_get_gid_attr (struct ib_device *device,u8 port_num,int index) { -const struct ib_gid_attr *attr =ERR_PTR (-EINVAL ); +const struct ib_gid_attr *attr =ERR_PTR (-ENODATA );struct ib_gid_table *table;unsigned long flags;
drivers/infiniband/core/sysfs.c +2 -1
@@ -391,7 +391,8 @@ static ssize_t _show_port_gid_attr( gid_attr =rdma_get_gid_attr (p->ibdev, p->port_num, tab_attr->index);if (IS_ERR (gid_attr)) -return PTR_ERR (gid_attr); +/* -EINVAL is returned for user space compatibility reasons. */ +return -EINVAL ; ret =rdma_put_gid_attr (gid_attr);
Signature · 1 line
2.26.2
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 2/4] RDMA/core: Modify enum ib_gid_type and enum rdma_network_type
In reply to Leon RomanovskyQuoted 15 lines · Leon Romanovsky
drivers/infiniband/core/cma_configfs.c +2
-4+++ b/drivers/infiniband/core/cma_configfs.c@@ -123,16 +123,21 @@ static ssize_t default_roce_mode_store(struct config_item *item, {struct cma_device *cma_dev;struct cma_dev_port_group *group; -int gid_type =ib_cache_gid_parse_type_str (buf); +int gid_type;ssize_t ret; -if (gid_type <0 ) -return -EINVAL ; - ret =cma_configfs_params_get (item, &cma_dev, &group);
drivers/infiniband/core/cma_configfs.c static ssize_t default_roce_mode_store(struct config_item *item, · line 139
Quoted 10 lines
drivers/infiniband/core/cma_configfs.c +7
-0if (ret)return ret; + gid_type =ib_cache_gid_parse_type_str (buf); +if (gid_type <0 ) +return -EINVAL ; + +if (gid_type ==IB_GID_TYPE_IB && +rdma_protocol_roce_eth_encap (cma_dev->device, group->port_num)) + gid_type =IB_GID_TYPE_ROCE ;
This logic should be in cma_set_default_gid_type() so as not to move struct cma_device
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 3/4] RDMA/core: Introduce new GID table query API
In reply to Leon RomanovskyQuoted 39 lines · Leon Romanovsky
From: Avihai Horon <[email protected]>
Introduce
rdma_query_gid_tablewhich enables querying all the GID tables of a given device and copying the attributes of all valid GID entries to a provided buffer.This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
Signed-off-by Avihai Horon <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/core/cache.c | 93 +++++++++++++++++++++++++ include/rdma/ib_cache.h | 5 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 8 +++ 3 files changed, 106 insertions(+)drivers/infiniband/core/cache.c +15
-0+++ b/drivers/infiniband/core/cache.c@@ -1247,6 +1247,99 @@ rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) }EXPORT_SYMBOL (rdma_get_gid_attr); +/** +* rdma_get_ndev_ifindex - Reads ndev ifindex of the given gid attr. +* @gid_attr: Pointer to the GID attribute. +* @ndev_ifindex: Pointer through which the ndev ifindex is returned. +* +* Returns 0 on success or appropriate error code. The netdevice must be in UP +* state. +*/ +int rdma_get_ndev_ifindex (const struct ib_gid_attr *gid_attr,u32 *ndev_ifindex) +{ +struct net_device *ndev; +int ret =0 ; + +if (rdma_protocol_ib (gid_attr->device, gid_attr->port_num)) {
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1273
Quoted 10 lines
drivers/infiniband/core/cache.c +10
-0+ *ndev_ifindex =0 ; +return 0 ; + } + +rcu_read_lock (); + ndev =rcu_dereference (gid_attr->ndev); +if (!ndev || (READ_ONCE (ndev->flags) &IFF_UP ) ==0 ) { + ret = -ENODEV ; + goto out; + }
None of this is necessary to read the ifindex, especially since the read_lock is being held.
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1294
Quoted 13 lines
drivers/infiniband/core/cache.c +13
-0+/** +* rdma_query_gid_table - Reads GID table entries of all the ports of a device up to max_entries. +* @device: The device to query. +* @entries: Entries where GID entries are returned. +* @max_entries: Maximum number of entries that can be returned. +* Entries array must be allocated to hold max_entries number of entries. +* @num_entries: Updated to the number of entries that were successfully read. +* +* Returns 0 on success or appropriate error code. +*/ +int rdma_query_gid_table (struct ib_device *device, +struct ib_uverbs_gid_entry *entries, +size_t max_entries,size_t *num_entries)
return ssize_t instead of the output pointer
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1301
Quoted 7 lines
drivers/infiniband/core/cache.c +7
-0+{ +const struct ib_gid_attr *gid_attr; +struct ib_gid_table *table; +unsigned int port_num; +unsigned long flags; +int ret; +int i;
i is unsigned
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Leon RomanovskyQuoted 98 lines · Leon Romanovsky
From: Avihai Horon <[email protected]>
Expose the query GID table and entry API to user space by adding two new methods and method handlers to the device object.
This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
Signed-off-by Avihai Horon <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../infiniband/core/uverbs_std_types_device.c | 180 +++++++++++++++++- include/rdma/ib_verbs.h | 6 +- include/uapi/rdma/ib_user_ioctl_cmds.h | 16 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 6 + 4 files changed, 204 insertions(+), 4 deletions(-)drivers/infiniband/core/uverbs_std_types_device.c +67
-0+++ b/drivers/infiniband/core/uverbs_std_types_device.c@@ -8,6 +8,7 @@#include "uverbs.h" #include <rdma/uverbs_ioctl.h> #include <rdma/opa_addr.h> +#include <rdma/ib_cache.h> /* * This ioctl method allows calling any defined write or write_ex @@ -266,6 +267,157 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)(return ucontext->device->ops.query_ucontext (ucontext, attrs); } +static int copy_gid_entries_to_user (struct uverbs_attr_bundle *attrs, +struct ib_uverbs_gid_entry *entries, +size_t num_entries,size_t user_entry_size) +{ +const struct uverbs_attr *attr; +void __user *user_entries; +size_t copy_len; +int ret; +int i; + +if (user_entry_size ==sizeof (*entries)) { + ret =uverbs_copy_to (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , + entries,sizeof (*entries) * num_entries); +return ret; + } + + copy_len =min_t (size_t, user_entry_size,sizeof (*entries)); + attr =uverbs_attr_get (attrs,UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +if (IS_ERR (attr)) +return PTR_ERR (attr); + + user_entries =u64_to_user_ptr (attr->ptr_attr.data); +for (i =0 ; i < num_entries; i++) { +if (copy_to_user (user_entries, entries, copy_len)) +return -EFAULT ; + +if (user_entry_size >sizeof (*entries)) { +if (clear_user (user_entries +sizeof (*entries), + user_entry_size -sizeof (*entries))) +return -EFAULT ; + } + + entries++; + user_entries += user_entry_size; + } + +return uverbs_output_written (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +} + +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_TABLE )( +struct uverbs_attr_bundle *attrs) +{ +struct ib_uverbs_gid_entry *entries; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +size_t user_entry_size; +size_t max_entries; +size_t num_entries; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&user_entry_size, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE ); +if (ret) +return ret; + + max_entries = uverbs_attr_ptr_get_array_size( + attrs, UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES,
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 344
Quoted 10 lines
drivers/infiniband/core/uverbs_std_types_device.c +10
-0+ user_entry_size); +if (max_entries <=0 ) +return -EINVAL ; + + ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device; + + entries =uverbs_zalloc (attrs, max_entries *sizeof (*entries));
This multiplication could overflow
Quoted 28 lines
drivers/infiniband/core/uverbs_std_types_device.c +28
-0+ +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_ENTRY )( +struct uverbs_attr_bundle *attrs) +{ +const struct ib_gid_attr *gid_attr; +struct ib_uverbs_gid_entry entry; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +u32 gid_index; +u32 port_num; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&port_num, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_PORT ); +if (ret) +return ret; + + ret =uverbs_get_const (&gid_index, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX ); +if (ret) +return ret; +
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 399
Quoted 10 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+ ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device; +if (!rdma_is_port_valid (ib_dev, port_num)) +return -EINVAL ; + +if (!rdma_ib_or_roce (ib_dev, port_num)) +return -EINVAL ;
Why these two tests? I would expect rdma_get_gid_attr() to do them
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 409
Quoted 9 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+ gid_attr =rdma_get_gid_attr (ib_dev, port_num, gid_index); +if (IS_ERR (gid_attr)) +return PTR_ERR (gid_attr); + +memcpy (&entry.gid, &gid_attr->gid,sizeof (gid_attr->gid)); + entry.gid_index = gid_attr->index; + entry.port_num = gid_attr->port_num; + entry.gid_type = gid_attr->gid_type; + ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex);
Use rdma_read_gid_attr_ndev_rcu()
Jason
Re: [PATCH rdma-next 2/4] RDMA/core: Modify enum ib_gid_type and enum rdma_network_type
In reply to Jason GunthorpeQuoted 20 lines · Jason Gunthorpe
Quoted 18 lines · Leon Romanovsky
drivers/infiniband/core/cma_configfs.c +2
-4+++ b/drivers/infiniband/core/cma_configfs.c@@ -123,16 +123,21 @@ static ssize_t default_roce_mode_store(struct config_item *item, {struct cma_device *cma_dev;struct cma_dev_port_group *group; -int gid_type =ib_cache_gid_parse_type_str (buf); +int gid_type;ssize_t ret; -if (gid_type <0 ) -return -EINVAL ; - ret =cma_configfs_params_get (item, &cma_dev, &group);if (ret)return ret;
Quoted 10 lines
drivers/infiniband/core/cma_configfs.c static ssize_t default_roce_mode_store(struct config_item *item, · line 139
Quoted 7 lines
drivers/infiniband/core/cma_configfs.c +7
-0+ gid_type =ib_cache_gid_parse_type_str (buf); +if (gid_type <0 ) +return -EINVAL ; + +if (gid_type ==IB_GID_TYPE_IB && +rdma_protocol_roce_eth_encap (cma_dev->device, group->port_num)) + gid_type =IB_GID_TYPE_ROCE ;This logic should be in
cma_set_default_gid_type()so as not to move structcma_device
We wanted to keep "this hack of git type" as close as possible to the actual needed place.
I'll fix.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 3/4] RDMA/core: Introduce new GID table query API
In reply to Jason GunthorpeQuoted 45 lines · Jason Gunthorpe
Quoted 44 lines · Leon Romanovsky
From: Avihai Horon <[email protected]>
Introduce
rdma_query_gid_tablewhich enables querying all the GID tables of a given device and copying the attributes of all valid GID entries to a provided buffer.This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
Signed-off-by Avihai Horon <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/core/cache.c | 93 +++++++++++++++++++++++++ include/rdma/ib_cache.h | 5 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 8 +++ 3 files changed, 106 insertions(+)drivers/infiniband/core/cache.c +20
-0+++ b/drivers/infiniband/core/cache.c@@ -1247,6 +1247,99 @@ rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) }EXPORT_SYMBOL (rdma_get_gid_attr); +/** +* rdma_get_ndev_ifindex - Reads ndev ifindex of the given gid attr. +* @gid_attr: Pointer to the GID attribute. +* @ndev_ifindex: Pointer through which the ndev ifindex is returned. +* +* Returns 0 on success or appropriate error code. The netdevice must be in UP +* state. +*/ +int rdma_get_ndev_ifindex (const struct ib_gid_attr *gid_attr,u32 *ndev_ifindex) +{ +struct net_device *ndev; +int ret =0 ; + +if (rdma_protocol_ib (gid_attr->device, gid_attr->port_num)) { + *ndev_ifindex =0 ; +return 0 ; + } + +rcu_read_lock ();
Quoted 8 lines
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1273
Quoted 5 lines
drivers/infiniband/core/cache.c +5
-0+ ndev =rcu_dereference (gid_attr->ndev); +if (!ndev || (READ_ONCE (ndev->flags) &IFF_UP ) ==0 ) { + ret = -ENODEV ; + goto out; + }None of this is necessary to read the ifindex, especially since the
read_lockis being held.
I see same rcu_read_lock->rcu_dereference->rcu_read_unlock pattern in rdma_read_gid_l2_fields(), why this function is different?
Quoted 16 lines
Quoted 4 lines
drivers/infiniband/core/cache.c +3
-0+/** +* rdma_query_gid_table - Reads GID table entries of all the ports of a device up to max_entries. +* @device: The device to query. drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1294
Quoted 10 lines
drivers/infiniband/core/cache.c +10
-0+* @entries: Entries where GID entries are returned. +* @max_entries: Maximum number of entries that can be returned. +* Entries array must be allocated to hold max_entries number of entries. +* @num_entries: Updated to the number of entries that were successfully read. +* +* Returns 0 on success or appropriate error code. +*/ +int rdma_query_gid_table (struct ib_device *device, +struct ib_uverbs_gid_entry *entries, +size_t max_entries,size_t *num_entries)return
ssize_tinstead of the output pointer
I'll change.
Quoted 10 lines
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1301
Quoted 8 lines
drivers/infiniband/core/cache.c +7
-0+{ +const struct ib_gid_attr *gid_attr; +struct ib_gid_table *table; +unsigned int port_num; +unsigned long flags; +int ret; +int i;i is unsigned
"i" is used as an iterator till table->sz while "sz" is declared as int. I'll change it to be unsigned int, but it is not needed.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Jason GunthorpeQuoted 103 lines · Jason Gunthorpe
Quoted 102 lines · Leon Romanovsky
From: Avihai Horon <[email protected]>
Expose the query GID table and entry API to user space by adding two new methods and method handlers to the device object.
This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
Signed-off-by Avihai Horon <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../infiniband/core/uverbs_std_types_device.c | 180 +++++++++++++++++- include/rdma/ib_verbs.h | 6 +- include/uapi/rdma/ib_user_ioctl_cmds.h | 16 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 6 + 4 files changed, 204 insertions(+), 4 deletions(-)drivers/infiniband/core/uverbs_std_types_device.c +71
-0+++ b/drivers/infiniband/core/uverbs_std_types_device.c@@ -8,6 +8,7 @@#include "uverbs.h" #include <rdma/uverbs_ioctl.h> #include <rdma/opa_addr.h> +#include <rdma/ib_cache.h> /* * This ioctl method allows calling any defined write or write_ex @@ -266,6 +267,157 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)(return ucontext->device->ops.query_ucontext (ucontext, attrs); } +static int copy_gid_entries_to_user (struct uverbs_attr_bundle *attrs, +struct ib_uverbs_gid_entry *entries, +size_t num_entries,size_t user_entry_size) +{ +const struct uverbs_attr *attr; +void __user *user_entries; +size_t copy_len; +int ret; +int i; + +if (user_entry_size ==sizeof (*entries)) { + ret =uverbs_copy_to (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , + entries,sizeof (*entries) * num_entries); +return ret; + } + + copy_len =min_t (size_t, user_entry_size,sizeof (*entries)); + attr =uverbs_attr_get (attrs,UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +if (IS_ERR (attr)) +return PTR_ERR (attr); + + user_entries =u64_to_user_ptr (attr->ptr_attr.data); +for (i =0 ; i < num_entries; i++) { +if (copy_to_user (user_entries, entries, copy_len)) +return -EFAULT ; + +if (user_entry_size >sizeof (*entries)) { +if (clear_user (user_entries +sizeof (*entries), + user_entry_size -sizeof (*entries))) +return -EFAULT ; + } + + entries++; + user_entries += user_entry_size; + } + +return uverbs_output_written (attrs, +UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES ); +} + +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_TABLE )( +struct uverbs_attr_bundle *attrs) +{ +struct ib_uverbs_gid_entry *entries; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +size_t user_entry_size; +size_t max_entries; +size_t num_entries; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&user_entry_size, attrs, +UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE ); +if (ret) +return ret; + + max_entries =uverbs_attr_ptr_get_array_size ( + attrs,UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES , + user_entry_size); +if (max_entries <=0 ) +return -EINVAL ; +
Quoted 8 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 344
Quoted 6 lines
drivers/infiniband/core/uverbs_std_types_device.c +6
-0+ ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device; + + entries =uverbs_zalloc (attrs, max_entries *sizeof (*entries));This multiplication could overflow
Right, I fixed it.
Quoted 34 lines
Quoted 33 lines
drivers/infiniband/core/uverbs_std_types_device.c +32
-0+ +static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_ENTRY )( +struct uverbs_attr_bundle *attrs) +{ +const struct ib_gid_attr *gid_attr; +struct ib_uverbs_gid_entry entry; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +u32 gid_index; +u32 port_num; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&port_num, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_PORT ); +if (ret) +return ret; + + ret =uverbs_get_const (&gid_index, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX ); +if (ret) +return ret; + + ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device;
Quoted 7 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 399
Quoted 5 lines
drivers/infiniband/core/uverbs_std_types_device.c +5
-0+if (!rdma_is_port_valid (ib_dev, port_num)) +return -EINVAL ; + +if (!rdma_ib_or_roce (ib_dev, port_num)) +return -EINVAL ;Why these two tests? I would expect
rdma_get_gid_attr()to do them
First check is not needed, but the second check doesn't exist in rdma_get_gid_attr(). We don't check that table returned from rdma_gid_table() call exists.
Quoted 12 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 409
Quoted 10 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+ gid_attr =rdma_get_gid_attr (ib_dev, port_num, gid_index); +if (IS_ERR (gid_attr)) +return PTR_ERR (gid_attr); + +memcpy (&entry.gid, &gid_attr->gid,sizeof (gid_attr->gid)); + entry.gid_index = gid_attr->index; + entry.port_num = gid_attr->port_num; + entry.gid_type = gid_attr->gid_type; + ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex);Use
rdma_read_gid_attr_ndev_rcu()
I don't want to bring below logic to uverbs* file.
1263 if (rdma_protocol_ib(gid_attr->device, gid_attr->port_num)) {
1264 *ndev_ifindex = 0;
1265 return 0;
1266 }Quoted 2 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 3/4] RDMA/core: Introduce new GID table query API
In reply to Leon RomanovskyQuoted 55 lines · Leon Romanovsky
Quoted 45 lines · Jason Gunthorpe
Quoted 44 lines · Leon Romanovsky
From: Avihai Horon <[email protected]>
Introduce
rdma_query_gid_tablewhich enables querying all the GID tables of a given device and copying the attributes of all valid GID entries to a provided buffer.This API provides a faster way to query a GID table using single call and will be used in libibverbs to improve current approach that requires multiple calls to open, close and read multiple sysfs files for a single GID table entry.
Signed-off-by Avihai Horon <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/core/cache.c | 93 +++++++++++++++++++++++++ include/rdma/ib_cache.h | 5 ++ include/uapi/rdma/ib_user_ioctl_verbs.h | 8 +++ 3 files changed, 106 insertions(+)drivers/infiniband/core/cache.c +20
-0+++ b/drivers/infiniband/core/cache.c@@ -1247,6 +1247,99 @@ rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) }EXPORT_SYMBOL (rdma_get_gid_attr); +/** +* rdma_get_ndev_ifindex - Reads ndev ifindex of the given gid attr. +* @gid_attr: Pointer to the GID attribute. +* @ndev_ifindex: Pointer through which the ndev ifindex is returned. +* +* Returns 0 on success or appropriate error code. The netdevice must be in UP +* state. +*/ +int rdma_get_ndev_ifindex (const struct ib_gid_attr *gid_attr,u32 *ndev_ifindex) +{ +struct net_device *ndev; +int ret =0 ; + +if (rdma_protocol_ib (gid_attr->device, gid_attr->port_num)) { + *ndev_ifindex =0 ; +return 0 ; + } + +rcu_read_lock ();Quoted 8 lines
drivers/infiniband/core/cache.c rdma_get_gid_attr(struct ib_device *device, u8 port_num, int index) · line 1273
Quoted 5 lines
drivers/infiniband/core/cache.c +5
-0+ ndev =rcu_dereference (gid_attr->ndev); +if (!ndev || (READ_ONCE (ndev->flags) &IFF_UP ) ==0 ) { + ret = -ENODEV ; + goto out; + }None of this is necessary to read the ifindex, especially since the
read_lockis being held.
Quoted 2 lines
I see same
rcu_read_lock->rcu_dereference->rcu_read_unlockpattern inrdma_read_gid_l2_fields(), why this function is different?
It doesn't hold the read_lock so it is using rcu to access attr->ndev. With the read_lock held attr->ndev is stable and none of this is needed.
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Leon RomanovskyQuoted 40 lines · Leon Romanovsky
Quoted 32 lines
Quoted 31 lines
drivers/infiniband/core/uverbs_std_types_device.c +31
-0+static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_ENTRY )( +struct uverbs_attr_bundle *attrs) +{ +const struct ib_gid_attr *gid_attr; +struct ib_uverbs_gid_entry entry; +struct ib_ucontext *ucontext; +struct ib_device *ib_dev; +u32 gid_index; +u32 port_num; +u32 flags; +int ret; + + ret =uverbs_get_flags32 (&flags, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS ,0 ); +if (ret) +return ret; + + ret =uverbs_get_const (&port_num, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_PORT ); +if (ret) +return ret; + + ret =uverbs_get_const (&gid_index, attrs, +UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX ); +if (ret) +return ret; + + ucontext =ib_uverbs_get_ucontext (attrs); +if (IS_ERR (ucontext)) +return PTR_ERR (ucontext); + ib_dev = ucontext->device;Quoted 7 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 399
Quoted 5 lines
drivers/infiniband/core/uverbs_std_types_device.c +5
-0+if (!rdma_is_port_valid (ib_dev, port_num)) +return -EINVAL ; + +if (!rdma_ib_or_roce (ib_dev, port_num)) +return -EINVAL ;Why these two tests? I would expect
rdma_get_gid_attr()to do them
Quoted 3 lines
First check is not needed, but the second check doesn't exist in
rdma_get_gid_attr(). We don't check that table returned fromrdma_gid_table()call exists.
Oh that is a bit exciting, maybe it should be checked...
Ideally we should also block this uapi entirely if the device doesn't have a gid table, so this should be -EOPNOTSUP and moved up to the top so it can be moved once we figure it out.
Quoted 12 lines
Quoted 11 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)( · line 409
Quoted 9 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+ gid_attr =rdma_get_gid_attr (ib_dev, port_num, gid_index); +if (IS_ERR (gid_attr)) +return PTR_ERR (gid_attr); + +memcpy (&entry.gid, &gid_attr->gid,sizeof (gid_attr->gid)); + entry.gid_index = gid_attr->index; + entry.port_num = gid_attr->port_num; + entry.gid_type = gid_attr->gid_type; + ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex);Use
rdma_read_gid_attr_ndev_rcu()
Quoted 6 lines
I don't want to bring below logic to uverbs* file.
1263 if (rdma_protocol_ib(gid_attr->device, gid_attr->port_num)) { 1264 *ndev_ifindex = 0; 1265 return 0; 1266 }
Shouldn't be needed, rdma_read_gid_attr_ndev_rcu() already returns -1 for IB
Jason
Re: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
Quoted 45 lines · Jason Gunthorpe
Quoted 40 lines · Leon Romanovsky
Quoted 32 lines
Quoted 31 lines
(quoted diff) +31
-0+static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( + struct uverbs_attr_bundle *attrs) +{ + const struct ib_gid_attr *gid_attr; + struct ib_uverbs_gid_entry entry; + struct ib_ucontext *ucontext; + struct ib_device *ib_dev; + u32 gid_index; + u32 port_num; + u32 flags; + int ret; + + ret = uverbs_get_flags32(&flags, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS, 0); + if (ret) + return ret; + + ret = uverbs_get_const(&port_num, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_PORT); + if (ret) + return ret; + + ret = uverbs_get_const(&gid_index, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX); + if (ret) + return ret; + + ucontext = ib_uverbs_get_ucontext(attrs); + if (IS_ERR(ucontext)) + return PTR_ERR(ucontext); + ib_dev = ucontext->device;Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (!rdma_is_port_valid(ib_dev, port_num)) + return -EINVAL; + + if (!rdma_ib_or_roce(ib_dev, port_num)) + return -EINVAL;Why these two tests? I would expect
rdma_get_gid_attr()to do themQuoted 3 lines
First check is not needed, but the second check doesn't exist in
rdma_get_gid_attr(). We don't check that table returned fromrdma_gid_table()call exists.
Quoted 5 lines
Oh that is a bit exciting, maybe it should be checked...
Ideally we should also block this uapi entirely if the device doesn't have a gid table, so this should be
-EOPNOTSUPand moved up to the top so it can be moved once we figure it out.
It is already in earliest possible stage, right after we get ib_device.
Quoted 15 lines
Quoted 15 lines
Quoted 12 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+ gid_attr = rdma_get_gid_attr(ib_dev, port_num, gid_index); + if (IS_ERR(gid_attr)) + return PTR_ERR(gid_attr); + + memcpy(&entry.gid, &gid_attr->gid, sizeof(gid_attr->gid)); + entry.gid_index = gid_attr->index; + entry.port_num = gid_attr->port_num; + entry.gid_type = gid_attr->gid_type; + ret = rdma_get_ndev_ifindex(gid_attr, &entry.netdev_ifindex);Use
rdma_read_gid_attr_ndev_rcu()I don't want to bring below logic to uverbs* file.
Quoted 7 lines
Quoted 4 lines
1263 if (rdma_protocol_ib(gid_attr->device, gid_attr->port_num)) { 1264 *ndev_ifindex = 0; 1265 return 0; 1266 }Shouldn't be needed,
rdma_read_gid_attr_ndev_rcu()already returns -1 for IB
It will be something like this:
drivers/infiniband/core/uverbs_std_types_device.c +13 -4
@@ -368,10 +368,11 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_TABLE)(static int UVERBS_HANDLER (UVERBS_METHOD_QUERY_GID_ENTRY )(struct uverbs_attr_bundle *attrs) { +struct ib_uverbs_gid_entry entry = {};const struct ib_gid_attr *gid_attr; -struct ib_uverbs_gid_entry entry;struct ib_ucontext *ucontext;struct ib_device *ib_dev; +struct net_device *ndev;u32 gid_index;u32 port_num;u32 flags;
@@ -408,9 +409,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( entry.gid_index = gid_attr->index; entry.port_num = gid_attr->port_num; entry.gid_type = gid_attr->gid_type; - ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex); -if (ret) - goto out; + +if (rdma_protocol_roce (ib_dev, port_num)) { +rcu_read_lock (); + ndev =rdma_read_gid_attr_ndev_rcu (gid_attr); +if (IS_ERR (ndev)) { +rcu_read_unlock (); + goto out; + } + entry.netdev_ifindex = ndev->ifindex; +rcu_read_unlock (); + } ret = uverbs_copy_to_struct_or_zero( attrs, UVERBS_ATTR_QUERY_GID_ENTRY_RESP_ENTRY, &entry,
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Jason Gunthorpe
Quoted 43 lines · Leon Romanovsky
Quoted 32 lines
Quoted 31 lines
(quoted diff) +31
-0+static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( + struct uverbs_attr_bundle *attrs) +{ + const struct ib_gid_attr *gid_attr; + struct ib_uverbs_gid_entry entry; + struct ib_ucontext *ucontext; + struct ib_device *ib_dev; + u32 gid_index; + u32 port_num; + u32 flags; + int ret; + + ret = uverbs_get_flags32(&flags, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS, 0); + if (ret) + return ret; + + ret = uverbs_get_const(&port_num, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_PORT); + if (ret) + return ret; + + ret = uverbs_get_const(&gid_index, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX); + if (ret) + return ret; + + ucontext = ib_uverbs_get_ucontext(attrs); + if (IS_ERR(ucontext)) + return PTR_ERR(ucontext); + ib_dev = ucontext->device;Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (!rdma_is_port_valid(ib_dev, port_num)) + return -EINVAL; + + if (!rdma_ib_or_roce(ib_dev, port_num)) + return -EINVAL;Why these two tests? I would expect
rdma_get_gid_attr()to do themFirst check is not needed, but the second check doesn't exist in
rdma_get_gid_attr(). We don't check that table returned fromrdma_gid_table()call exists.
Quoted 7 lines
Quoted 5 lines
Oh that is a bit exciting, maybe it should be checked...
Ideally we should also block this uapi entirely if the device doesn't have a gid table, so this should be
-EOPNOTSUPand moved up to the top so it can be moved once we figure it out.It is already in earliest possible stage, right after we get
ib_device.
I usually put it before the uverbs_get_XX.. but doesn't matter
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( · line 413
Quoted 9 lines
drivers/infiniband/core/uverbs_std_types_device.c +2
-3@@ -408,9 +409,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( entry.gid_index = gid_attr->index; entry.port_num = gid_attr->port_num; entry.gid_type = gid_attr->gid_type; - ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex); -if (ret) - goto out; + +if (rdma_protocol_roce (ib_dev, port_num)) {
Not sure this is needed? non-roce still has gid table entries, and the attr->ndev should be null so it will do the ENODEV naturally.
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( · line 422
Quoted 9 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+rcu_read_lock (); + ndev =rdma_read_gid_attr_ndev_rcu (gid_attr); +if (IS_ERR (ndev)) { +rcu_read_unlock (); + goto out; + } + entry.netdev_ifindex = ndev->ifindex; +rcu_read_unlock (); + }
This is better than what is in rdma_get_ndev_ifindex()
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Leon RomanovskyQuoted 4 lines · Leon Romanovsky
It depends on how you want to treat errors from
rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.
Quoted 8 lines
Otherwise we will find ourselves with something like this:
ndev =rdma_read_gid_attr_ndev_rcu(gid_attr);if (IS_ERR(ndev)) { if (rdma_protocol_roce()) goto error; if (ERR_PTR(ndev) != -ENODEV) goto error; }
Isn't it just
if (IS_ERR(ndev)) {
if (ERR_PTR(ndev) != -ENODEV)
goto error;
index = -1;
}
Which seems fine and clear enough
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Leon RomanovskyQuoted 29 lines · Leon Romanovsky
Quoted 28 lines · Jason Gunthorpe
Quoted 26 lines · Leon Romanovsky
Quoted 16 lines · Jason Gunthorpe
Quoted 4 lines · Leon Romanovsky
It depends on how you want to treat errors from
rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.Quoted 8 lines
Otherwise we will find ourselves with something like this:
ndev =rdma_read_gid_attr_ndev_rcu(gid_attr);if (IS_ERR(ndev)) { if (rdma_protocol_roce()) goto error; if (ERR_PTR(ndev) != -ENODEV) goto error; }Isn't it just
Quoted 7 lines
if (IS_ERR(ndev)) { if (ERR_PTR(ndev) != -ENODEV) goto error; index = -1; }Which seems fine and clear enough
It is a problem if roce device returned
-ENODEV.
Quoted 7 lines
Quoted 3 lines
Can it happen? RCU I suppose, but I think this is an issue in
rdma_read_gid_attr_ndev_rcu()- it should not return ENODEV if the RCU shows thegid_attris being concurrently destroyedFrom RoCE point of view, it is a problem if device is destroyed or gid not valid, the different returned values won't change much. For the IB, we don't care.
I'm saying I prefer things layered properly. Having random rdma_protocol's all over the place has proven to be hard to maintain
One inside rdma_read_gid_attr_ndev_rcu() to make a NULL netdev an error return for roce is much more understandable
Jason
Re: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Jason GunthorpeQuoted 57 lines · Jason Gunthorpe
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Jason Gunthorpe
Quoted 43 lines · Leon Romanovsky
Quoted 32 lines
Quoted 31 lines
(quoted diff) +31
-0+static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( + struct uverbs_attr_bundle *attrs) +{ + const struct ib_gid_attr *gid_attr; + struct ib_uverbs_gid_entry entry; + struct ib_ucontext *ucontext; + struct ib_device *ib_dev; + u32 gid_index; + u32 port_num; + u32 flags; + int ret; + + ret = uverbs_get_flags32(&flags, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_FLAGS, 0); + if (ret) + return ret; + + ret = uverbs_get_const(&port_num, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_PORT); + if (ret) + return ret; + + ret = uverbs_get_const(&gid_index, attrs, + UVERBS_ATTR_QUERY_GID_ENTRY_GID_INDEX); + if (ret) + return ret; + + ucontext = ib_uverbs_get_ucontext(attrs); + if (IS_ERR(ucontext)) + return PTR_ERR(ucontext); + ib_dev = ucontext->device;Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (!rdma_is_port_valid(ib_dev, port_num)) + return -EINVAL; + + if (!rdma_ib_or_roce(ib_dev, port_num)) + return -EINVAL;Why these two tests? I would expect
rdma_get_gid_attr()to do themFirst check is not needed, but the second check doesn't exist in
rdma_get_gid_attr(). We don't check that table returned fromrdma_gid_table()call exists.Quoted 7 lines
Quoted 5 lines
Oh that is a bit exciting, maybe it should be checked...
Ideally we should also block this uapi entirely if the device doesn't have a gid table, so this should be
-EOPNOTSUPand moved up to the top so it can be moved once we figure it out.It is already in earliest possible stage, right after we get
ib_device.I usually put it before the
uverbs_get_XX.. but doesn't matter
Quoted 12 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( · line 413
Quoted 9 lines
drivers/infiniband/core/uverbs_std_types_device.c +2
-3@@ -408,9 +409,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( entry.gid_index = gid_attr->index; entry.port_num = gid_attr->port_num; entry.gid_type = gid_attr->gid_type; - ret =rdma_get_ndev_ifindex (gid_attr, &entry.netdev_ifindex); -if (ret) - goto out; + +if (rdma_protocol_roce (ib_dev, port_num)) {Not sure this is needed? non-roce still has gid table entries, and the
attr->ndevshould be null so it will do the ENODEV naturally.
It depends on how you want to treat errors from rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.
Otherwise we will find ourselves with something like this:
ndev = rdma_read_gid_attr_ndev_rcu(gid_attr);
if (IS_ERR(ndev)) {
if (rdma_protocol_roce())
goto error;
if (ERR_PTR(ndev) != -ENODEV)
goto error;
}Quoted 14 lines
drivers/infiniband/core/uverbs_std_types_device.c static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)( · line 422
Quoted 10 lines
drivers/infiniband/core/uverbs_std_types_device.c +9
-0+rcu_read_lock (); + ndev =rdma_read_gid_attr_ndev_rcu (gid_attr); +if (IS_ERR (ndev)) { +rcu_read_unlock (); + goto out; + } + entry.netdev_ifindex = ndev->ifindex; +rcu_read_unlock (); + }This is better than what is in
rdma_get_ndev_ifindex()Jason
Re: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Jason GunthorpeQuoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Jason Gunthorpe
Quoted 4 lines · Leon Romanovsky
It depends on how you want to treat errors from
rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.Quoted 8 lines
Otherwise we will find ourselves with something like this:
ndev =rdma_read_gid_attr_ndev_rcu(gid_attr);if (IS_ERR(ndev)) { if (rdma_protocol_roce()) goto error; if (ERR_PTR(ndev) != -ENODEV) goto error; }Isn't it just
if (IS_ERR(ndev)) { if (ERR_PTR(ndev) != -ENODEV) goto error; index = -1; }
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Which seems fine and clear enough
It is a problem if roce device returned
-ENODEV.Can it happen? RCU I suppose, but I think this is an issue in
rdma_read_gid_attr_ndev_rcu()- it should not return ENODEV if the RCU shows thegid_attris being concurrently destroyed
From RoCE point of view, it is a problem if device is destroyed or gid not valid, the different returned values won't change much. For the IB, we don't care.
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
In reply to Jason GunthorpeQuoted 16 lines · Jason Gunthorpe
Quoted 4 lines · Leon Romanovsky
It depends on how you want to treat errors from
rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.Quoted 8 lines
Otherwise we will find ourselves with something like this:
ndev =rdma_read_gid_attr_ndev_rcu(gid_attr);if (IS_ERR(ndev)) { if (rdma_protocol_roce()) goto error; if (ERR_PTR(ndev) != -ENODEV) goto error; }Isn't it just
Quoted 7 lines
if (IS_ERR(ndev)) { if (ERR_PTR(ndev) != -ENODEV) goto error; index = -1; }Which seems fine and clear enough
It is a problem if roce device returned -ENODEV. I don't want to paper-out this case by setting index = 0.
Thanks
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 4/4] RDMA/uverbs: Expose the new GID query API to user space
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Jason Gunthorpe
Quoted 4 lines · Leon Romanovsky
It depends on how you want to treat errors from
rdma_read_gid_attr_ndev_rcu(). Current check allows us to ensure that any error returned by this call is handled.Quoted 8 lines
Otherwise we will find ourselves with something like this:
ndev =rdma_read_gid_attr_ndev_rcu(gid_attr);if (IS_ERR(ndev)) { if (rdma_protocol_roce()) goto error; if (ERR_PTR(ndev) != -ENODEV) goto error; }Isn't it just
if (IS_ERR(ndev)) { if (ERR_PTR(ndev) != -ENODEV) goto error; index = -1; }
Quoted 3 lines
Quoted 1 line
Which seems fine and clear enough
It is a problem if roce device returned
-ENODEV.
Can it happen? RCU I suppose, but I think this is an issue in rdma_read_gid_attr_ndev_rcu() - it should not return ENODEV if the RCU shows the gid_attr is being concurrently destroyed
Jason