A rendezvous module
cover A rendezvous module
From: Kaike Wan <[email protected]>
RDMA transactions on RC QPs have a high demand for memory for HPC applications such as MPI, especially on nodes with high cpu core count, where a process is normally dispatched to each core and an RC QP is created for each remote process. For a 100-node fabric, about 10 GB - 100 GB memory is required for WQEs/Buffers/QP states on each server node. This high demand imposes a severe restriction on the scalability of HPC fabric.
A number of solutions have been implemented over the years. UD based solutions solve the scalability problem by requiring only one UD QP per process that can send a message to or receive a message from any other process. However, it does not have the performance of an RC QP and the application has to manage the segmentation for large messages.
SRQ reduces the memory demand by sharing a receive queue among multiple QPs. However, it still requires an RC QP for each remote process and each RC QP still requires a send queue. In addition, it is an optional feature.
XRC further reduces the memory demand by allowing a single QP per process to communicate with any process on a remote node. In this mode, each process requires only one QP for each remote node. Again, this is optional and not all vendors support it.
Dynamically Connected transport minimizes the memory usage, but requires vendor specific hardware and changes in applications. Addtionally, all of these mechanisms can induce latency jitter due to use of more QPs, more QP state, and hence additional
PCIe transactions at scale where NIC QP/CQ caches overflow.
Based on this, we are here proposing a generic, vendor-agnostic approach to address the RC scalalibity and potentially improve RDMA performance for larger messages which uses RDMA Write as part of a rendezvous protocol.
Here are the key points of the proposal:
- For each device used by a job on a given node, there are a fixed number of RC connections (default: 4) between this node and any other remote node, no matter how many processes are running on each node for this job. That is, for a given job and device, all the processes on a given node will communicate with all the processes of the same job on a remote node through the same fixed number of connections. This eliminates the increased memory demand caused by core count increase and reduces the overall RC connection setup costs.
- Memory region cache is added to reduce the MR registration/deregistration costs, as the same buffers tend to be used repeatedly by applications. The mr cache is per process and can be accessed only by processes in the same job.
Here is how we are proposing to implement this application enabling rendezvous module that take advantage of the various features of the RDMA subsystem:
- Create a rendezvous module (rv) under drivers/infiniband/ulp/rv/. This can be changed if a better location is recommended.
- The rv modules adds a char device file /dev/rv that user applications/middlewares can open to communicate with the rv module (PSM3 OFI provider in libfabric 1.12.0 being the primary use case). The file interface is crucial for associating the job/process with the mr, the connection endpoints, and RDMA requests. rv_file_open - Open a handle to the rv module for the current user process.
rv_file_close - Close the handle and clean up. rv_file_ioctl - The main communication methods: to attach to a device,
to register/deregister mr, to create connection endpoint, to connect, to query, to get stats, to do RDMA transactions, etc.
rv_file_mmap - Mmap an event ring into user space for the current user
process.
- Basic mode of operations (PSM3 is used as an example for user applications):
- A middleware (like MPI) has out-of-band communication channels between any two nodes, which are used to establish high performance communications for providers such as PSM3.
- On each node, the PSM3 provider opens the rv module, and issues an attach request to bind to a specific local device and specifies information specific to the job. This associates the user process with the RDMA device and the job.
- On each node, the PSM3 provider will establish user space low latency eager and control message communications, typically via user space UD QPs.
- On each node, the PSM3 provider will mmap a ring buffer for events from the rv module.
- On each node, the PSM3 provider creates a set of connection endpoints by passing the destination info to the rv module. If the local node is chosen as the listener for the connection (based on address comparison between the two nodes), it will start to listen for any incoming IB CM connection requests and accept them when whey arrive. This step associates the shared connection endpoints with the job and device.
- On each node, the PSM3 provider will request connection establishment through the rv module. On the client node, rv sends an IB CM request to the remote node. On the listener node, nothing additional needs to be done for the connect request from the PSM3 provider.
- On each node, the PSM3 provider will wait until the RC connection is established. The PSM3 provider will query the rv module about the connection status periodically.
- As large MPI messages are requested, the PSM3 provider will request rv to register MRs for the MPI application’s send and receive buffers. The rv module makes use of its MR cache to limit when such requests need to interact with verbs MR calls for the NIC. The PSM3 provider control message mechanisms are used to exchange IO virtual addresses and rkeys for such buffers and MRs.
- For RDMA transactions, the RDMA WRITE WITH IMMED request will be used. The immediate data will contain info about the target process on the remote node and which outstanding rendezvous message this RDMA is for. For send completion and receive completion, an event will be posted into the event ring buffer and the PSM3 provider can poll for completion.
- As RDMA transactions complete, the PSM3 provider will indicate completion of the corresponding MPI send/receive to the MPI middleware/application. The PSM3 provider will also inform the rv module it may deregister the MR. The deregistration allows rv to cache the MR for use in future requests by the PSM3 provider to register the same memory for use in another MPI message. The cached MR will be removed if the user buffer is freed and a MMU notifier event is received by the rv module.
Please comment,
Thank you,
Kaike
Kaike Wan (9):
RDMA/rv: Public interferce for the RDMA Rendezvous module RDMA/rv: Add the internal header files
RDMA/rv: Add the rv module
RDMA/rv: Add functions for memory region cache RDMA/rv: Add function to register/deregister memory region RDMA/rv: Add connection management functions RDMA/rv: Add functions for RDMA transactions RDMA/rv: Add functions for file operations RDMA/rv: Integrate the file operations into the rv module
MAINTAINERS | 6 + drivers/infiniband/Kconfig | 1 + drivers/infiniband/core/core_priv.h | 4 + drivers/infiniband/core/rdma_core.c | 237 ++ drivers/infiniband/core/uverbs_cmd.c | 54 +- drivers/infiniband/ulp/Makefile | 1 + drivers/infiniband/ulp/rv/Kconfig | 11 + drivers/infiniband/ulp/rv/Makefile | 9 + drivers/infiniband/ulp/rv/rv.h | 892 ++++++ drivers/infiniband/ulp/rv/rv_conn.c | 3037 ++++++++++++++++++++ drivers/infiniband/ulp/rv/rv_file.c | 1196 ++++++++ drivers/infiniband/ulp/rv/rv_main.c | 271 ++ drivers/infiniband/ulp/rv/rv_mr.c | 393 +++ drivers/infiniband/ulp/rv/rv_mr_cache.c | 428 +++ drivers/infiniband/ulp/rv/rv_mr_cache.h | 152 + drivers/infiniband/ulp/rv/rv_rdma.c | 416 +++ drivers/infiniband/ulp/rv/trace.c | 7 + drivers/infiniband/ulp/rv/trace.h | 10 + drivers/infiniband/ulp/rv/trace_conn.h | 547 ++++ drivers/infiniband/ulp/rv/trace_dev.h | 82 + drivers/infiniband/ulp/rv/trace_mr.h | 109 + drivers/infiniband/ulp/rv/trace_mr_cache.h | 181 ++ drivers/infiniband/ulp/rv/trace_rdma.h | 254 ++ drivers/infiniband/ulp/rv/trace_user.h | 321 +++ include/rdma/uverbs_types.h | 10 + include/uapi/rdma/rv_user_ioctls.h | 558 ++++ 26 files changed, 9139 insertions(+), 48 deletions(-) create mode 100644 drivers/infiniband/ulp/rv/Kconfig create mode 100644 drivers/infiniband/ulp/rv/Makefile create mode 100644 drivers/infiniband/ulp/rv/rv.h create mode 100644 drivers/infiniband/ulp/rv/rv_conn.c create mode 100644 drivers/infiniband/ulp/rv/rv_file.c create mode 100644 drivers/infiniband/ulp/rv/rv_main.c create mode 100644 drivers/infiniband/ulp/rv/rv_mr.c create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.c create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.h create mode 100644 drivers/infiniband/ulp/rv/rv_rdma.c create mode 100644 drivers/infiniband/ulp/rv/trace.c create mode 100644 drivers/infiniband/ulp/rv/trace.h create mode 100644 drivers/infiniband/ulp/rv/trace_conn.h create mode 100644 drivers/infiniband/ulp/rv/trace_dev.h create mode 100644 drivers/infiniband/ulp/rv/trace_mr.h create mode 100644 drivers/infiniband/ulp/rv/trace_mr_cache.h create mode 100644 drivers/infiniband/ulp/rv/trace_rdma.h create mode 100644 drivers/infiniband/ulp/rv/trace_user.h create mode 100644 include/uapi/rdma/rv_user_ioctls.h
Signature · 1 line
2.18.1
RFC PATCH 1/9 RDMA/rv: Public interferce for the RDMA Rendezvous module
From: Kaike Wan <[email protected]>
The RDMA Rendezvous (rv) module provides an interface for HPC middlewares to improve performance by caching memory region registration, and improve the scalibity of RDMA transaction through connection managements between nodes. This mechanism is implemented through the following ioctl requests:
- ATTACH: to attach to an RDMA device.
- REG_MEM: to register a user/kernel memory region.
- DEREG_MEM: to release application use of MR, allowing it to remain in cache.
- GET_CACHE_STATS: to get cache statistics.
- CONN_CREATE: to create an RC connection.
- CONN_CONNECT: to start the connection.
- CONN_GET_CONN_COUNT: to use as part of error recovery from lost messages in the application.
- CONN_GET_STATS: to get connection statistics.
- GET_EVENT_STATS: to get the RDMA event statistics.
- POST_RDMA_WR_IMMED: to post an RDMA WRITE WITH IMMED request.
--- include/uapi/rdma/rv_user_ioctls.h | 558 +++++++++++++++++++++++++++++ 1 file changed, 558 insertions(+) create mode 100644 include/uapi/rdma/rv_user_ioctls.h
include/uapi/rdma/rv_user_ioctls.h new file +558 -0
@@ -0,0 +1,558 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* +* Copyright(c) 2020 - 2021 Intel Corporation. +*/ +#ifndef __RV_USER_IOCTL_H__ +#define __RV_USER_IOCTL_H__ +#include <rdma/rdma_user_ioctl.h> +#include <rdma/ib_user_sa.h> +#include <rdma/ib_user_verbs.h> + +/* Checking /Documentation/userspace-api/ioctl/ioctl-number.rst */ +#define RV_MAGIC RDMA_IOCTL_MAGIC +#define RV_FILE_NAME "/dev/rv" + +/* +* Handles are opaque to application; they are meaningful only to the +* RV driver +*/ + +/* this version of ABI */ +#define RV_ABI_VER_MAJOR 1 +#define RV_ABI_VER_MINOR 0 + +struct rv_query_params_out { +/* ABI version */ +__u16 major_rev; +__u16 minor_rev; +__u32 resv1; +__aligned_u64 capability; +__aligned_u64 resv2[6 ]; +}; + +#define RV_IOCTL_QUERY _IOR(RV_MAGIC, 0xFC, struct rv_query_params_out) + +/* Mode for use of rv module by PSM */ +#define RV_RDMA_MODE_USER 0/* user MR caching only */ +#define RV_RDMA_MODE_KERNEL 1/* + kernel RC QPs with kernel MR caching */ +#define RV_RDMA_MODE_MAX 1 + +#define RV_MAX_DEV_NAME_LEN IB_DEVICE_NAME_MAX +#define RV_MAX_NUM_CONN 16 +#define RV_MAX_INDEX_BITS 12 +#define RV_MAX_JOB_KEY_LEN 16 +#define RV_MAX_CQ_ENTRIES 10000 + +/* +* mr_cache_size is in MBs and if 0 will use module param as default +* num_conn - number of QPs between each pair of nodes +* loc_addr - used to select client/listen vs rem_addr +* index_bits - num high bits of immed data with rv index +* loc_gid_index - SGID for client connections +* loc_gid[16] - to double check gid_index unchanged +* job_key[RV_MAX_JOB_KEY_LEN] = unique uuid per job +* job_key_len - len, if 0 matches jobs with len==0 only +* q_depth - size of QP and per QP CQs +* reconnect_timeout - in seconds from loss to restoration +* hb_interval - in milliseconds between heartbeats +*/ +struct rv_attach_params_in { +char dev_name[RV_MAX_DEV_NAME_LEN ]; +__u32 mr_cache_size; +__u8 rdma_mode; + +/* additional information for RV_RDMA_MODE_KERNEL */ +__u8 port_num; +__u8 num_conn; +__u32 loc_addr; +__u8 index_bits; +__u16 loc_gid_index; +__u8 loc_gid[16 ]; +__u8 job_key[RV_MAX_JOB_KEY_LEN ]; +__u8 job_key_len; +__aligned_u64 service_id; +__aligned_u64 context; +__u32 cq_entries; +__u32 q_depth; +__u32 reconnect_timeout; +__u32 hb_interval; +}; + +/* +* rv_index - unique within job on given NIC +* mr_cache_size - in MBs +* q_depth - size of QP and per QP CQs +* reconnect_timeout - value being used +*/ +struct rv_attach_params_out { +__u32 rv_index; +__u32 mr_cache_size; +__u32 q_depth; +__u32 reconnect_timeout; +}; + +struct rv_attach_params { +union { +struct rv_attach_params_in in; +struct rv_attach_params_out out; + }; +}; + +#define RV_IOCTL_ATTACH _IOWR(RV_MAGIC, 0xF1, struct rv_attach_params) + +/* The buffer is used to register a kernel mr */ +#define IBV_ACCESS_KERNEL 0x80000000 + +/* +* ibv_pd_handle - user space appl allocated pd +* ulen - driver_udata inlen +* *udata - driver_updata inbuf +*/ +struct rv_mem_params_in { +__u32 ibv_pd_handle; +__u32 cmd_fd_int; +__aligned_u64 addr; +__aligned_u64 length; +__u32 access; +size_t ulen; +void *udata; +}; + +struct rv_mem_params_out { +__aligned_u64 mr_handle; +__aligned_u64 iova; +__u32 lkey; +__u32 rkey; +}; + +struct rv_mem_params { +union { +struct rv_mem_params_in in; +struct rv_mem_params_out out; + }; +}; + +#define RV_IOCTL_REG_MEM _IOWR(RV_MAGIC, 0xF3, struct rv_mem_params) + +struct rv_dereg_params_in { +__aligned_u64 mr_handle; +__aligned_u64 addr; +__aligned_u64 length; +__u32 access; +}; + +#define RV_IOCTL_DEREG_MEM _IOW(RV_MAGIC, 0xF4, struct rv_dereg_params_in) + +/* +* MR cache statistics are as follows: +* cache_size - Current memory in the cache in bytes +* max_cache_size - Maximum of cache_size in bytes +* limit_cache_size - Maximum allowed cache_size in MB +* count - Current number of MRs in the cache +* max_count - Maximum of count +* inuse - Current number of MRs in the cache with refcount > 0 +* max_inuse - Maximum of inuse +* inuse_bytes - Current number of bytes in cache for MRs with refcount > 0 +* max_inuse_bytes - Maximum of inuse_bytes +* max_refcount - Maximum of refcount for an MR +* Event counts: +* hit - Cache hit +* miss - Cache miss and added +* full - Cache miss but can't add since full +* failed - Cache miss but can't add since reg_mr failed +* remove - Refcount == 0 and removed by mmu notifier event or cache deinit +* evict - Removed from cache due to lack of cache space +* Number of valid IOCTL_REG_MEM calls = hit+miss+full+failed +* (counts omit some EINVAL and EFAULT REG_MEM use cases) +*/ + +struct rv_cache_stats_params_out { +__aligned_u64 cache_size; +__aligned_u64 max_cache_size; +__u32 limit_cache_size; +__u32 count; +__u32 max_count; +__u32 inuse; +__u32 max_inuse; +__aligned_u64 inuse_bytes; +__aligned_u64 max_inuse_bytes; +__u32 max_refcount; +__aligned_u64 hit; +__aligned_u64 miss; +__aligned_u64 full; +__aligned_u64 failed; +__aligned_u64 remove; +__aligned_u64 evict; +}; + +#define RV_IOCTL_GET_CACHE_STATS _IOR(RV_MAGIC, 0xF7, \ + struct rv_cache_stats_params_out) + +/* +* The create provides an ah_attr. Field use is as follows: +* sanity checked with attach: +* grh.sgid_index, port_num +* possibly in future: src_path_bits +* identify QPs which can be shared +* rem_addr, ah.is_global, grh.dgid or dlid +* possibly in future: sl, traffic_class, flow_label, maybe static_rate +* sanity checked with connect path: +* dlid, grh.dgid +* could check: sl +* don't want to check: static_rate (could negotiate down in path) +* validated with inbound REQ +* port_num, grh.dgid or dlid +* Not used: hop_limit (locally resolve) +* rem_addr - used to select client/listen vs loc_addr +*/ +struct rv_conn_create_params_in { +struct ib_uverbs_ah_attr ah; +__u32 rem_addr; +__aligned_u64 context; +}; + +/* +* handle - rv_user_conn for future conn/discon calls +* conn_handle - rv_conn for completions only +*/ +struct rv_conn_create_params_out { +__aligned_u64 handle; +__aligned_u64 conn_handle; +}; + +struct rv_conn_create_params { +union { +struct rv_conn_create_params_in in; +struct rv_conn_create_params_out out; + }; +}; + +#define RV_IOCTL_CONN_CREATE _IOWR(RV_MAGIC, 0xF8, \ + struct rv_conn_create_params) + +struct rv_conn_connect_params_in { +__aligned_u64 handle; +struct ib_user_path_rec path; +}; + +#define RV_IOCTL_CONN_CONNECT _IOW(RV_MAGIC, 0xF9, \ + struct rv_conn_connect_params_in) + +struct rv_conn_connected_params_in { +__aligned_u64 handle; +}; + +#define RV_IOCTL_CONN_CONNECTED _IOW(RV_MAGIC, 0xFA, \ + struct rv_conn_connected_params_in) + +/* +* get connection count for a specific sconn +* returns: +* 0 - count returned +* EIO - connection lost and unrecoverable +* EINVAL - invalid handle and/or index +* A 32b count is sufficient to handle constant RV reconnects, with a +* 100ms delay between each established connection, for up to 13 years. +*/ +struct rv_conn_get_conn_count_params_in { +__aligned_u64 handle; +__u8 index; +}; + +/* we return count as an output parameter (vs ioctl ret) so can use full 32b */ +struct rv_conn_get_conn_count_params_out { +__u32 count; +}; + +struct rv_conn_get_conn_count_params { +union { +struct rv_conn_get_conn_count_params_in in; +struct rv_conn_get_conn_count_params_out out; + }; +}; + +#define RV_IOCTL_CONN_GET_CONN_COUNT _IOWR(RV_MAGIC, 0xFF, \ + struct rv_conn_get_conn_count_params) + +/* index to get agg of sconn in given conn */ +#define RV_CONN_STATS_AGGREGATE 255 + +/* +* handle - if 0, aggregate of all rv's sconn returned +* index - ignored if !handle, otherwise specific sconn index +*/ +struct rv_conn_get_stats_params_in { +__aligned_u64 handle; +__u8 index; +}; + +/* +* flags can be combined when get aggregate results +* so separate bits for client vs server +*/ +#define RV_CONN_STAT_FLAG_SERVER 0x01 +#define RV_CONN_STAT_FLAG_CLIENT 0x02 +#define RV_CONN_STAT_FLAG_WAS_CONNECTED 0x04 + +/* +* index - mimics input value +* flags +* num_conn - total QPs included +* CM events +* req_error - IB_CM_REQ_ERROR +* req_recv - IB_CM_REQ_RECEIVED +* rep_error - IB_CM_REP_ERROR +* rep_recv - IB_CM_REP_RECEIVED +* rtu_recv - IB_CM_RTU_RECEIVED +* established - IB_CM_USER_ESTABLISHED - via ib_cm_notify +* dreq_error - IB_CM_DREQ_ERROR +* dreq_recv - IB_CM_DREQ_RECEIVED +* drep_recv - IB_CM_DREP_RECEIVED +* timewait - IB_CM_TIMEWAIT_EXIT +* mra_recv - IB_CM_MRA_RECEIVED +* rej_recv - IB_CM_REJ_RECEIVED +* lap_error - IB_CM_LAP_ERROR +* lap_recv - IB_CM_LAP_RECEIVED +* apr_recv - IB_CM_APR_RECEIVED +* unexp_event - SIDR and any others +* outbound CM messages +* req_sent - CM REQ +* rep_sent - CM REP +* rtu_sent - CM RTU +* rej_sent - CM REJ +* dreq_sent - CM DREQ +* drep_sent - CM DREP +* (re)connect time does not include wait nor resolver time +* wait_time - microseconds for initial connect +* resolve_time - microseconds for initial connect +* connect_time - microseconds for initial connect +* connected_time - microseconds were connected +* resolve - attempts at resolving +* resolve_fail - hard failures +* conn_recovery - # times recovered connection +* rewait_time - microseconds for connect recovery +* reresolve_time - microseconds for connect recovery +* reconnect_time - microseconds for connect recovery +* max_rewait_time - microseconds for connect recovery +* max_reresolve_time - microseconds for connect recovery +* max_reconnect_time - microseconds for connect recovery +* reresolve - attempts at resolving +* reresolve_fail - hard failures +* post_write - successful post_rdma_write +* post_write_fail - failed at time of posting +* post_write_bytes - for successful post +* outstand_send_write - sent RDMA Write waiting for CQE +* send_write_cqe - successful sent RDMA Write CQE +* send_write_cqe_fail - sent RDMA Write CQE with bad status +* recv_write_cqe - successful recv RDMA Write CQE +* recv_write_bytes - successful recv RDMA Write +* recv_cqe_fail - recv CQE with bad status +* post_hb - successful post of heartbeat +* post_hb_fail - failed at time of posting +* send_hb_cqe - successful sent heartbeat CQE +* send_hb_cqe_fail - sent heartbeat CQE with bad status +* recv_hb_cqe - successful recv heartbeat CQE +*/ +struct rv_conn_get_stats_params_out { +__u8 index; +__u8 flags; +__u32 num_conn; + +/* CM events */ +__u32 req_error; +__u32 req_recv; +__u32 rep_error; +__u32 rep_recv; +__u32 rtu_recv; +__u32 established; +__u32 dreq_error; +__u32 dreq_recv; +__u32 drep_recv; +__u32 timewait; +__u32 mra_recv; +__u32 rej_recv; +__u32 lap_error; +__u32 lap_recv; +__u32 apr_recv; +__u32 unexp_event; + +/* outbound CM messages */ +__u32 req_sent; +__u32 rep_sent; +__u32 rtu_sent; +__u32 rej_sent; +__u32 dreq_sent; +__u32 drep_sent; +__aligned_u64 wait_time; +__aligned_u64 resolve_time; +__aligned_u64 connect_time; +__aligned_u64 connected_time; +__u32 resolve; +__u32 resolve_fail; +__u32 conn_recovery; +__aligned_u64 rewait_time; +__aligned_u64 reresolve_time; +__aligned_u64 reconnect_time; +__aligned_u64 max_rewait_time; +__aligned_u64 max_reresolve_time; +__aligned_u64 max_reconnect_time; +__u32 reresolve; +__u32 reresolve_fail; + +__aligned_u64 post_write; +__aligned_u64 post_write_fail; +__aligned_u64 post_write_bytes; +__u32 outstand_send_write; +__aligned_u64 send_write_cqe; +__aligned_u64 send_write_cqe_fail; + +__aligned_u64 recv_write_cqe; +__aligned_u64 recv_write_bytes; +__aligned_u64 recv_cqe_fail; + +__aligned_u64 post_hb; +__aligned_u64 post_hb_fail; +__aligned_u64 send_hb_cqe; +__aligned_u64 send_hb_cqe_fail; +__aligned_u64 recv_hb_cqe; +}; + +struct rv_conn_get_stats_params { +union { +struct rv_conn_get_stats_params_in in; +struct rv_conn_get_stats_params_out out; + }; +}; + +#define RV_IOCTL_CONN_GET_STATS _IOWR(RV_MAGIC, 0xFD, \ + struct rv_conn_get_stats_params) + +/* +* send_write_cqe - successful sent RDMA Write CQE +* send_write_cqe_fail - sent RDMA Write CQE with bad status +* send_write_bytes - for successful send +* recv_write_cqe - successful recv RDMA Write CQE +* recv_write_cqe_fail - recv RDMA Write CQE with bad status +* recv_write_bytes - successful recv RDMA Write +*/ +struct rv_event_stats_params_out { +__aligned_u64 send_write_cqe; +__aligned_u64 send_write_cqe_fail; +__aligned_u64 send_write_bytes; + +__aligned_u64 recv_write_cqe; +__aligned_u64 recv_write_cqe_fail; +__aligned_u64 recv_write_bytes; +}; + +#define RV_IOCTL_GET_EVENT_STATS _IOR(RV_MAGIC, 0xFE, \ + struct rv_event_stats_params_out) + +/* +* handle - from create_conn +* application source buffer and a kernel lkey for it +* loc_addr +* loc_mr_handle +* local MR - selected by 3-tuple (addr, len, access) +* loc_mr_addr +* loc_mr_length +* loc_mr_access +* remote application dest buffer and a kernel rkey for it +* rem_addr +* rkey +* length +* wr_id - application context, included in RV SQ completion events +* immed +*/ +struct rv_post_write_params_in { +__aligned_u64 handle; +__aligned_u64 loc_addr; +__aligned_u64 loc_mr_handle; +__aligned_u64 loc_mr_addr; +__aligned_u64 loc_mr_length; +__u32 loc_mr_access; +__u32 rkey; +__aligned_u64 rem_addr; +__aligned_u64 length; +__aligned_u64 wr_id; +__u32 immed; +}; + +struct rv_post_write_params_out { +__u8 sconn_index; +__u32 conn_count; +}; + +struct rv_post_write_params { +union { +struct rv_post_write_params_in in; +struct rv_post_write_params_out out; + }; +}; + +#define RV_IOCTL_POST_RDMA_WR_IMMED _IOWR(RV_MAGIC, 0xFB, \ + struct rv_post_write_params) + +enum rv_event_type { +RV_WC_RDMA_WRITE ,/* send RDMA Write CQE */ +RV_WC_RECV_RDMA_WITH_IMM ,/* recv RDMA Write w/immed CQE */ +}; + +/* +* events placed on ring buffer for delivery to user space. +* Carefully sized to be a multiple of 64 bytes for cache alignment. +* Must pack to get good field alignment and desired 64B overall size +* Unlike verbs, all rv_event fields are defined even when +* rv_event.wc.status != IB_WC_SUCCESS. Only sent writes can report bad status. +* event_type - enum rv_event_type +* wc - send or recv work completions +* status - ib_wc_status +* resv1 - alignment +* imm_data - for RV_WC_RECV_RDMA_WITH_IMM only +* wr_id - PSM wr_id for RV_WC_RDMA_WRITE only +* conn_handle - conn handle. For efficiency in completion processing, this +* handle is the rv_conn handle, not the rv_user_conn. +* Main use is sanity checks. On Recv PSM must use imm_data to +* efficiently identify source. +* byte_len - unlike verbs API, this is always valid +* resv2 - alignment +* cache_align - not used, but forces overall struct to 64B size +*/ +struct rv_event { +__u8 event_type; +union { +struct { +__u8 status; +__u16 resv1; +__u32 imm_data; +__aligned_u64 wr_id; +__aligned_u64 conn_handle; +__u32 byte_len; +__u32 resv2; + } __attribute__((__packed__ )) wc; +struct { +__u8 pad[7 ]; +uint64_t pad2[7 ]; + } __attribute__((__packed__ )) cache_align; + }; +} __attribute__((__packed__ )); + +/* +* head - consumer removes here +* tail - producer adds here +* overflow_cnt - number of times producer overflowed ring and discarded +* pad - 64B cache alignment for entries +*/ +struct rv_ring_header { +volatile __u32 head; +volatile __u32 tail; +volatile __u64 overflow_cnt; +__aligned_u64 pad[6 ]; +struct rv_event entries[]; +}; + +#define RV_RING_ALLOC_LEN (num_entries) \ + ((__u32)(((num_entries) * sizeof(struct rv_event)) + \ + sizeof(struct rv_ring_header))) + +#endif /* __RV_USER_IOCTL_H__ */
Signature · 1 line
2.18.1
Jason Gunthorpe
maintainerRe: [PATCH RFC 1/9] RDMA/rv: Public interferce for the RDMA Rendezvous module
Quoted 33 lines · On Fri, Mar 19, 2021 at 08:56:27AM -0400, [email protected] wrote:
From: Kaike Wan <[email protected]>
The RDMA Rendezvous (rv) module provides an interface for HPC middlewares to improve performance by caching memory region registration, and improve the scalibity of RDMA transaction through connection managements between nodes. This mechanism is implemented through the following ioctl requests:
- ATTACH: to attach to an RDMA device.
-REG_MEM: to register a user/kernel memory region.
-DEREG_MEM: to release application use of MR, allowing it to remain in cache.
-GET_CACHE_STATS: to get cache statistics.
-CONN_CREATE: to create an RC connection.
-CONN_CONNECT: to start the connection.
-CONN_GET_CONN_COUNT: to use as part of error recovery from lost messages in the application.
-CONN_GET_STATS: to get connection statistics.
-GET_EVENT_STATS: to get the RDMA event statistics.
-POST_RDMA_WR_IMMED: to post an RDMA WRITE WITH IMMED request.Signed-off-by Todd Rimmer <[email protected]>Signed-off-by Kaike Wan <[email protected]>include/uapi/rdma/rv_user_ioctls.h | 558 +++++++++++++++++++++++++++++ 1 file changed, 558 insertions(+) create mode 100644 include/uapi/rdma/rv_user_ioctls.hinclude/uapi/rdma/rv_user_ioctls.h new file +3
-0+++ b/include/uapi/rdma/rv_user_ioctls.h@@ -0,0 +1,558 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ +/*
include/uapi/rdma/rv_user_ioctls.h line 12
Quoted 10 lines
include/uapi/rdma/rv_user_ioctls.h new file +10
-0+* Copyright(c) 2020 - 2021 Intel Corporation. +*/ +#ifndef __RV_USER_IOCTL_H__ +#define __RV_USER_IOCTL_H__ +#include <rdma/rdma_user_ioctl.h> +#include <rdma/ib_user_sa.h> +#include <rdma/ib_user_verbs.h> + +/* Checking /Documentation/userspace-api/ioctl/ioctl-number.rst */ +#define RV_MAGIC RDMA_IOCTL_MAGIC
No.
include/uapi/rdma/rv_user_ioctls.h line 13
Quoted 1 line
include/uapi/rdma/rv_user_ioctls.h new file +1
-0+#define RV_FILE_NAME "/dev/rv"
No.
include/uapi/rdma/rv_user_ioctls.h line 22
Quoted 9 lines
include/uapi/rdma/rv_user_ioctls.h new file +9
-0+ +/* +* Handles are opaque to application; they are meaningful only to the +* RV driver +*/ + +/* this version of ABI */ +#define RV_ABI_VER_MAJOR 1 +#define RV_ABI_VER_MINOR 0
No, see my remarks to your other intel colleagues doing the ioasid stuff.
include/uapi/rdma/rv_user_ioctls.h line 30
Quoted 7 lines
include/uapi/rdma/rv_user_ioctls.h new file +7
-0+struct rv_query_params_out { +/* ABI version */ +__u16 major_rev; +__u16 minor_rev; +__u32 resv1; +__aligned_u64 capability; +__aligned_u64 resv2[6 ];
No pre-adding reserved stuff
include/uapi/rdma/rv_user_ioctls.h line 37
Quoted 7 lines
include/uapi/rdma/rv_user_ioctls.h new file +7
-0+}; + +#define RV_IOCTL_QUERY _IOR(RV_MAGIC, 0xFC, struct rv_query_params_out) + +/* Mode for use of rv module by PSM */ +#define RV_RDMA_MODE_USER 0/* user MR caching only */ +#define RV_RDMA_MODE_KERNEL 1/* + kernel RC QPs with kernel MR caching */
Huh? Mode sounds bad.
Quoted 5 lines
include/uapi/rdma/rv_user_ioctls.h new file +5
-0+/* +* mr_cache_size is in MBs and if 0 will use module param as default +* num_conn - number of QPs between each pair of nodes +* loc_addr - used to select client/listen vs rem_addr +* index_bits - num high bits of immed data with rv index
include/uapi/rdma/rv_user_ioctls.h line 60
Quoted 10 lines
include/uapi/rdma/rv_user_ioctls.h new file +10
-0+* loc_gid_index - SGID for client connections +* loc_gid[16] - to double check gid_index unchanged +* job_key[RV_MAX_JOB_KEY_LEN] = unique uuid per job +* job_key_len - len, if 0 matches jobs with len==0 only +* q_depth - size of QP and per QP CQs +* reconnect_timeout - in seconds from loss to restoration +* hb_interval - in milliseconds between heartbeats +*/ +struct rv_attach_params_in { +char dev_name[RV_MAX_DEV_NAME_LEN ];
Guessing this is a no too.
include/uapi/rdma/rv_user_ioctls.h line 66
Quoted 6 lines
include/uapi/rdma/rv_user_ioctls.h new file +6
-0+__u32 mr_cache_size; +__u8 rdma_mode; + +/* additional information for RV_RDMA_MODE_KERNEL */ +__u8 port_num; +__u8 num_conn;
Lots of alignment holes, don't do that either.
include/uapi/rdma/rv_user_ioctls.h line 99
Quoted 6 lines
include/uapi/rdma/rv_user_ioctls.h new file +6
-0+struct rv_attach_params { +union { +struct rv_attach_params_in in; +struct rv_attach_params_out out; + }; +};
Yikes, no
include/uapi/rdma/rv_user_ioctls.h line 104
Quoted 2 lines
include/uapi/rdma/rv_user_ioctls.h new file +2
-0+/* The buffer is used to register a kernel mr */ +#define IBV_ACCESS_KERNEL 0x80000000
Huh? WTF on on many levels
include/uapi/rdma/rv_user_ioctls.h line 113
Quoted 8 lines
include/uapi/rdma/rv_user_ioctls.h new file +8
-0+/* +* ibv_pd_handle - user space appl allocated pd +* ulen - driver_udata inlen +* *udata - driver_updata inbuf +*/ +struct rv_mem_params_in { +__u32 ibv_pd_handle; +__u32 cmd_fd_int;
Really? You want to reach into the command FD from a ULP and extract objects? Double yikes. Did you do this properly, taking care of every lifetime rule and handling disassociation?
include/uapi/rdma/rv_user_ioctls.h line 118
Quoted 5 lines
include/uapi/rdma/rv_user_ioctls.h new file +5
-0+__aligned_u64 addr; +__aligned_u64 length; +__u32 access; +size_t ulen; +void *udata;
'void *' is wrong for ioctls.
include/uapi/rdma/rv_user_ioctls.h line 356
Quoted 1 line
include/uapi/rdma/rv_user_ioctls.h new file +1
-0+struct rv_conn_get_stats_params_out {
Too many stats, don't you think? Most of the header seems to be stats of one thing or another.
Quoted 28 lines
include/uapi/rdma/rv_user_ioctls.h new file +28
-0+/* +* events placed on ring buffer for delivery to user space. +* Carefully sized to be a multiple of 64 bytes for cache alignment. +* Must pack to get good field alignment and desired 64B overall size +* Unlike verbs, all rv_event fields are defined even when +* rv_event.wc.status != IB_WC_SUCCESS. Only sent writes can report bad status. +* event_type - enum rv_event_type +* wc - send or recv work completions +* status - ib_wc_status +* resv1 - alignment +* imm_data - for RV_WC_RECV_RDMA_WITH_IMM only +* wr_id - PSM wr_id for RV_WC_RDMA_WRITE only +* conn_handle - conn handle. For efficiency in completion processing, this +* handle is the rv_conn handle, not the rv_user_conn. +* Main use is sanity checks. On Recv PSM must use imm_data to +* efficiently identify source. +* byte_len - unlike verbs API, this is always valid +* resv2 - alignment +* cache_align - not used, but forces overall struct to 64B size +*/ +struct rv_event { + __u8 event_type; + union { + struct { + __u8 status; + __u16 resv1; + __u32 imm_data; + __aligned_u64 wr_id;
include/uapi/rdma/rv_user_ioctls.h line 538
Quoted 10 lines
include/uapi/rdma/rv_user_ioctls.h new file +10
-0+__aligned_u64 conn_handle; +__u32 byte_len; +__u32 resv2; + } __attribute__((__packed__)) wc; +struct { +__u8 pad[7 ]; +uint64_t pad2[7 ]; + } __attribute__((__packed__ )) cache_align; + }; +} __attribute__((__packed__));
Uhh, mixing packed and aligned_u64 is pretty silly. I don't think this needs to be packed or written in this tortured way.
include/uapi/rdma/rv_user_ioctls.h line 549
Quoted 11 lines
include/uapi/rdma/rv_user_ioctls.h new file +11
-0+ +/* +* head - consumer removes here +* tail - producer adds here +* overflow_cnt - number of times producer overflowed ring and discarded +* pad - 64B cache alignment for entries +*/ +struct rv_ring_header { +volatile __u32 head; +volatile __u32 tail; +volatile __u64 overflow_cnt;
No on volatile, and missed a __aligned here
This uapi needs to be much better. It looks like the mess from the PSM char dev just re-imported here.
At the very least split the caching from the other operations and follow normal ioctl design
And you need to rethink the uverbs stuff.
Jason
kernel test robot
build botRe: [PATCH RFC 1/9] RDMA/rv: Public interferce for the RDMA Rendezvous module
In reply to [email protected][If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on rdma/for-next]
[also build test ERROR on linus/master v5.12-rc3 next-20210319] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/kaike-wan-intel-com/A-rendezvous-module/20210319-210005 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: i386-randconfig-s002-20210318 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.3-277-gc089cd2d-dirty # https://github.com/0day-ci/linux/commit/96b4d25534a5c1ebaab65a35fd9cd242613cd6d1 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review kaike-wan-intel-com/A-rendezvous-module/20210319-210005 git checkout 96b4d25534a5c1ebaab65a35fd9cd242613cd6d1 # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from <command-line>:32:
>> ./usr/include/rdma/rv_user_ioctls.h:117:2: error: unknown type name 'size_t'
117 | size_t ulen;
| ^~~~~~
>> ./usr/include/rdma/rv_user_ioctls.h:535:4: error: unknown type name 'uint64_t'
535 | uint64_t pad2[7];
| ^~~~~~~~---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 2/9 RDMA/rv: Add the internal header files
In reply to [email protected]From: Kaike Wan <[email protected]>
The two header files include the defines, structures, MACROs, function prototypes, and functions used in the module.
--- drivers/infiniband/ulp/rv/rv.h | 892 ++++++++++++++++++++++++ drivers/infiniband/ulp/rv/rv_mr_cache.h | 152 ++++ 2 files changed, 1044 insertions(+) create mode 100644 drivers/infiniband/ulp/rv/rv.h create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.h
drivers/infiniband/ulp/rv/rv.h new file +892 -0
@@ -0,0 +1,892 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +/* +* Copyright(c) 2020 - 2021 Intel Corporation. +*/ + +#ifndef __RV_H__ +#define __RV_H__ +#include <linux/types.h> +#include <linux/list.h> +#include <linux/mutex.h> +#include <linux/scatterlist.h> +#include <rdma/ib_verbs.h> +#include <rdma/ib_sa.h> +#include <rdma/ib_cm.h> +#include <rdma/rdma_cm.h> +#include <linux/mutex.h> +#include <linux/socket.h> +#include <linux/timer.h> +#include <rdma/ib_addr.h> +#include <rdma/ib_cm.h> +#include <linux/moduleparam.h> + +#include "rv_mr_cache.h" +#include <rdma/rv_user_ioctls.h> + +/* +* Lock Heirachy +* In order that locks can be acquired: +* mm->mmap_lock - held during mmap call and mmu_notifier_register +* rv_user.mutex +* rv_job_dev_list_mutex +* rv_job_dev.conn_list_mutex +* rv_device.listener_mutex +* rv_sconn.mutex - we can never hold this while calling destroy_cm_id +* because destroy cm_id will wait for handlers via it's mutex and deadlock +* rv_job_dev_list rcu +* rv_dev_list_lock +* rv_job_dev.conn_list rcu - no sub locks +* rv_device.listener_lock - no sub locks +* rv_user.umrs.cache.lock - no sub locks +* rv_job_dev.user_array_lock - no sub locks +* ring.lock - no sub locks +* mr_pd_uobject_lock - no sub locks +* rv_conn.next_lock - no sub locks +* rv_sconn.drain_lock - no sub locks +*/ + +/* +* Our goal is to allocate shared resources, but all processes are equal peers. +* So we use a get_alloc approach where processes attempt to get a resource +* reference (lookup and get) and if it's not on the list, it is allocated and +* added. This is used for rv_job_dev, rv_listener and rv_conn. +* Each such list is protected by a mutex to prevent duplicate additions +* and an RCU for actual list access. The mutex also protects +* get_alloc/list_del races. All get_alloc calls are in premptable +* app ioctl call context. +* rv_listener list is rarely searched, so it simply uses a mutex and spinlock. +*/ + +/* +* When working with IB CM, Async events and timeouts we can't predict if and +* when their callbacks will occur so they can't have a reference held in +* advance. This presents opportunities for the callback to race with rv_conn +* destruction. To solve this problem, we use rv_conn_get_check to get a +* reference to rv_conn only if rv_conn.kref != 0. rv_conn and rv_sconn +* destruction uses destroy_cm_id and del_timer_sync to stop all callbacks (and +* allow any outstanding handlers to finish) before it proceeds to free rv_conn. +* This ensures the cm_id, etc can remain valid enough for the callback to +* call rv_conn_get_check and decide whether to simply ignore the event. +* It also allows the rv_sconn to be the ib_cm, CQ, QP and timer context and +* avoid searches. Once the callback has the reference, it is protected +* from other threads calling rv_conn_release. +*/ + +#define DRIVER_NAME "rv" + +#define RV_INVALID -1 + +/* For errors to surface on the console */ +#define rv_err (idx, fmt, ...) \ + pr_err("[%s:%s %d]: " fmt, DRIVER_NAME, __func__, idx, ##__VA_ARGS__) + +#define rv_ptr_err (prefix, ptr, fmt, ...) \ + pr_err("[%s:%s %s 0x%p]: " fmt, DRIVER_NAME, __func__, prefix, ptr, \ + ##__VA_ARGS__) + +/* For debugging*/ +#define rv_dbg (idx, fmt, ...) \ + pr_debug("[%s:%s %d]: " fmt, DRIVER_NAME, __func__, idx, ##__VA_ARGS__) + +/* For general console info */ +#define rv_info (idx, fmt, ...) \ + pr_info("[%s:%s %d]: " fmt, DRIVER_NAME, __func__, idx, ##__VA_ARGS__) + +/* For debugging with any pointer */ +#define rv_ptr_dbg (prefix, ptr, fmt, ...) \ + pr_debug("[%s:%s %s 0x%p]: " fmt, DRIVER_NAME, __func__, prefix, ptr, \ + ##__VA_ARGS__) + +#define rv_ptr_info (prefix, ptr, fmt, ...) \ + pr_info("[%s:%s %s 0x%p]: " fmt, DRIVER_NAME, __func__, prefix, ptr, \ + ##__VA_ARGS__) + +/* For debugging with rv_conn_info */ +#define rv_conn_err (ptr, fmt, ...) rv_ptr_err("sconn", ptr, fmt, ##__VA_ARGS__) +#define rv_conn_info (ptr, fmt, ...) rv_ptr_info("sconn", ptr, fmt,\ + ##__VA_ARGS__) +#define rv_conn_dbg (ptr, fmt, ...) rv_ptr_dbg("sconn", ptr, fmt, ##__VA_ARGS__) + +/* For debugging with rv_device */ +#define rv_dev_err (ptr, fmt, ...) rv_ptr_err("dev", ptr, fmt, ##__VA_ARGS__) +#define rv_dev_info (ptr, fmt, ...) rv_ptr_info("dev", ptr, fmt, ##__VA_ARGS__) +#define rv_dev_dbg (ptr, fmt, ...) rv_ptr_dbg("dev", ptr, fmt, ##__VA_ARGS__) + +/* For debugging with ib cm id */ +#define rv_cm_err (ptr, fmt, ...) rv_ptr_err("cm_id", ptr, fmt, ##__VA_ARGS__) +#define rv_cm_info (ptr, fmt, ...) rv_ptr_info("cm_id", ptr, fmt, ##__VA_ARGS__) +#define rv_cm_dbg (ptr, fmt, ...) rv_ptr_dbg("cm_id", ptr, fmt, ##__VA_ARGS__) + +struct rv_device ; + +/* +* A listener can handle more than 1 job on a given dev. +* There is a listener for each unique service_id on each dev and +* could be as few as 1 even for multiple jobs. +* For IB, a listener services all ports on a given HCA. +* listener_entry - entry on rv_device.listener_list +* dev - device being listened on +* cm_id - actual IB CM listener, stores service_id here +*/ +struct rv_listener { +struct list_head listener_entry; +struct rv_device *dev; +struct ib_cm_id *cm_id; +struct kref kref; +}; + +/* +* For each physical device, RV establishes a single rv_device which +* is shared across all jobs, listeners, etc. +* ib_dev - underlying RDMA device +* dev_entry - entry on rv_dev_list +* +* IB cm listener management: +* listener_mutex, listener_lock, listener_list +* +* user_list - list of rv_user (protected by rv_dev_list_lock) +*/ +struct rv_device { +struct ib_device *ib_dev; +struct list_head dev_entry; +struct kref kref; +struct ib_event_handler event_handler; + +struct mutex listener_mutex;/* avoid duplicate add in get_alloc */ +spinlock_t listener_lock;/* protect list search, add, remove */ +struct list_head listener_list; + +struct list_head user_list; +}; + +#define RV_CONN_MAX_ACTIVE_WQ_ENTRIES 100/* used for conn handling */ + +#define RV_RESOLVER_RETRY 5/* max retries (0 == try once) */ +#define RV_RESOLVER_TIMEOUT 10000/* in milliseconds */ + +/* duration client spends in RV_DELAY before re-attempt reconnect */ +#define RV_RECONNECT_DELAY msecs_to_jiffies(100) + +/* private data in REQ and REP is used to negotiate features and exchange +* additional informaton. +* REQ.ver is the version of the sender and defines the priv_data structure. +* Newer versions of struct should be a superset of older versions. +* REP.ver is the version the listener is accepting and defines REP priv_data. +* New client, old listerner: +* listener accepts REQ >= it's version, responds with it's version in REP +* Old client, new listener: +* listener accepts REQ, responds with REP matching client's version +* Protocol ver is now 2. For simplicity protocol 0 and 1 (pre-releases) are +* not accepted. +*/ + +#define RV_PRIVATE_DATA_MAGIC 0x00125550534d2121ULL +#define RV_PRIVATE_DATA_VER 2 + +/* +* Private data used in CM REQ +* laid out for good alignment of fields without packing +* index indicates index of rv_sconn within it's rv_conn and differentiates +* the multiple connection REQs for num_conn > 1 +* The job_key selects the rv_job_dev on receiver side +* uid is 32b +*/ +struct rv_req_priv_data { +u64 magic; +u32 ver; +u16 resv1; +u8 index; +u8 job_key_len; +u8 job_key[RV_MAX_JOB_KEY_LEN ]; +uid_t uid; +}; + +/* +* Private data used in CM REP +* laid out for good alignment of fields without packing +*/ +struct rv_rep_priv_data { +u64 magic; +u32 ver; +}; + +/* +* RV_INIT - client side create_conn done +* RV_WAITING - server side create_conn done, listening +* RV_RESOLVING - client side resolving ethernet dmac via ARP +* RV_CONNECTING - client or server side going through CM states +* RV_CONNECTED - connection established +* RV_DISCONNECTING - connection teardown +* RV_DELAY- RV delay before restarting client connection +* RV_ERROR - connection lost or failed +* destroying - ref count == 0 +*/ +enum rv_sconn_state { +RV_INIT =1 , +RV_WAITING , +RV_RESOLVING , +RV_CONNECTING , +RV_CONNECTED , +RV_DISCONNECTING , +RV_DELAY , +RV_ERROR , +}; + +/* +* Meaning of each state and status of key fields once in state and +* have unlocked mutex. +* In all states (except while destroying) immutable fields in rv_conn valid. +* +* WAS_CONNECTED flag essentially creates a superstate indicating connection +* recovery for WAITING, RESOLVING, and CONNECTING. During connection +* recovery, conn_timer is running for reconnect_timeout and once it fires, +* the connect recovery is aborted and moved to ERROR +* +* If WAS_CONNECTED, post write in wrong state returns EAGAIN instead of EINVAL +* +* A complication is identifying when a connection is down for a receiving +* end of a traffic pattern. The receiver may see no packets and can't tell +* a path down from an idle QP. To address this periodic RDMA Write zero +* can be sent if no traffic has been sent or received for a while. This +* situation is no worse that user space UD PSM as the job failure may +* occur over an out of band network to kill the job. A gap is the client side +* of a connection desiring recovery, which requires the heatbeat to recognize. +* +* If the receiver happens to also be the server side of rv_sconn, we may +* get a REQ while in connected because the sender may get a QP timeout +* long before the receiver heartbeat notices. We treat this as disconnect +* and if appropriate (likely) begin connection recovery. +* +* RV_INIT: initial state for client side connections (after 1st create_conn) +* cm_id, primary_path are NULL +* dev_addr, resolver_retry_left uninitialized +* qp in RESET state, no outstanding CQEs nor WQEs +* conn_timer not running (no reconnect_timeout) +* delay_timer not running +* hb_timer not running +* Next States: +* RESOLVING - user calls cm_connect +* destroying - user close +* RV_WAITING: initial state for server side connections (after 1st create_conn) +* a listener exists at rv_job_dev level, rv_cm_server_handler +* cm_id NULL +* start_time is set when enter state +* qp in RESET state, no outstanding CQEs nor WQEs +* if WAS_CONNECTED, conn_timer is running for reconnect_timeout +* delay_timer not running +* hb_timer not running +* DRAINING, SQ_DRAINED and RQ_DRAINED flags clear +* Next States: +* CONNECTING - inbound REQ +* ERROR - reconnect_timeout expires +* destroying - user close +* RV_RESOLVING: 1st step in establishing a client connection (ARP) +* For non-ethernet, this is a brief transient state (only inside mutex) +* cm_id established (client side), rv_cm_handler +* resolver_retry_left valid +* primary_path != NULL, but contents incomplete +* dmac, route_resolved, hop__limit uninitialized +* dev_addr undefined +* start_time is set when enter state +* a rdma_resolve_ip callback is scheduled +* qp in RESET state, no outstanding CQEs nor WQEs +* if WAS_CONNECTED, conn_timer is running for reconnect_timeout +* delay_timer not running +* hb_timer not running +* DRAINING, SQ_DRAINED and RQ_DRAINED flags clear +* Next States: +* CONNECTING - resolution successfully complete +* ERROR - resolving hard fail or retry exceeded or connect timeout +* RESOLVING - cb error and < retry limit +* DELAY - cb error > retrylimit and WAS_CONNECTED and reconnect +* destroying - user close +* RV_CONNECTING: client or server connection in hands of IB CM +* cm_id established (either side), rv_cm_handler +* primary_path NULL +* dev_addr valid on client side +* resolver_retry_left undefined +* start_time is set when enter state +* CM may have a rep or rtu outstanding +* qp progressing through RESET->INIT->RTR->RTS via IB CM states/events +* if WAS_CONNECTED, conn_timer is running for reconnect_timeout +* delay_timer not running +* hb_timer not running +* DRAINING, SQ_DRAINED and RQ_DRAINED flags clear +* Next States: +* CONNECTED - client gets REP, server gets RTU or Established +* ERROR - REJ, REQ err, REP err, DREQ, QP event or connect timeout +* DISCONNECTING - WAS_CONNECTED and REQ err, REJ, REP err, QP evt +* or REQ +* destroying - user close +* RV_CONNECTED: client or server connection usable +* cm_id established (either side), rv_cm_handler +* primary_path NULL +* dev_addr valid on client side +* resolver_retry_left undefined +* no outbound CM operations outstanding +* start_time is set when enter state +* qp in RTS, has recv WQEs posted +* qp may have outstanding CQEs and WQEs +* conn_timer not running (no reconnect_timeout) +* delay_timer not running +* hb_timer may be running (client side only) +* WAS_CONNECTED is set on entry +* DRAINING, SQ_DRAINED and RQ_DRAINED flags clear +* heartbeat timer running +* record post SQ and RQ CQE count when start timer +* when fires, if counts same, send 0 len RDMA Write +* TBD - set larger than timeout*retry or simply track SQ so +* don't overflow SQ +* TBD - stagger heartbeat sends on different sconns +* Next States: +* ERROR - no reconnect and get DREQ, QP event, REQ +* DISCONNECTING - reconnect and get DREQ, QP event (send DREQ), +* get REQ (remote end reconnecting) +* destroying - user close +* RV_DISCONNECTING: client or server connection cleanup in prep for reconnect +* cm_id established (either side), rv_cm_handler +* primary_path NULL +* dev_addr valid on client side +* resolver_retry_left undefined +* DREP or DREQ outbound CM operations may be outstanding +* start_time is set when enter state +* qp in ERROR, SCONN_DRAINING flag set, +* - waiting for "drain indicator" WQEs' CQEs on RQ and SQ +* - drain indicator CQEs set SQ_DRAINED or RQ_DRAINED +* qp may have other outstanding CQEs and WQEs +* if WAS_CONNECTED, conn_timer may be is running for reconnect_timeout +* delay_timer not running +* hb_timer not running +* Note: we do not depend on DREP and ignore DREQ Err or DREP Err +* path to remote node may be down +* Next States: +* ERROR - unable to queue CM_id destroy or reset QP +* or connect timeout +* DELAY - SQ & RQ drain finish for client +* WAITING - SQ & RQ drain finish for server (reset QP, +* cm_id NULL and queued for destroy) +* destroying - user close +* RV_DELAY: client reconnection delay +* cm_id established, rv_cm_handler +* primary_path NULL +* dev_addr valid on client side +* resolver_retry_left undefined +* DREP or DREQ outbound CM operations may be outstanding on prior cm_id +* start_time is set when enter state +* qp in RESET +* prior qp has no outstanding CQEs and WQEs +* if WAS_CONNECTED, conn_timer may be is running for reconnect_timeout +* delay_timer is running +* hb_timer not running +* DRAINING, SQ_DRAINED and RQ_DRAINED flags clear +* while in this state all other errors are ignored +* Next States: +* RESOLVING - timer expires +* ERROR - reconnect timeout +* destroying - user close +* RV_ERROR: terminal state for lost/failed connection +* may enter from any state +* cm_id may be NULL or established with rv_cm_handler +* CM may have an outstanding message +* typically REJ, DREQ or DREP, but could be REQ, REP or RTU +* primary_path NULL +* dev_addr, resolver_retry_left undefined +* no rdma_resolve_ip callback scheduled (cancel when enter RV_ERROR) +* qp in ERROR, SCONN_DRAINING flag set or qp in RESET or no QP +* SQ_DRAINED and RQ_DRAINED flags may be progressing on drain +* qp may have outstanding CQEs and WQEs +* conn_timer not running (no reconnect_timeout) +* delay_timer not running +* hb_timer not running +* Next States: +* destroying - user close +* destroying: rv_conn kref is 0 +* may enter from any state +* on entry: +* cm_id may be NULL or established with rv_cm_handler +* a rdma_resolve_ip callback may be scheduled +* qp may have outstanding CQEs and WQEs and callbacks active +* conn_timer may be running (reconnect_timeout or reconnect delay) +* all fields (including immitable) will be released via rv_sconn_deinit +* no way out other than kfree of parent rv_conn +* may prempt, does not require mutex (kref==0 protects races with cb) +* Next States: +* N/A - conn/sconn freed +* +* typical client side state sequence: +* RV_INIT->RV_RESOLVING->RV_CONNECTING->RV_CONNECTED->destroying +* typical server side state sequence: +* RV_WAITING->RV_CONNECTING->RV_CONNECTED->destroying +* +* typical client side recovery state sequence: +* ...->RV_CONNECTED->DISCONNECTING->DELAY->RV_RESOLVING->... +* typical server side recovery state sequence: +* ...->RV_CONNECTED->DISCONNECTING->WAITING->... +*/ + +/* +* rv_sconn.flags bit numbers +* RV_SCONN_SERVER - server vs client side (immutable) +* RV_SCONN_WAS_CONNECTED - was RV_CONNECTED at least once +* RV_SCONN_DRAINING - started draining in DISCONNECTING +* RV_SCONN_SQ_DRAINED - SQ drained in DISCONNECTING +* RV_SCONN_RQ_DRAINED - RQ drained in DISCONNECTING +* RV_SCONN_ASYNC - got async event +*/ +#define RV_SCONN_SERVER 0 +#define RV_SCONN_WAS_CONNECTED 1 +#define RV_SCONN_DRAINING 2 +#define RV_SCONN_SQ_DRAINED 3 +#define RV_SCONN_RQ_DRAINED 4 + +/* +* rv_sconn.stats.cm_evt_cnt[] classify events > CM_APR_RECEIVED as unexpected. +* Beyond that are only SIDR events +*/ +#define RV_CM_EVENT_MAX IB_CM_APR_RECEIVED +#define RV_CM_EVENT_UNEXP ((enum ib_cm_event_type)(RV_CM_EVENT_MAX + 1)) + +/* +* a single QP/connection +* mutex - protects state driven fields +* +* These are set once at create and can be read without lock +* index - unique index for connection within rv_conn +* qp, send_cq, recv_cq, max_send_wr (from ib_qp_cap) +* cqe - for recv completions +* +* Fields below require the mutex and their validity depends on the current +* value of rv_sconn.state and the RV_SCONN_SERVER flag. +* +* Basic fields: +* state, flags +* start_time - when started waiting, resolving or connecting +* cm_id - our actual connection +* path - from PSM. Consistency checked on listener, for client connect +* we use path.dlid != 0 to identify if path has been initialized +* +* Client only fields: +* resolver_retry_left, primary_path, dev_addr +* For RoCE our resolver step fills in dev_addr with resolved RoCE HW +* addresses (aka MAC address) +* +* Async QP Draining, +* drain_lock - protects these and enter;test of RC_SCONN_*DRAIN* flags +* drain_lock, rdrain_cqe, sdrain_cqe, drain_work (for drain done handler) +* drain_timer - for drain timeout +* done_wr_list - most recent completed pend_wr's +* done_wr_count - number of entries on list +* +* reconnect_timeout timer: conn_timer, timer_work +* RV_DELAY timer: delay_timer, delay_work +* Heartbeat: hb_timer, act_count (activity since last hb), hb_cqe, hb_work +* +* Stats: +* all but atomic CQE stats protected by rv_sconn.mutex +* connection: +* *_time values are in microseconds +* max_* is the largest observed for all reconnects on this sconn +* cm_evt_cnt - extra +1 to include EVENT_MAX and +1 for UNEXP +* stats for each CM packet explicitly send (req, rep, rtu, dreq, drep) +* initial connect: wait_time, resolve_time, connect_time +* connect time does not include wait_time nor resolve_time +* resolve - attempts, resolve_fail - unexpected local issues +* connected_time - total time connected (after initial + after recovery) +* conn_recovery - # times recovered connection +* connect recovery: rewait_time, reresolve_time, reconnect_time +* reconnect time does not include wait_time nor resolve_time +* reresolve - attempts, reresolve_fail - unexpected local issues +* data movement: +* post_* is SQ posts (success, fail, payload byte count) +* outstand_send_write - current send writes waiting for CQE +* send_write_cqe_* is SQ cqes (RDMA Write w/Immediate) +* recv_write_cqe_* is RQ cqes (RDMA Write w/Immediate) +* recv_cqe_fail - RQ CQEs with bad status (opcode undefined) +* *_hb_* - heartbeat +*/ + +struct rv_sconn { +struct mutex mutex;/* lock for state driven fields */ +u8 index; +struct ib_qp *qp; +struct ib_cq *send_cq; +struct ib_cq *recv_cq; +struct ib_cqe cqe; +struct rv_conn *parent; + +unsigned long flags; +enum rv_sconn_state state; +ktime_t start_time; +struct ib_cm_id *cm_id; +struct ib_user_path_rec path; + +u32 resolver_retry_left; +struct sa_path_rec *primary_path; +struct rdma_dev_addr dev_addr; + +/* protects these & enter;test RC_SCONN_*DRAIN* flags */ +spinlock_t drain_lock; +struct ib_cqe rdrain_cqe; +struct ib_cqe sdrain_cqe; +struct work_struct drain_work; +struct timer_list drain_timer; + +struct timer_list conn_timer; +struct work_struct timer_work; + +struct timer_list delay_timer; +struct work_struct delay_work; + +struct timer_list hb_timer; +u64 act_count; +struct ib_cqe hb_cqe; +struct work_struct hb_work; + +struct { +u32 cm_evt_cnt[RV_CM_EVENT_MAX +2 ]; +u32 req_sent; +u32 rep_sent; +u32 rtu_sent; +u32 rej_sent; +u32 dreq_sent; +u32 drep_sent; +u64 wait_time; +u64 resolve_time; +u64 connect_time; +u64 connected_time; +u32 resolve; +u32 resolve_fail; +u32 conn_recovery; +u64 rewait_time; +u64 reresolve_time; +u64 reconnect_time; +u64 max_rewait_time; +u64 max_reresolve_time; +u64 max_reconnect_time; +u32 reresolve; +u32 reresolve_fail; +u64 post_write; +u64 post_write_fail; +u64 post_write_bytes; +u64 post_hb; +u64 post_hb_fail; + +atomic_t outstand_send_write; +atomic64_t send_write_cqe; +atomic64_t send_write_cqe_fail; +atomic64_t recv_write_cqe; +atomic64_t recv_write_bytes; +atomic64_t recv_cqe_fail; +atomic64_t send_hb_cqe; +atomic64_t send_hb_cqe_fail; +atomic64_t recv_hb_cqe; + } stats; +}; + +/* +* A load balanced multi QP connection using multiple underlying connections +* and is shared by multiple rv_user's. +* num_conn, jdev, ah and rem_addr are immutable (set once at create) +* Entry in rv_job_dev.conn_list: conn_entry, rcu +* sconn round robin IO: next_lock, next +* next_lock also protects read;inc of sconn->stats.outstand_send_write +*/ +struct rv_conn { +u8 num_conn; +struct rv_job_dev *jdev; +struct ib_uverbs_ah_attr ah; +u32 rem_addr; + +struct list_head conn_entry; +struct rcu_head rcu; + +struct kref kref; +struct work_struct put_work; + +spinlock_t next_lock;/* protect rv_conn.next & read;inc outstand_wr */ +u8 next; +struct work_struct free_work; + +struct rv_sconn sconn_arr[]; +}; + +/* +* from IBTA 1.4 Vol1 section A3.2.3.4. Externally Admined Service IDs +* 1h:OUI:OUI:OUI:hh:hh:hh:hh +* using Intel 00:12:55 OUI +*/ +#define RV_DFLT_SERVICE_ID 0x1000125500000001ULL + +/* +* the set of MRs registered by a single rv_user (on a single NIC) +* These are cached for efficiency. +* When using kernel rendezvous QPs (eg. rv_conn) these MRs are +* registered with the rv_job_dev.pd. +* When using user space QPs, we register with the user supplied pd +* cache has it's own lock +* jdev, rv_inx , CQs, QP set once at alloc +* need parent rv_user.mutex for: cqe, post REG_MR to QP (doit_reg_mem), stats +*/ +struct rv_user_mrs { +struct rv_mr_cache cache; +struct rv_job_dev *jdev; +int rv_inx;/* our creator, for logging */ + +struct kref kref; +struct work_struct put_work; + +struct { +u64 failed;/* cache miss and failed to register */ + } stats; +}; + +/* +* Resources shared among all rv_user's using a given NIC port within a job +* for a given kuid. On the wire we use uid to limit jobs to a single user. +* This approach has similarities to the hfi1 and Omni-Path jkey concept +* but we are not restricted to the 16b HW jkey field here, so we use all +* 32b of uid on the wire. +* +* There will typically be 1-8 NICs per node and 1-2 concurrent jobs +* with one rv_user per CPU core per NIC. +* +* kref tracks total jdev references while user_kref is the subset of +* references representing attached rv_user objects. user_kref <= kref. +* To get_alloc a jdev we must successfully get both. Once user_kref +* is zero, the jdev is removed from rv_job_dev_list and future attach +* or inbound REQ processing will not find it anymore. At this point it is +* destined for destruction as soon as the remaining callbacks with a reference +* occur/finish. jdev get_alloc searches which see (and skip) jdevs with +* user_kref==0 but kfef>0 only happen when a new job starts without a unique +* job key, as the previous job is cleaning up. Such cases are rare. +* By decoupling the jobs into different jdev's the new job may also have +* different attach and connection params. +* +* These fields are immutable and can be accessed without a lock: +* kuid, uid, dev, pd, dev_name, port_num, +* num_conn - number of rv_sconn per rv_conn +* index_bits - number of high bits of RDMA immed data to hold rv index +* loc_gid_index - SGID - only used on client rv_sconn +* loc_addr - abstraction of address compared with rem_addr to select +* client/server mode for each rv_sconn at conn_create time. +* log_gid - SGID - to double check loc_gid_index still is same GID +* service_id, q_depth +* qp_depth - max send and recv WQEs to use (N/A space for drain) +* reconnect_timeout (seconds), hb_interval (milliseconds), +* sgid_attr - local NIC gid & address for use by resolver +* max_users - 1<<index_bits +* +* job_dev_entry, rcu - entry on rv_job_dev_list +* conn_list - list of shared rv_conn, protected by RCU and conn_list_mutex +* conn_list_mutex prevents duplicate add in get_alloc +* listener - created and shared when 1st server rv_conn is added +* user_array[max_users] - rv_users sharing this jdev. user_array_lock protects +* rv_user.index - subscript to this array for given rv_user so +* RDMA recv immediate data can simply index to find which +* rv_user to deliver the completion to. +* user_array_next - where to start next search for free slot +*/ +struct rv_job_dev { +uid_t uid; +kuid_t kuid; +struct rv_device *dev; +struct ib_pd *pd; +char dev_name[RV_MAX_DEV_NAME_LEN ]; +u8 port_num; +u8 num_conn; +u8 index_bits; +u16 loc_gid_index; +u32 loc_addr; +u8 loc_gid[16 ]; +u8 job_key[RV_MAX_JOB_KEY_LEN ]; +u8 job_key_len; +u64 service_id; +u32 q_depth; +u32 qp_depth; +u32 reconnect_timeout; +u32 hb_interval; +const struct ib_gid_attr *sgid_attr; +int max_users; + +struct kref kref; +struct list_head job_dev_entry; +struct rcu_head rcu; + +struct mutex conn_list_mutex;/* prevent duplicate add in get_alloc */ +struct list_head conn_list; +struct rv_listener *listener; + +spinlock_t user_array_lock;/* protects add/remove from user_array */ +u32 user_array_next; +struct kref user_kref; +struct rv_user *user_array[]; +}; + +/* +* rv_user represents a single open fd from a user +* In multi-rail a process may have multiple rv_user (separate open/close) +* +* mutex - prevents concurrent ATTACH ioctl and protects conn_list +* also protects doit_reg vs self and vs doit_dreg races +* attached - set last during ATTACH after dev/jdev, rdma_mode and cq_entries +* have been set. We have no detach. +* inx - immutable ID assignd to rv_user strictly for logging +* rdma_mode - indicates USER (MRs only) or KERNEL (jdev, conns, etc) attach +* For rdma_mode KERNEL these are also valid: +* context, cq_entries +* index - rv_user index within rv_job_dev.user_array[] +* cqr - event ring to deliver send and recv RDMA completion +* events to PSM (only if ATTACH.cq_entries!=0) +* rv_user_mrs - MRs registered by this rv_user +* conn_xa - ID indexed list of rv_conn (entries assigned on create_conn) +* ID 0 is reserved, PSM uses this value to indicate uninitialized rv intf +* rv_user.mutex protects, so no need to use xas_lock. +* user_entry - entry in rv_device.user_list +* compl - completion for detach +*/ +struct rv_user { +struct mutex mutex;/* single thread most actions for a user */ + +int inx; + +u8 rdma_mode; +u8 attached; +u8 was_attached; +union { +struct rv_device *dev; +struct rv_job_dev *jdev; + }; + +u64 context; +u32 cq_entries; +u16 index; +struct rv_user_ring *cqr; +struct rv_user_mrs *umrs; + +struct xarray conn_xa; + +struct list_head user_entry; +struct completion compl; +}; + +/* +* an event ring for use by a single rv_user +* allows events to be efficiently passed from rv to PSM for PSM polling +* Immutable fields set on alloc: +* rv_inx - our creator's rv_user.inx (for logging) +* num_entries, page, order +* lock - protects kernel posting to ring and stats +* hdr - mmapped into PSM +* stats - index for each is rv_event_type excluding RV_TEST_EVENT +*/ +struct rv_user_ring { +int rv_inx; +u32 num_entries; +unsigned long page; +unsigned int order; + +spinlock_t lock;/* protects posting to ring and stats*/ +struct rv_ring_header *hdr; +struct { +u64 cqe[2 ]; +u64 cqe_fail[2 ]; +u64 bytes[2 ]; + } stats; +}; + +/* +* an inflight RDMA write on behalf of an rv_user +* +* user_index - rv_user index within rv_job_dev.user_array[] +* umrs - MR cache holding the local MR +* mrc, loc_addr - local MR (with lkey), source memory address for RDMA Write +* rkey, rem_addr - dest memory address for RDMA Write +* wr_id - for completion event to PSM (supplied by PSM) +* done_wr_entry - Entry in rv_sconn.done_wr_list +*/ +struct rv_pend_write { +struct ib_cqe cqe; +u16 user_index; +struct rv_user_mrs *umrs; +struct rv_sconn *sconn; + +struct rv_mr_cached *mrc; +u64 loc_addr; +u32 rkey; +u64 rem_addr; +size_t length; +u32 immed; +u64 wr_id; +}; + +extern unsigned int enable_user_mr; + +/* Prototypes */ +struct rv_device *rv_device_get_add_user (char *dev_name,struct rv_user *rv); +void rv_device_get (struct rv_device *dev); +void rv_device_put (struct rv_device *dev); +int rv_device_del_user (struct rv_user *rv); + +void rv_listener_get (struct rv_listener *listener); +void rv_listener_put (struct rv_listener *listener); +struct rv_listener *rv_listener_get_alloc (struct rv_device *dev, +u64 service_id, +ib_cm_handler handler); + +int rv_file_init (void ); +void rv_file_uninit (void ); +void rv_queue_work (struct work_struct *work); +void rv_queue_work2 (struct work_struct *work); +void rv_flush_work2 (void ); +void rv_queue_work3 (struct work_struct *work); + +int doit_reg_mem (struct rv_user *rv,unsigned long arg); +int doit_dereg_mem (struct rv_user *rv,unsigned long arg); +void rv_user_mrs_get (struct rv_user_mrs *umrs); +void rv_user_mrs_put (struct rv_user_mrs *umrs); +struct rv_user_mrs *rv_user_mrs_alloc (struct rv_user *rv,u32 cache_size); +void rv_user_mrs_attach (struct rv_user *rv); +int rv_drv_api_dereg_mem (struct mr_info *mr); + +int rv_drv_prepost_recv (struct rv_sconn *sconn); +void rv_recv_done (struct ib_cq *cq,struct ib_wc *wc); +void rv_report_cqe_error (struct ib_cq *cq,struct ib_wc *wc, +struct rv_sconn *sconn,const char *opname); +int doit_post_rdma_write (struct rv_user *rv,unsigned long arg); + +static inline struct rv_conn *user_conn_find (struct rv_user *rv,u64 handle) +{ +return xa_load (&rv->conn_xa, handle); +} + +void rv_conn_put (struct rv_conn *conn); +int rv_conn_get_check (struct rv_conn *conn); +void rv_conn_get (struct rv_conn *conn); +int doit_conn_create (struct rv_user *rv,unsigned long arg); +int doit_conn_connect (struct rv_user *rv,unsigned long arg); +int doit_conn_connected (struct rv_user *rv,unsigned long arg); +int doit_conn_get_conn_count (struct rv_user *rv,unsigned long arg); +int doit_conn_get_stats (struct rv_user *rv,unsigned long arg); +int cmp_gid (const void *gid1,const void *gid2); + +void rv_job_dev_get (struct rv_job_dev *jdev); +void rv_job_dev_put (struct rv_job_dev *jdev); + +static inline bool rv_job_dev_has_users (struct rv_job_dev *jdev) +{ +return kref_read (&jdev->user_kref); +} + +static inline bool rv_jdev_protocol_roce (const struct rv_job_dev *jdev) +{ +return rdma_protocol_roce (jdev->dev->ib_dev, jdev->port_num); +} + +struct rv_sconn * +rv_find_sconn_from_req (struct ib_cm_id *id, +const struct ib_cm_req_event_param *param, +struct rv_req_priv_data *priv_data); + +void rv_detach_user (struct rv_user *rv); + +#endif /* __RV_H__ */
drivers/infiniband/ulp/rv/rv_mr_cache.h new file +152 -0
152 lines not on this page · raw message
Signature · 1 line
2.18.1
Jason Gunthorpe
maintainerRe: [PATCH RFC 2/9] RDMA/rv: Add the internal header files
drivers/infiniband/ulp/rv/rv_mr_cache.h line 58
Quoted 4 lines · On Fri, Mar 19, 2021 at 08:56:28AM -0400, [email protected] wrote:
drivers/infiniband/ulp/rv/rv_mr_cache.h new file +4
-0+struct rv_mr_cache { +u64 max_size; +void *ops_arg; +struct mmu_notifier mn;
Nope on using raw mmu_notifier, this code isn't done right in hfi1, I'm not looking at it again here. Use the interval notifier.
Jason
RFC PATCH 3/9 RDMA/rv: Add the rv module
In reply to [email protected]From: Kaike Wan <[email protected]>
Add the rv module, the Makefile, and Kconfig file.
Also add the functions to manage IB devices.
--- MAINTAINERS | 6 + drivers/infiniband/Kconfig | 1 + drivers/infiniband/ulp/Makefile | 1 + drivers/infiniband/ulp/rv/Kconfig | 11 ++ drivers/infiniband/ulp/rv/Makefile | 9 + drivers/infiniband/ulp/rv/rv_main.c | 266 ++++++++++++++++++++++++++ drivers/infiniband/ulp/rv/trace.c | 7 + drivers/infiniband/ulp/rv/trace.h | 5 + drivers/infiniband/ulp/rv/trace_dev.h | 82 ++++++++ 9 files changed, 388 insertions(+) create mode 100644 drivers/infiniband/ulp/rv/Kconfig create mode 100644 drivers/infiniband/ulp/rv/Makefile create mode 100644 drivers/infiniband/ulp/rv/rv_main.c create mode 100644 drivers/infiniband/ulp/rv/trace.c create mode 100644 drivers/infiniband/ulp/rv/trace.h create mode 100644 drivers/infiniband/ulp/rv/trace_dev.h
MAINTAINERS +6 -0
@@ -15547,6 +15547,12 @@ L: [email protected] S: Maintained F: drivers/infiniband/ulp/rtrs/ +RV DRIVER +M: Kaike Wan <[email protected]> +L: [email protected] +S: Supported +F: drivers/infiniband/ulp/rv + RXRPC SOCKETS (AF_RXRPC) M: David Howells <[email protected]> L: [email protected]
drivers/infiniband/Kconfig +1 -0
@@ -107,5 +107,6 @@ source "drivers/infiniband/ulp/isert/Kconfig" source "drivers/infiniband/ulp/rtrs/Kconfig" source "drivers/infiniband/ulp/opa_vnic/Kconfig" +source "drivers/infiniband/ulp/rv/Kconfig" endif # INFINIBAND
drivers/infiniband/ulp/Makefile +1 -0
@@ -6,3 +6,4 @@ obj-$(CONFIG_INFINIBAND_ISER) += iser/ obj-$(CONFIG_INFINIBAND_ISERT) += isert/ obj-$(CONFIG_INFINIBAND_OPA_VNIC) += opa_vnic/ obj-$(CONFIG_INFINIBAND_RTRS) += rtrs/ +obj-$(CONFIG_INFINIBAND_RV) += rv/
drivers/infiniband/ulp/rv/Kconfig new file +11 -0
@@ -0,0 +1,11 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +# +# Copyright(c) 2020 - 2021 Intel Corporation. +# +config INFINIBAND_RV + tristate "InfiniBand Rendezvous Module" + depends on X86_64 && INFINIBAND + help + The rendezvous module provides mechanisms for HPC middlewares + to cache memory region registration, to manage connections + between nodes, and improve the scability of RDMA transactions.
drivers/infiniband/ulp/rv/Makefile new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +# +# Copyright(c) 2020 - 2021 Intel Corporation. +# +obj-$(CONFIG_INFINIBAND_RV) += rv.o + +rv-y := rv_main.o trace.o + +CFLAGS_trace.o = -I$(src)
drivers/infiniband/ulp/rv/rv_main.c new file +266 -0
266 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace.c new file +7 -0
@@ -0,0 +1,7 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +/* +* Copyright(c) 2020 - 2021 Intel Corporation. +*/ +#define CREATE_TRACE_POINTS +#include <rdma/rv_user_ioctls.h> +#include "trace.h"
drivers/infiniband/ulp/rv/trace.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +/* +* Copyright(c) 2020 - 2021 Intel Corporation. +*/ +#include "trace_dev.h"
drivers/infiniband/ulp/rv/trace_dev.h new file +82 -0
82 lines not on this page · raw message
Signature · 1 line
2.18.1
RFC PATCH 4/9 RDMA/rv: Add functions for memory region cache
In reply to [email protected]From: Kaike Wan <[email protected]>
The MR cache is implemented through an rb tree. Each node is indexed by a simple (address, length, access_flags) tuple, without any consideration of buffer overlapping. When a node's refcount goes down to 0, it is not removed from the cache. Instead, it is put into an LRU list that could be evicted if the cache memory limit is reached. However, if the user buffer for the memory region is freed, the node will be removed when the MMU notice is received.
--- drivers/infiniband/ulp/rv/Makefile | 2 +- drivers/infiniband/ulp/rv/rv_mr_cache.c | 428 +++++++++++++++++++++ drivers/infiniband/ulp/rv/trace.h | 1 + drivers/infiniband/ulp/rv/trace_mr_cache.h | 181 +++++++++ 4 files changed, 611 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.c create mode 100644 drivers/infiniband/ulp/rv/trace_mr_cache.h
drivers/infiniband/ulp/rv/Makefile +1 -1
@@ -4,6 +4,6 @@ # obj-$(CONFIG_INFINIBAND_RV) += rv.o -rv-y := rv_main.o trace.o +rv-y := rv_main.o rv_mr_cache.o trace.o CFLAGS_trace.o = -I$(src)
drivers/infiniband/ulp/rv/rv_mr_cache.c new file +428 -0
428 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace.h +1 -0
@@ -2,4 +2,5 @@/* * Copyright(c) 2020 - 2021 Intel Corporation. */ +#include "trace_mr_cache.h" #include "trace_dev.h"
drivers/infiniband/ulp/rv/trace_mr_cache.h new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
2.18.1
RFC PATCH 5/9 RDMA/rv: Add function to register/deregister memory region
In reply to [email protected]From: Kaike Wan <[email protected]>
This patch adds functions for user application to register/deregister memory region (mr) for user buffers. Two types of mrs are supported:
user mrs and kernel mrs.
User mrs are used soley by the user application. The reason that the user mrs are cached in the rv module instead of in the user application is that a middleware application may not known when a user buffer (allocated by a upper lay application) is freed in order to free any stale nodes. On the other hand, the rv module can register an MMU notifier callback so that it can promptly remove any stale cache nodes.
Kernel mrs are used by the rv module for any RDMA transactions between nodes.
A user mr is registered in a way similar to that in the verbs interface. A kernel mr is registered similar to that in ib_reg_user_mr() for on-demand paging. An RDMA hardware may have to be qualified for this mechanism.
--- drivers/infiniband/core/core_priv.h | 4 + drivers/infiniband/core/rdma_core.c | 237 ++++++++++++++ drivers/infiniband/core/uverbs_cmd.c | 54 +--- drivers/infiniband/ulp/rv/Makefile | 2 +- drivers/infiniband/ulp/rv/rv_file.c | 31 ++ drivers/infiniband/ulp/rv/rv_mr.c | 393 ++++++++++++++++++++++++ drivers/infiniband/ulp/rv/rv_mr_cache.c | 2 +- drivers/infiniband/ulp/rv/trace.h | 2 + drivers/infiniband/ulp/rv/trace_mr.h | 109 +++++++ drivers/infiniband/ulp/rv/trace_user.h | 66 ++++ include/rdma/uverbs_types.h | 10 + 11 files changed, 860 insertions(+), 50 deletions(-) create mode 100644 drivers/infiniband/ulp/rv/rv_file.c create mode 100644 drivers/infiniband/ulp/rv/rv_mr.c create mode 100644 drivers/infiniband/ulp/rv/trace_mr.h create mode 100644 drivers/infiniband/ulp/rv/trace_user.h
drivers/infiniband/core/core_priv.h +4 -0
@@ -404,4 +404,8 @@ void rdma_umap_priv_init(struct rdma_umap_priv *priv,void ib_cq_pool_cleanup (struct ib_device *dev); +struct ib_mr *uverbs_reg_mr (struct uverbs_attr_bundle *attrs,u32 pd_handle, +u64 start,u64 length,u64 hca_va, +u32 access_flags,struct ib_udata *driver_udata); +#endif /* _CORE_PRIV_H */
drivers/infiniband/core/rdma_core.c +237 -0
240 lines not on this page · raw message
drivers/infiniband/core/uverbs_cmd.c +6 -48
72 lines not on this page · raw message
drivers/infiniband/ulp/rv/Makefile +1 -1
@@ -4,6 +4,6 @@ # obj-$(CONFIG_INFINIBAND_RV) += rv.o -rv-y := rv_main.o rv_mr_cache.o trace.o +rv-y := rv_main.o rv_mr_cache.o rv_file.o rv_mr.o trace.o CFLAGS_trace.o = -I$(src)
drivers/infiniband/ulp/rv/rv_file.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +/* +* Copyright(c) 2020 - 2021 Intel Corporation. +*/ + +#include "rv.h" + +/* A workqueue for all */ +static struct workqueue_struct *rv_wq; + +void rv_queue_work (struct work_struct *work) +{ +queue_work (rv_wq, work); +} + +void rv_job_dev_get (struct rv_job_dev *jdev) +{ +kref_get (&jdev->kref); +} + +static void rv_job_dev_release (struct kref *kref) +{ + struct rv_job_dev *jdev = container_of(kref, struct rv_job_dev, kref); + +kfree_rcu (jdev, rcu); +} + +void rv_job_dev_put (struct rv_job_dev *jdev) +{ +kref_put (&jdev->kref, rv_job_dev_release); +}
drivers/infiniband/ulp/rv/rv_mr.c new file +393 -0
393 lines not on this page · raw message
drivers/infiniband/ulp/rv/rv_mr_cache.c +1 -1
@@ -374,7 +374,7 @@ static void do_remove(struct rv_mr_cache *cache, struct list_head *del_list)while (!list_empty (del_list)) { mrc = list_first_entry(del_list, struct rv_mr_cached, list);list_del (&mrc->list); -/* Deregister the mr here */ +rv_drv_api_dereg_mem (&mrc->mr);kfree (mrc); } }
drivers/infiniband/ulp/rv/trace.h +2 -0
@@ -4,3 +4,5 @@*/ #include "trace_mr_cache.h" #include "trace_dev.h" +#include "trace_mr.h" +#include "trace_user.h"
drivers/infiniband/ulp/rv/trace_mr.h new file +109 -0
109 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace_user.h new file +66 -0
66 lines not on this page · raw message
include/rdma/uverbs_types.h +10 -0
@@ -134,6 +134,16 @@ static inline void uverbs_uobject_get(struct ib_uobject *uobject) }void uverbs_uobject_put (struct ib_uobject *uobject); +struct ib_mr *rdma_reg_user_mr (struct ib_device *ib_dev,u32 cmd_fd, +u32 pd_handle,u64 start,u64 length, +u32 access_flags,size_t ulen,void *udata, +struct fd *fd); +int rdma_dereg_user_mr (struct ib_mr *mr,struct fd *fd); +struct ib_mr *rdma_reg_kernel_mr (u32 cmd_fd,struct ib_pd *kern_pd,u64 start, +u64 length,u32 access_flags,size_t ulen, +void *udata,struct fd *fd); +int rdma_dereg_kernel_mr (struct ib_mr *mr,struct fd *fd); +struct uverbs_obj_fd_type {/* * In fd based objects, uverbs_obj_type_ops points to generic
Signature · 1 line
2.18.1
RFC PATCH 6/9 RDMA/rv: Add connection management functions
In reply to [email protected]From: Kaike Wan <[email protected]>
To improve scalibity of RDMA transaction, there will be only one connection between any two nodes. Within each node pair, one node will be the client and the other node will be the server, depending the lids/gids of the two nodes. However, to make best use of the bandwidth, each connection could have multiple RC QPs to share among all of the processes within a job. Connection is established through the IB CM interface.
This patch adds the following functions:
- Listerner functions to wait for any IB CM requests from the same job.
- Client functions to send IB CM requests.
- Functions to manage the lifetime of the connection object.
- Functions to manage RC QPs.
- IB CM event handlers for client and server.
--- drivers/infiniband/ulp/rv/Makefile | 2 +- drivers/infiniband/ulp/rv/rv_conn.c | 3037 ++++++++++++++++++++++++ drivers/infiniband/ulp/rv/rv_file.c | 125 + drivers/infiniband/ulp/rv/rv_rdma.c | 103 + drivers/infiniband/ulp/rv/trace.h | 2 + drivers/infiniband/ulp/rv/trace_conn.h | 529 +++++ drivers/infiniband/ulp/rv/trace_rdma.h | 129 + drivers/infiniband/ulp/rv/trace_user.h | 57 +- 8 files changed, 3982 insertions(+), 2 deletions(-) create mode 100644 drivers/infiniband/ulp/rv/rv_conn.c create mode 100644 drivers/infiniband/ulp/rv/rv_rdma.c create mode 100644 drivers/infiniband/ulp/rv/trace_conn.h create mode 100644 drivers/infiniband/ulp/rv/trace_rdma.h
drivers/infiniband/ulp/rv/Makefile +1 -1
@@ -4,6 +4,6 @@ # obj-$(CONFIG_INFINIBAND_RV) += rv.o -rv-y := rv_main.o rv_mr_cache.o rv_file.o rv_mr.o trace.o +rv-y := rv_main.o rv_mr_cache.o rv_file.o rv_mr.o trace.o rv_rdma.o rv_conn.o CFLAGS_trace.o = -I$(src)
drivers/infiniband/ulp/rv/rv_conn.c new file +3037 -0
2,993 lines not on this page · raw message
drivers/infiniband/ulp/rv/rv_file.c +125 -0
125 lines not on this page · raw message
drivers/infiniband/ulp/rv/rv_rdma.c new file +103 -0
103 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace_conn.h new file +529 -0
529 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace_rdma.h new file +129 -0
129 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace_user.h +56 -1
57 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.18.1
Part of the diff is not shown here. View the raw message.
RFC PATCH 7/9 RDMA/rv: Add functions for RDMA transactions
In reply to [email protected]From: Kaike Wan <[email protected]>
The only RDMA request used by this module is the RDMA WRITE WITH IMMEDIATE request. Part of the immediate data is used as a tag to encode the intended receiving rv_user in the rv_conn object, and remaining bits are reserved for the user application (eg. to associate the inbound completion with a specific outstanding rendezvous IO).
This patch adds the following functions:
- Send RDMA write with immediate request.
- Handle the send completion event.
- Receive the RDMA write with immediate request.
- Post events to the event ring.
--- drivers/infiniband/ulp/rv/rv_rdma.c | 313 +++++++++++++++++++++++++ drivers/infiniband/ulp/rv/trace_rdma.h | 125 ++++++++++ drivers/infiniband/ulp/rv/trace_user.h | 31 +++ 3 files changed, 469 insertions(+)
drivers/infiniband/ulp/rv/rv_rdma.c +313 -0
@@ -6,6 +6,56 @@#include "rv.h" #include "trace.h" +/* +* select next sconn to post and claim WQE by inc outstand_send_write +* if all sconn SQs are full, next is left back where it started +*/ +static struct rv_sconn *rv_conn_next_sconn_to_post (struct rv_conn *conn) +{ +unsigned long flags; +struct rv_sconn *sconn; +u8 i; +u32 qp_depth = conn->jdev->qp_depth; + +spin_lock_irqsave (&conn->next_lock, flags); +for (i =0 ; i < conn->num_conn; i++) { + sconn = &conn->sconn_arr[conn->next]; + conn->next = (conn->next +1 ) % conn->num_conn; +if (atomic_read (&sconn->stats.outstand_send_write) < qp_depth) { +atomic_inc (&sconn->stats.outstand_send_write); + goto unlock; + } + } + sconn =NULL ; +unlock: +spin_unlock_irqrestore (&conn->next_lock, flags); +return sconn; +} + +static int rv_drv_post_write_immed (struct rv_pend_write *pend_wr) +{ +struct ib_rdma_wr wr; +const struct ib_send_wr *bad_wr; +struct ib_sge list; +struct rv_mr_cached *mrc = pend_wr->mrc; + +/* we xlat the user space loc_addr to an iova appropriate for the MR */ + list.addr = mrc->mr.ib_mr->iova + (pend_wr->loc_addr - mrc->addr); + list.length = pend_wr->length; + list.lkey = mrc->mr.ib_mr->lkey; + + wr.wr.next =NULL ; + wr.wr.wr_cqe = &pend_wr->cqe; + wr.wr.sg_list = &list; + wr.wr.num_sge =1 ; + wr.wr.opcode =IB_WR_RDMA_WRITE_WITH_IMM ; + wr.wr.send_flags =IB_SEND_SIGNALED ; + wr.wr.ex.imm_data =cpu_to_be32 (pend_wr->immed); + wr.remote_addr = pend_wr->rem_addr; + wr.rkey = pend_wr->rkey; +return ib_post_send (pend_wr->sconn->qp, &wr.wr, &bad_wr); +} +/* * This is called in Soft IRQs for CQE handling. * We just report errors here, let the QP Async Event deal with
@@ -22,6 +72,246 @@ void rv_report_cqe_error(struct ib_cq *cq, struct ib_wc *wc, wc->wr_cqe); } +static void rv_user_ring_post_event (struct rv_user_ring *ring, +struct rv_event *ev) +{ +unsigned long flags; +struct rv_ring_header *hdr = ring->hdr; +int next; + +trace_rv_user_ring_post_event (ring->rv_inx, ring->num_entries, + ring->hdr->head, ring->hdr->tail); +trace_rv_event_post (ev->event_type, ev->wc.status, ev->wc.imm_data, + ev->wc.wr_id, ev->wc.conn_handle, + ev->wc.byte_len); +spin_lock_irqsave (&ring->lock, flags); + next = hdr->tail +1 ; +if (next == ring->num_entries) + next =0 ; +if (next == hdr->head) { + hdr->overflow_cnt++; +rv_err (ring->rv_inx,"event ring full: head %u tail %u\n" , + hdr->head, hdr->tail); + goto unlock; + } + +smp_rmb ();/* ensure we read tail before writing event */ + hdr->entries[hdr->tail] = *ev; +smp_wmb ();/* ensure ev written before advance tail */ + + hdr->tail = next; +if (ev->wc.status) { + ring->stats.cqe_fail[ev->event_type]++; + }else { + ring->stats.cqe[ev->event_type]++; + ring->stats.bytes[ev->event_type] += ev->wc.byte_len; + } +unlock: +spin_unlock_irqrestore (&ring->lock, flags); +} + +static void rv_post_user_event_by_index (struct rv_job_dev *jdev,u16 index, +struct rv_event *ev) +{ +unsigned long flags; +struct rv_user *rv; + +spin_lock_irqsave (&jdev->user_array_lock, flags); +if (index >= jdev->max_users) + goto unlock; + rv = jdev->user_array[index]; +if (rv && rv->cqr) +rv_user_ring_post_event (rv->cqr, ev); +unlock: +spin_unlock_irqrestore (&jdev->user_array_lock, flags); +} + +/* +* We have a rv_conn reference for the pend_wr +* pass all failures to PSM to deal with. We can't attempt +* to retry the write in rv since it might have succeeded on remote +* end (eg. ack lost) and remote end may be using buffer for something +* else already +*/ +static void rv_rdma_write_done (struct ib_cq *cq,struct ib_wc *wc) +{ +struct rv_pend_write *pend_wr =container_of (wc->wr_cqe, + struct rv_pend_write, cqe); +struct rv_sconn *sconn = pend_wr->sconn; +struct rv_event ev = {0 }; + +atomic_dec (&sconn->stats.outstand_send_write); +trace_rv_wc_write_done (pend_wr->wr_id, wc->status, wc->opcode, + wc->byte_len,be32_to_cpu (wc->ex.imm_data)); +trace_rv_pend_write_done (pend_wr->user_index, pend_wr->sconn, pend_wr, + pend_wr->loc_addr, pend_wr->rkey, + pend_wr->rem_addr, pend_wr->length, + pend_wr->immed, pend_wr->wr_id); + +if (unlikely (wc->status !=IB_WC_SUCCESS )) +rv_report_cqe_error (cq, wc, pend_wr->sconn,"RDMA Write" ); +else if (wc->qp != sconn->qp) +rv_report_cqe_error (cq, wc, pend_wr->sconn,"Stale RDMA Write" ); + + ev.event_type =RV_WC_RDMA_WRITE ; + ev.wc.status = wc->status; + ev.wc.wr_id = pend_wr->wr_id; + ev.wc.conn_handle = (u64 )pend_wr->sconn->parent; + ev.wc.byte_len = pend_wr->length; +trace_rv_event_write_done (ev.event_type, ev.wc.status, ev.wc.imm_data, + ev.wc.wr_id, ev.wc.conn_handle, + ev.wc.byte_len); + +rv_mr_cache_put (&pend_wr->umrs->cache, pend_wr->mrc); + +rv_post_user_event_by_index (pend_wr->sconn->parent->jdev, + pend_wr->user_index, &ev); + +if (wc->status) +atomic64_inc (&sconn->stats.send_write_cqe_fail); +else +atomic64_inc (&sconn->stats.send_write_cqe); + +/* our rv_conn ref prevents user_mrs_put from triggering job cleanup */ +rv_user_mrs_put (pend_wr->umrs); + +/* rv_conn_put can put rv_job_dev and trigger whole job cleanup */ +rv_conn_put (sconn->parent); + +kfree (pend_wr); +} + +/* +* we do not need a queue inside rv of unposted writes. If this fails +* PSM will try to repost later. +* We use loc_addr/length/access to lookup MR in cache and then verify RDMA is +* consistent with loc_addr and length +*/ +int doit_post_rdma_write (struct rv_user *rv,unsigned long arg) +{ +struct rv_post_write_params pparams; +struct rv_conn *conn; +struct rv_sconn *sconn; +struct rv_mr_cached *mrc; +struct rv_pend_write *pend_wr; +int ret; + +if (copy_from_user (&pparams.in, (void __user *)arg, +sizeof (pparams.in))) +return -EFAULT ; + +mutex_lock (&rv->mutex); + + conn =user_conn_find (rv, pparams.in.handle); +if (!conn) { +rv_err (rv->inx,"post_write: No connection found\n" ); + ret = -EINVAL ; + goto bail_unlock; + } + sconn =rv_conn_next_sconn_to_post (conn); +if (unlikely (!sconn)) { + ret = -ENOMEM ; + goto bail_unlock; + } + + mrc =rv_mr_cache_search_get (&rv->umrs->cache, pparams.in.loc_mr_addr, + pparams.in.loc_mr_length, + pparams.in.loc_mr_access, false); +if (!mrc) { +rv_err (rv->inx,"post_write: bad loc_mr\n" ); + ret = -EINVAL ; + goto bail_dec; + } + +if (mrc->addr > (u64 )pparams.in.loc_addr || + mrc->addr + mrc->len < + (u64 )pparams.in.loc_addr + pparams.in.length) { +rv_err (rv->inx,"post_write: addr inconsistent with loc_mr\n" ); + ret = -EINVAL ; + goto bail_put_mr; + } +if (!(mrc->access &IBV_ACCESS_KERNEL )) { +rv_err (rv->inx,"post_write: loc_mr not a kernel MR\n" ); + ret = -EINVAL ; + goto bail_put_mr; + } + + pend_wr =kzalloc (sizeof (*pend_wr),GFP_KERNEL ); +if (!pend_wr) { + ret = -ENOMEM ; + goto bail_put_mr; + } + pend_wr->cqe.done = rv_rdma_write_done; + pend_wr->user_index = rv->index; + +rv_user_mrs_get (rv->umrs); + pend_wr->umrs = rv->umrs; + +rv_conn_get (sconn->parent); + pend_wr->sconn = sconn; + + pend_wr->mrc = mrc; + pend_wr->loc_addr = (u64 )pparams.in.loc_addr; + pend_wr->rem_addr = pparams.in.rem_addr; + pend_wr->rkey = pparams.in.rkey; + pend_wr->length = pparams.in.length; + pend_wr->immed = pparams.in.immed; + pend_wr->wr_id = pparams.in.wr_id; + +mutex_lock (&sconn->mutex); +if (sconn->state !=RV_CONNECTED ) { +if (sconn->state ==RV_ERROR ) + ret = -EIO ; +else if (test_bit (RV_SCONN_WAS_CONNECTED , &sconn->flags)) + ret = -EAGAIN ; +else + ret = -EINVAL ; +mutex_unlock (&sconn->mutex); + goto bail_free_pend; + } + +trace_rv_pend_write_post (pend_wr->user_index, pend_wr->sconn, pend_wr, + pend_wr->loc_addr, pend_wr->rkey, + pend_wr->rem_addr, pend_wr->length, + pend_wr->immed, pend_wr->wr_id); + ret =rv_drv_post_write_immed (pend_wr); +if (ret) { + sconn->stats.post_write_fail++; + }else { + sconn->stats.post_write++; + sconn->stats.post_write_bytes += pparams.in.length; + } + + pparams.out.sconn_index = sconn->index; + pparams.out.conn_count = sconn->stats.conn_recovery +1 ; + +mutex_unlock (&sconn->mutex); +if (ret) { +rv_err (rv->inx,"post_write: failed: %d\n" , ret); + goto bail_free_pend; + } + +if (copy_to_user ((void __user *)arg, &pparams.out,sizeof (pparams.out))) + ret = -EFAULT ; + +mutex_unlock (&rv->mutex); + +return 0 ; + +bail_free_pend: +rv_conn_put (pend_wr->sconn->parent); +rv_user_mrs_put (pend_wr->umrs); +kfree (pend_wr); + +bail_put_mr: +rv_mr_cache_put (&rv->umrs->cache, mrc); +bail_dec: +atomic_dec (&sconn->stats.outstand_send_write); +bail_unlock: +mutex_unlock (&rv->mutex); +return ret; +} +static int rv_drv_post_recv (struct rv_sconn *sconn) {struct ib_recv_wr wr;
@@ -54,6 +344,27 @@ int rv_drv_prepost_recv(struct rv_sconn *sconn)return 0 ; } +static void rv_recv_rdma_write (struct rv_sconn *sconn,struct ib_wc *wc) +{ +struct rv_job_dev *jdev = sconn->parent->jdev; +u32 index =be32_to_cpu (wc->ex.imm_data) >> (32 - jdev->index_bits); +struct rv_event ev = {0 }; + + ev.event_type =RV_WC_RECV_RDMA_WITH_IMM ; + ev.wc.status = wc->status; + ev.wc.resv1 =0 ; + ev.wc.imm_data =be32_to_cpu (wc->ex.imm_data); + ev.wc.wr_id =0 ;/* N/A */ + ev.wc.conn_handle = (u64 )sconn->parent; + ev.wc.byte_len = wc->byte_len; + ev.wc.resv2 =0 ; +trace_rv_event_recv_write (ev.event_type, ev.wc.status, ev.wc.imm_data, + ev.wc.wr_id, ev.wc.conn_handle, + ev.wc.byte_len); + +rv_post_user_event_by_index (jdev, index, &ev); +} +/* drain_lock makes sure no recv WQEs get reposted after a drain WQE */ void rv_recv_done (struct ib_cq *cq,struct ib_wc *wc) {
@@ -93,6 +404,8 @@ void rv_recv_done(struct ib_cq *cq, struct ib_wc *wc)if (unlikely (wc->opcode !=IB_WC_RECV_RDMA_WITH_IMM ))rv_report_cqe_error (cq, wc, sconn,"Recv bad opcode" ); +else +rv_recv_rdma_write (sconn, wc); repost:spin_lock_irqsave (&sconn->drain_lock, flags);if (likely (!test_bit (RV_SCONN_DRAINING , &sconn->flags)))
drivers/infiniband/ulp/rv/trace_rdma.h +125 -0
146 lines not on this page · raw message
drivers/infiniband/ulp/rv/trace_user.h +31 -0
@@ -112,6 +112,37 @@ DEFINE_EVENT(/* event */TP_ARGS (rv_inx, jdev, total_size, max_size, refcount) ); +DECLARE_EVENT_CLASS (/* user_ring */ + rv_user_ring_template, + TP_PROTO(int rv_inx, u32 count, u32 hd, u32 tail), +TP_ARGS (rv_inx, count, hd, tail), +TP_STRUCT__entry (/* entry */ + __field(int, rv_inx) + __field(u32, count) + __field(u32, head) +__field (u32, tail) + ), +TP_fast_assign (/* assign */ + __entry->rv_inx = rv_inx; + __entry->count = count; + __entry->head = hd; + __entry->tail = tail; + ), +TP_printk (/* print */ +"rv_inx %d entries %u head %u tail %u" , + __entry->rv_inx, + __entry->count, + __entry->head, + __entry->tail + ) +); + +DEFINE_EVENT (/* event */ + rv_user_ring_template, rv_user_ring_post_event, + TP_PROTO(int rv_inx, u32 count, u32 hd, u32 tail), +TP_ARGS (rv_inx, count, hd, tail) +); +#endif /* __RV_TRACE_USER_H */ #undef TRACE_INCLUDE_PATH
Signature · 1 line
2.18.1
RFC PATCH 8/9 RDMA/rv: Add functions for file operations
In reply to [email protected]From: Kaike Wan <[email protected]>
A process communicates with the rv module through the file interface:
- The process opens the /dev/rv device file.
- The process sends an Attach request to bind to an RDMA device.
- The process registers a number of user/kernel memory regions (MR).
- The process sends Create Conn request to create connection between two nodes. If the local node is the server, it will start to listen to IB CM for any incoming connection request.
- The process sends Connect request to start connection. For a server node, it does nothing. However, for a client node, it will send the IB CM connection request.
- The process will wait for all connections to be established by polling the rv module. Receiving buffers will be posted.
- The process mmaps the event ring into user space.
- The process starts RDMA transaction by sending RDMA write with immediate requests to rv module. Send completion events will be posted to the event ring.
- On the responder side, RDMA write with immediate requests will be received and receive completion events will be posted to the event ring buffer.
- The process will poll the event ring for completion events.
- When RDMA transactions are done, the process deregisters the memory regions.
- The process closes the file. In this process, any connection will be torn down, and the RDMA device will be detached if there is no more user. Explicit detach and disconnection are not required.
Technically, the MR registration, RDMA transactions, and MR deregistration occurs for every application IO.
This patch adds the functions for the file operations and integrates the functions with memory region registration, connection management, and RDMA transactions.
--- drivers/infiniband/ulp/rv/rv_file.c | 1040 ++++++++++++++++++++++++ drivers/infiniband/ulp/rv/rv_main.c | 2 +- drivers/infiniband/ulp/rv/trace_conn.h | 18 + drivers/infiniband/ulp/rv/trace_user.h | 169 ++++ 4 files changed, 1228 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rv/rv_file.c +1040 -0
@@ -9,11 +9,58 @@#include "rv.h" #include "trace.h" +static unsigned long service_id =RV_DFLT_SERVICE_ID ; + +module_param (service_id, ulong,0444 ); +MODULE_PARM_DESC (service_id,"Default service_id for IB CM QP connections" ); + +static unsigned int num_conn =4 ; + +module_param (num_conn, uint,0444 ); +MODULE_PARM_DESC (num_conn,"Default # QPs between each pair of nodes" ); + +/* +* these are per node to node connection. +* +* conservative for now, we should service CQEs fast enough that smaller numbers +* would work, however at 100 remote nodes (hence 100 connections), 400,000 +* CQEs at say 64B each, is only 50MB including send and recv +* +* A given PSM receiver process +* will not allow any more than HFI_TF_NFLOWS (32) inflight RDMA's coming +* toward it. So if we have ~100 processes per node, that limits inflight to +* 3200 coming toward us. +* while a given sender can have many in flight, the total a given destination +* process will allow is HFI_TF_NFLOWS (32), so size this same as recv CQ +* same reasoning applies to QP sizes +*/ +#define RV_Q_DEPTH 4000/* some headroom */ + +static unsigned int q_depth =RV_Q_DEPTH ; + +module_param (q_depth, uint,0444 ); +MODULE_PARM_DESC (q_depth,"Default size of queues per remote node" ); + +static int rv_file_mmap (struct file *fp,struct vm_area_struct *vma); +static void rv_user_ring_free (struct rv_user_ring *ring); +static void rv_user_detach_all (struct rv_user *rv); + +static atomic_t seq;/* A workqueue for all */ static struct workqueue_struct *rv_wq;static struct workqueue_struct *rv_wq2;static struct workqueue_struct *rv_wq3; +/* Device file access */ +struct rv_devdata { +struct class *class; +dev_t dev; +struct cdev user_cdev; +struct device user_device; +}; + +static struct rv_devdata *rv_dd; +/* * We expect relatively few jobs per node (typically 1) * and relatively few devices per node (typically 1 to 8)
@@ -24,7 +71,9 @@ static struct workqueue_struct *rv_wq3;* mutex avoids duplicate get_alloc adds, RCU protects list access. * See rv.h comments about "get_alloc" for more information. */ +static struct mutex rv_job_dev_list_mutex;static struct list_head rv_job_dev_list; +static atomic_t rv_job_dev_cnt;void rv_queue_work (struct work_struct *work) {
@@ -46,16 +95,268 @@ void rv_flush_work2(void)flush_workqueue (rv_wq2); } +static int doit_query (struct rv_user *rv,unsigned long arg) +{ +struct rv_query_params_out params = {0 }; +int ret =0 ; + + params.major_rev =RV_ABI_VER_MAJOR ; + params.minor_rev =RV_ABI_VER_MINOR ; + params.capability =0 ; + +if (copy_to_user ((void __user *)arg, ¶ms,sizeof (params))) + ret = -EFAULT ; + +return ret; +} + +/* +* only for use by rv_job_dev_get_alloc, +* all other callers must use rv_job_dev_get_alloc or rv_job_dev_get +* user_array is sized at end of rv_job_dev +* When num_conn>1 we will stripe across all sconn, so +* any given sconn's qp_depth can be smaller. However, this striping can be +* imperfect since not every IO is the same size. We allocate an +* extra 5% to minimize skipping sconns on send scheduling +*/ +static struct rv_job_dev * +rv_job_dev_alloc (int rv_inx,struct rv_device *dev, +const struct rv_attach_params *params) +{ +int ret; +struct rv_job_dev *jdev; +int max_users = (1 << params->in.index_bits); + + jdev =kzalloc (sizeof (*jdev) +sizeof (jdev->user_array[0 ]) * max_users, +GFP_KERNEL ); +if (!jdev) { + ret = -ENOMEM ; + goto bail; + } + + jdev->kuid =current_uid (); + jdev->uid =from_kuid (current_user_ns (), jdev->kuid); + jdev->max_users = max_users; +spin_lock_init (&jdev->user_array_lock); +kref_init (&jdev->kref); +kref_init (&jdev->user_kref); +mutex_init (&jdev->conn_list_mutex); +INIT_LIST_HEAD (&jdev->conn_list); + jdev->dev = dev; + jdev->pd =ib_alloc_pd (jdev->dev->ib_dev,0 ); +if (IS_ERR (jdev->pd)) { +rv_err (rv_inx,"Could not allocate PD\n" ); + ret =PTR_ERR (jdev->pd); + goto bail_free; + } + +memcpy (jdev->dev_name, params->in.dev_name,sizeof (jdev->dev_name)); + jdev->port_num = params->in.port_num; + jdev->num_conn = params->in.num_conn; + jdev->index_bits = params->in.index_bits; + jdev->loc_gid_index = params->in.loc_gid_index; + jdev->loc_addr = params->in.loc_addr; +memcpy (jdev->loc_gid, params->in.loc_gid,sizeof (jdev->loc_gid)); +memcpy (jdev->job_key, params->in.job_key,sizeof (jdev->job_key)); + jdev->job_key_len = params->in.job_key_len; + jdev->service_id = params->in.service_id; + jdev->q_depth = params->in.q_depth; + jdev->qp_depth = (jdev->q_depth + jdev->num_conn -1 ) / jdev->num_conn; +if (jdev->num_conn >1 ) + jdev->qp_depth += jdev->qp_depth /20 ; + jdev->reconnect_timeout = params->in.reconnect_timeout; + jdev->hb_interval = params->in.hb_interval; + jdev->sgid_attr =rdma_get_gid_attr (jdev->dev->ib_dev, jdev->port_num, + jdev->loc_gid_index); +if (!jdev->sgid_attr) { +rv_err (rv_inx,"can't resolve sgid_attr\n" ); + ret = -ENOENT ; + goto bail_mr; + } +if (memcmp (&jdev->loc_gid, &jdev->sgid_attr->gid, +sizeof (jdev->loc_gid))) { +rv_err (rv_inx,"sgid_attr gid and loc_gid mismatch\n" ); + ret = -ENOENT ; + goto bail_put; + } +trace_rv_jdev_alloc (jdev, jdev->dev_name, jdev->num_conn, + jdev->index_bits, jdev->loc_gid_index, + jdev->loc_addr, jdev->job_key_len, + jdev->job_key, jdev->service_id, + jdev->q_depth, jdev->user_array_next, +kref_read (&jdev->kref)); +return jdev; + +bail_put: +rdma_put_gid_attr (jdev->sgid_attr); +bail_mr: +ib_dealloc_pd (jdev->pd); +bail_free: +kfree (jdev); +bail: +return ERR_PTR (ret); +} + +static int rv_job_dev_consistent (struct rv_job_dev *jdev, +const struct rv_attach_params *params) +{ +return (params->in.num_conn == jdev->num_conn && + params->in.index_bits == jdev->index_bits && + params->in.loc_gid_index == jdev->loc_gid_index && + params->in.loc_addr == jdev->loc_addr && + !memcmp (jdev->loc_gid, params->in.loc_gid, +sizeof (jdev->loc_gid)) && + params->in.service_id == jdev->service_id && + params->in.q_depth == jdev->q_depth && + params->in.reconnect_timeout == jdev->reconnect_timeout && + params->in.hb_interval == jdev->hb_interval); +} + +static void rv_job_dev_list_init (void ) +{ +mutex_init (&rv_job_dev_list_mutex); +INIT_LIST_HEAD (&rv_job_dev_list); +atomic_set (&rv_job_dev_cnt,0 ); +} + +static void rv_job_dev_user_release (struct kref *kref) +{ + struct rv_job_dev *jdev = container_of(kref, struct rv_job_dev, + user_kref); + +mutex_lock (&rv_job_dev_list_mutex); +list_del_rcu (&jdev->job_dev_entry); +mutex_unlock (&rv_job_dev_list_mutex); +} + +static void rv_job_dev_put_user (struct rv_job_dev *jdev) +{ +kref_put (&jdev->user_kref, rv_job_dev_user_release); +} + +/* +* get a job_dev matching the given ATTACH. If none is found, create one +* The job_dev returned must be released with rv_job_dev_put when done using. +* Get device 1st to reduce lock nesting. Device search should be quick. +* While searching for jdev, likely to have more devs than jobs, so filter on +* dev 1st. job_key_len can be 0, which matches only jobs with job_key_len==0 +* Ideally each job should have a unique job_key (really just a job identifer), +* but all jobs or processes with the same job_key must have the same params. +*/ +static struct rv_job_dev *rv_job_dev_get_alloc (struct rv_user *rv, +struct rv_attach_params *params) +{ +struct rv_job_dev *jdev; +struct rv_device *dev; + + dev =rv_device_get_add_user (params->in.dev_name, rv); +if (!dev) { +rv_err (rv->inx,"attach: KERNEL ib_dev %s not found\n" , + params->in.dev_name); +return ERR_PTR (-ENODEV ); + } + +mutex_lock (&rv_job_dev_list_mutex); +rcu_read_lock (); +list_for_each_entry_rcu (jdev, &rv_job_dev_list, job_dev_entry) { +if (!uid_eq (jdev->kuid,current_uid ()) || + dev != jdev->dev || + params->in.port_num != jdev->port_num || + params->in.job_key_len != jdev->job_key_len || + (params->in.job_key_len && +memcmp (params->in.job_key, jdev->job_key, + jdev->job_key_len))) +continue ; +if (!kref_get_unless_zero (&jdev->kref)) +continue ; +if (!kref_get_unless_zero (&jdev->user_kref)) { +rv_job_dev_put (jdev); +continue ; + } +rcu_read_unlock (); +if (!rv_job_dev_consistent (jdev, params)) { +mutex_unlock (&rv_job_dev_list_mutex); +rv_job_dev_put_user (jdev); +rv_job_dev_put (jdev); + jdev =ERR_PTR (-EBUSY ); + goto bail_put; + } +mutex_unlock (&rv_job_dev_list_mutex); +rv_device_put (dev); +return jdev; + } +rcu_read_unlock (); + jdev =rv_job_dev_alloc (rv->inx, dev, params); +if (IS_ERR (jdev)) + goto bail_unlock; + +list_add_tail_rcu (&jdev->job_dev_entry, &rv_job_dev_list); +atomic_inc (&rv_job_dev_cnt); + +mutex_unlock (&rv_job_dev_list_mutex); +return jdev; + +bail_unlock: +mutex_unlock (&rv_job_dev_list_mutex); +bail_put: +rv_device_del_user (rv); +rv_device_put (dev); +return jdev; +} +void rv_job_dev_get (struct rv_job_dev *jdev) {kref_get (&jdev->kref); } +struct rv_dest_pd_work_item { +struct work_struct destroy_work; +struct ib_pd *pd; +struct rv_device *dev; +}; + +static void rv_handle_destroy_pd (struct work_struct *work) +{ +struct rv_dest_pd_work_item *item =container_of (work, + struct rv_dest_pd_work_item, destroy_work); + +flush_workqueue (rv_wq2); +ib_dealloc_pd (item->pd); +rv_device_put (item->dev); +kfree (item); +} +static void rv_job_dev_release (struct kref *kref) { struct rv_job_dev *jdev = container_of(kref, struct rv_job_dev, kref); +struct rv_dest_pd_work_item *item; + +trace_rv_jdev_release (jdev, jdev->dev_name, jdev->num_conn, + jdev->index_bits, jdev->loc_gid_index, + jdev->loc_addr, jdev->job_key_len, + jdev->job_key, jdev->service_id, + jdev->q_depth, jdev->user_array_next, +kref_read (&jdev->kref)); +WARN_ON (!list_empty (&jdev->conn_list));/* RCU safe */ + +if (jdev->listener) +rv_listener_put (jdev->listener); +rdma_put_gid_attr (jdev->sgid_attr); + item =kzalloc (sizeof (*item),GFP_KERNEL ); +if (item) { +INIT_WORK (&item->destroy_work, rv_handle_destroy_pd); + item->pd = jdev->pd; + item->dev = jdev->dev; + jdev->pd =NULL ; + jdev->dev =NULL ; +rv_queue_work3 (&item->destroy_work); + }else { +ib_dealloc_pd (jdev->pd); +rv_device_put (jdev->dev); + }kfree_rcu (jdev, rcu); +atomic_dec (&rv_job_dev_cnt); }void rv_job_dev_put (struct rv_job_dev *jdev)
@@ -63,6 +364,316 @@ void rv_job_dev_put(struct rv_job_dev *jdev)kref_put (&jdev->kref, rv_job_dev_release); } +/* +* make a bi-directional linkage between rv_user and rv_job_dev +* Each rv_user is assigned a unique index within it's job_dev. +* This will be placed in RDMA immediate data on +* remote node so in recv CQE we can figure out which rv to deliver +* the RDMA w/immediate recv CQE event to +*/ +static int rv_job_dev_add_user (struct rv_job_dev *jdev,struct rv_user *rv) +{ +unsigned long flags; +int i; +struct rv_user **jentry; +u32 next; + +spin_lock_irqsave (&jdev->user_array_lock, flags); + next = jdev->user_array_next; + jentry = &jdev->user_array[next]; +for (i =0 ; i < jdev->max_users && *jentry; i++) { +if (++next >= jdev->max_users) { + next =0 ; + jentry = &jdev->user_array[0 ]; + }else { + jentry++; + } + } +if (i >= jdev->max_users) { + i = -ENOMEM ; + goto unlock; + } + i = next; +if (++next >= jdev->max_users) + next =0 ; + jdev->user_array_next = next; + *jentry = rv; + rv->index = i; + +unlock: +spin_unlock_irqrestore (&jdev->user_array_lock, flags); +return i; +} + +/* break the bi-directional linkage between rv_user and rv_job_dev */ +static void rv_job_dev_del_user (struct rv_job_dev *jdev,struct rv_user *rv) +{ +unsigned long flags; + +spin_lock_irqsave (&jdev->user_array_lock, flags); +WARN_ON (rv->index >= jdev->max_users); +WARN_ON (jdev->user_array[rv->index] != rv); + jdev->user_array[rv->index] =NULL ; + rv->index =RV_INVALID ; +spin_unlock_irqrestore (&jdev->user_array_lock, flags); +} + +/* attach for rdma_mode KERNEL */ +static int rv_user_attach_kernel (struct rv_user *rv, +struct rv_attach_params *params) +{ +struct rv_job_dev *jdev; +int ret; + + jdev =rv_job_dev_get_alloc (rv, params); +if (IS_ERR (jdev)) +return PTR_ERR (jdev); + + rv->jdev = jdev; + rv->context = params->in.context; + rv->cq_entries = params->in.cq_entries; + + ret =rv_job_dev_add_user (jdev, rv); +if (ret <0 ) { + rv->jdev =NULL ; + rv->context =0 ; + rv->cq_entries =0 ; +rv_job_dev_put_user (jdev); +rv_job_dev_put (jdev); +return ret; + } +return 0 ; +} + +static void rv_user_detach_kernel (struct rv_user *rv) +{ +if (!rv->jdev) +return ; +rv_job_dev_del_user (rv->jdev, rv); +rv_job_dev_put (rv->jdev); +} + +/* +* Attach an rv_user to a jdev +* Note hb_interval must be less than reconnect_timeout otherwise listener +* could timeout before client side discovers it must reconnect +* To avoid deadlock rv_user_mrs_alloc must be called without rv->mutex +* because it will acquire mm->mmap_lock. +*/ +static int doit_attach (struct rv_user *rv,unsigned long arg) +{ +struct rv_attach_params params; +int ret; +struct rv_user_mrs *umrs; +u32 reconnect_timeout =0 ; +u32 depth_out =0 ; + +if (copy_from_user (¶ms.in, (void __user *)arg,sizeof (params.in))) +return -EFAULT ; + +trace_rv_attach_req (rv->inx, params.in.dev_name, params.in.rdma_mode, + params.in.port_num, params.in.loc_addr, + params.in.job_key_len, params.in.job_key, + params.in.service_id, + params.in.cq_entries, params.in.q_depth, + params.in.reconnect_timeout, + params.in.hb_interval); +if (!params.in.dev_name[0 ] || +strnlen (params.in.dev_name,RV_MAX_DEV_NAME_LEN ) >= +RV_MAX_DEV_NAME_LEN ) { +rv_err (rv->inx, +"attach: dev_name empty or not nul terminated\n" ); +return -EINVAL ; + } +if (params.in.rdma_mode >RV_RDMA_MODE_MAX ) { +rv_err (rv->inx,"attach: rdma_mode invalid\n" ); +return -EINVAL ; + } +if (params.in.rdma_mode ==RV_RDMA_MODE_KERNEL ) { +if (!params.in.port_num) { +rv_err (rv->inx,"attach: port_num invalid\n" ); +return -EINVAL ; + } +if (params.in.num_conn >RV_MAX_NUM_CONN ) { +rv_err (rv->inx, +"attach: num_conn too large %d max %d\n" , + params.in.num_conn,RV_MAX_NUM_CONN ); +return -EINVAL ; + } +if (params.in.index_bits >RV_MAX_INDEX_BITS ) { +rv_err (rv->inx, +"attach: index_bits too large %d max %d\n" , + params.in.index_bits,RV_MAX_INDEX_BITS ); +return -EINVAL ; + } +if (params.in.job_key_len >RV_MAX_JOB_KEY_LEN ) { +rv_err (rv->inx, +"attach: job_key too large %u max %u\n" , + params.in.job_key_len,RV_MAX_JOB_KEY_LEN ); +return -EINVAL ; + } +if (params.in.cq_entries >RV_MAX_CQ_ENTRIES ) { +rv_err (rv->inx, +"attach: cq_entries too large %d max %d\n" , + params.in.cq_entries,RV_MAX_CQ_ENTRIES ); +return -EINVAL ; + } + }else if (!enable_user_mr) { +rv_err (rv->inx,"attach: rdma_mode user disabled\n" ); +return -EINVAL ; + } + +if (!params.in.num_conn) + params.in.num_conn = num_conn; +if (!params.in.num_conn) + params.in.num_conn =1 ; +if (!params.in.service_id) + params.in.service_id = service_id; +if (!params.in.q_depth) + params.in.q_depth = q_depth; +if (params.in.reconnect_timeout && + params.in.reconnect_timeout *1000 <= params.in.hb_interval) { +rv_err (rv->inx, +"reconnect_timeout (%u secs) < hb_interval (%u msec)\n" , + params.in.reconnect_timeout, params.in.hb_interval); +return -EINVAL ; + } + + umrs =rv_user_mrs_alloc (rv, params.in.mr_cache_size); +if (IS_ERR (umrs)) +return PTR_ERR (umrs); + +mutex_lock (&rv->mutex); +if (rv->attached || rv->was_attached) { +rv_err (rv->inx,"attach: already attached to device\n" ); + ret = -EBUSY ; + goto unlock; + } + + rv->rdma_mode = params.in.rdma_mode; + +switch (rv->rdma_mode) { +case RV_RDMA_MODE_USER : + rv->dev =rv_device_get_add_user (params.in.dev_name, rv); +if (!rv->dev) { +rv_err (rv->inx,"attach: USER ib_dev %s not found\n" , + params.in.dev_name); + ret = -ENODEV ; + goto unlock; + } + rv->index =RV_INVALID ;/* N/A */ + depth_out =0 ;/* N/A */ + reconnect_timeout =0 ;/* N/A */ +break ; +case RV_RDMA_MODE_KERNEL : + ret =rv_user_attach_kernel (rv, ¶ms); +if (ret) { +rv_err (rv->inx,"attach: for kernel mode\n" ); + goto unlock; + } + depth_out = rv->jdev->q_depth; + reconnect_timeout = rv->jdev->reconnect_timeout; +break ; + } + + rv->umrs = umrs; +rv_user_mrs_attach (rv); + params.out.rv_index = rv->index; + params.out.mr_cache_size = umrs->cache.max_size / (1024 *1024 ); + params.out.q_depth = depth_out; + params.out.reconnect_timeout = reconnect_timeout; +if (copy_to_user ((void __user *)arg, ¶ms.out,sizeof (params.out))) { + ret = -EFAULT ; + goto bail_detach; + } + +/* set attached last */ + rv->attached =1 ; + rv->was_attached =1 ; +trace_rv_user_attach (rv->inx, rv->rdma_mode, rv->attached, + params.in.dev_name, rv->cq_entries, rv->index); +mutex_unlock (&rv->mutex); + +return 0 ; + +bail_detach: +rv_device_del_user (rv); +rv_user_detach_all (rv); +mutex_unlock (&rv->mutex); +return ret; +unlock: +mutex_unlock (&rv->mutex); +rv_user_mrs_put (umrs); +return ret; +} + +/* +* detach everything we find, for USER or KERNEL rdma_mode. +* must hold rv->mutex before calling. +* Once rv_user.attached is 0, the rv_user.dev/jdev is not valid +* we are paranoid and detach everything we find even for USER mode +* We wait for umrs->kref to be 1, to ensure all pending writes have put +* their MRs. This way our final umrs_put will free the mr_cache. +* Note: other callers of rv_user_mrs_put don't get rv->mutex +*/ +static void rv_user_detach_all (struct rv_user *rv) +{ +struct rv_conn *conn; +unsigned long sleep_time =msecs_to_jiffies (100 ); + +XA_STATE (xas, &rv->conn_xa,0 ); + +trace_rv_msg_detach_all (rv->inx,"rv_user_detach_all" ,0 ,0 ); + rv->attached =0 ; +if (rv->rdma_mode ==RV_RDMA_MODE_KERNEL && rv->jdev) +rv_job_dev_put_user (rv->jdev); +xas_for_each (&xas, conn,UINT_MAX ) { +trace_rv_msg_uconn_remove (rv->inx,"rv_user remove uconn" , + (u64 )conn,0 ); +xas_store (&xas,NULL ); +rv_conn_put (conn); + } + +if (rv->umrs) { +while (kref_read (&rv->umrs->kref) >1 ) +schedule_timeout_interruptible (sleep_time); + +rv_user_mrs_put (rv->umrs); + rv->umrs =NULL ; +flush_workqueue (rv_wq); + } + +if (rv->rdma_mode ==RV_RDMA_MODE_USER && rv->dev) { +rv_device_put (rv->dev); + rv->dev =NULL ; + }else if (rv->rdma_mode ==RV_RDMA_MODE_KERNEL && rv->jdev) { +rv_user_detach_kernel (rv); + rv->jdev =NULL ; + } + + rv->rdma_mode =RV_RDMA_MODE_USER ; + +complete (&rv->compl); +} + +/* Other cleanup at file close time. Must hold rv->mutex. */ +static void rv_user_cleanup (struct rv_user *rv) +{ +if (rv->cqr) { +trace_rv_msg_cleanup (rv->inx,"freeing event ring" , + (u64 )rv->cqr,0 ); +rv_user_ring_free (rv->cqr); + rv->cqr =NULL ; + } +} + +void rv_detach_user (struct rv_user *rv) +{ +mutex_lock (&rv->mutex); +rv_user_detach_all (rv); +mutex_unlock (&rv->mutex); +} +/* * confirm that we expected a REQ from this remote node on this port. * Note CM swaps src vs dest so dest is remote node here
@@ -154,3 +765,432 @@ rv_find_sconn_from_req(struct ib_cm_id *id,return sconn; } + +/* Technically don't need rv_user.mutex, but play it safe */ +static int doit_get_cache_stats (struct rv_user *rv,unsigned long arg) +{ +struct rv_cache_stats_params_out params; +int ret =0 ; +unsigned long flags; + +mutex_lock (&rv->mutex); + +memset (¶ms,0 ,sizeof (params)); +if (rv->attached && rv->umrs) { +struct rv_mr_cache *cache = &rv->umrs->cache; + +spin_lock_irqsave (&cache->lock, flags); + + params.cache_size = cache->total_size; + params.max_cache_size = cache->stats.max_cache_size; + params.limit_cache_size = cache->max_size / (1024 *1024 ); + params.count = cache->stats.count; + params.max_count = cache->stats.max_count; + params.inuse = cache->stats.inuse; + params.max_inuse = cache->stats.max_inuse; + params.inuse_bytes = cache->stats.inuse_bytes; + params.max_inuse_bytes = cache->stats.max_inuse_bytes; + params.max_refcount = cache->stats.max_refcount; + params.hit = cache->stats.hit; + params.miss = cache->stats.miss; + params.full = cache->stats.full; + params.failed = rv->umrs->stats.failed; + params.remove = cache->stats.remove; + params.evict = cache->stats.evict; + +spin_unlock_irqrestore (&cache->lock, flags); + } + +if (copy_to_user ((void __user *)arg, ¶ms,sizeof (params))) + ret = -EFAULT ; + +mutex_unlock (&rv->mutex); + +return ret; +} + +/* Technically don't need rv_user.mutex, but play it safe */ +static int doit_get_event_stats (struct rv_user *rv,unsigned long arg) +{ +struct rv_event_stats_params_out params; +int ret =0 ; +unsigned long flags; + +mutex_lock (&rv->mutex); + +memset (¶ms,0 ,sizeof (params)); +if (rv->attached && rv->cqr) { +struct rv_user_ring *cqr = rv->cqr; + +spin_lock_irqsave (&cqr->lock, flags); + + params.send_write_cqe = cqr->stats.cqe[RV_WC_RDMA_WRITE ]; + params.send_write_cqe_fail = + cqr->stats.cqe_fail[RV_WC_RDMA_WRITE ]; + params.send_write_bytes = cqr->stats.bytes[RV_WC_RDMA_WRITE ]; + + params.recv_write_cqe = + cqr->stats.cqe[RV_WC_RECV_RDMA_WITH_IMM ]; + params.recv_write_cqe_fail = + cqr->stats.cqe_fail[RV_WC_RECV_RDMA_WITH_IMM ]; + params.recv_write_bytes = + cqr->stats.bytes[RV_WC_RECV_RDMA_WITH_IMM ]; + +spin_unlock_irqrestore (&cqr->lock, flags); + } + +if (copy_to_user ((void __user *)arg, ¶ms,sizeof (params))) + ret = -EFAULT ; + +mutex_unlock (&rv->mutex); + +return ret; +} + +static int rv_file_open (struct inode *inode,struct file *fp) +{ +struct rv_user *rv; + + rv =kzalloc (sizeof (*rv),GFP_KERNEL ); +if (!rv) +return -ENOMEM ; +mutex_init (&rv->mutex); +xa_init_flags (&rv->conn_xa,XA_FLAGS_ALLOC ); +INIT_LIST_HEAD (&rv->user_entry); +init_completion (&rv->compl); + rv->inx =atomic_inc_return (&seq); + rv->index =RV_INVALID ; + fp->private_data = rv; +trace_rv_user_open (rv->inx, rv->rdma_mode, rv->attached,"n/a" , + rv->cq_entries, rv->index); + +return 0 ; +} + +/* +* Remove rv_user from dev->user_list first. If it is already +* removed from dev->user_list by remove_one(), wait for the +* detach to finish. Otherwise detach_all here will remove it. +*/ +static int rv_file_close (struct inode *inode,struct file *fp) +{ +struct rv_user *rv = fp->private_data; + +trace_rv_user_close (rv->inx, rv->rdma_mode, rv->attached,"n/a" , + rv->cq_entries, rv->index); + + fp->private_data =NULL ; + +mutex_lock (&rv->mutex); +if (rv->attached &&rv_device_del_user (rv)) + goto unlock; + +rv_user_detach_all (rv); +unlock: +mutex_unlock (&rv->mutex); +wait_for_completion (&rv->compl); + +mutex_lock (&rv->mutex); +rv_user_cleanup (rv); +mutex_unlock (&rv->mutex); +xa_destroy (&rv->conn_xa); +kfree (rv); + +return 0 ; +} + +static long rv_file_ioctl (struct file *fp,unsigned int cmd,unsigned long arg) +{ +struct rv_user *rv = fp->private_data; + +trace_rv_ioctl (rv->inx, cmd); +switch (cmd) { +case RV_IOCTL_QUERY : +return doit_query (rv, arg); + +case RV_IOCTL_ATTACH : +return doit_attach (rv, arg); + +case RV_IOCTL_REG_MEM : +return doit_reg_mem (rv, arg); + +case RV_IOCTL_DEREG_MEM : +return doit_dereg_mem (rv, arg); + +case RV_IOCTL_GET_CACHE_STATS : +return doit_get_cache_stats (rv, arg); + +case RV_IOCTL_CONN_CREATE : +return doit_conn_create (rv, arg); + +case RV_IOCTL_CONN_CONNECT : +return doit_conn_connect (rv, arg); + +case RV_IOCTL_CONN_CONNECTED : +return doit_conn_connected (rv, arg); + +case RV_IOCTL_CONN_GET_CONN_COUNT : +return doit_conn_get_conn_count (rv, arg); + +case RV_IOCTL_CONN_GET_STATS : +return doit_conn_get_stats (rv, arg); + +case RV_IOCTL_GET_EVENT_STATS : +return doit_get_event_stats (rv, arg); + +case RV_IOCTL_POST_RDMA_WR_IMMED : +return doit_post_rdma_write (rv, arg); + +default : +return -EINVAL ; + } +} + +static const struct file_operations rv_file_ops = { + .owner =THIS_MODULE , + .open = rv_file_open, + .release = rv_file_close, + .unlocked_ioctl = rv_file_ioctl, + .llseek = noop_llseek, + .mmap = rv_file_mmap, +}; + +static void rv_dev_release (struct device *dev) +{ +kfree (rv_dd); + rv_dd =NULL ; +} + +static char *rv_devnode (struct device *dev,umode_t *mode) +{ +if (mode) + *mode =0666 ; +return kasprintf (GFP_KERNEL ,"%s" ,dev_name (dev)); +} + +int rv_file_init (void ) +{ +int ret; +struct device *device; +struct cdev *cdev; + +atomic_set (&seq,0 ); +rv_job_dev_list_init (); + rv_wq =alloc_workqueue ("rv_wq" , +WQ_SYSFS |WQ_HIGHPRI |WQ_CPU_INTENSIVE , +RV_CONN_MAX_ACTIVE_WQ_ENTRIES ); +if (!rv_wq) +return -ENOMEM ; + + rv_wq2 =alloc_workqueue ("rv_wq2" , +WQ_SYSFS |WQ_HIGHPRI |WQ_CPU_INTENSIVE , +RV_CONN_MAX_ACTIVE_WQ_ENTRIES ); +if (!rv_wq2) { + ret = -ENOMEM ; + goto fail_wq; + } + rv_wq3 =alloc_workqueue ("rv_wq3" , +WQ_SYSFS |WQ_HIGHPRI |WQ_CPU_INTENSIVE , +10 ); +if (!rv_wq3) { + ret = -ENOMEM ; + goto fail_wq; + } + + rv_dd =kzalloc (sizeof (*rv_dd),GFP_KERNEL ); +if (!rv_dd) { + ret = -ENOMEM ; + goto fail_wq; + } + + rv_dd->class =class_create (THIS_MODULE ,DRIVER_NAME ); +if (IS_ERR (rv_dd->class)) { + ret =PTR_ERR (rv_dd->class); +pr_err ("Could not create device class: %d\n" , ret); + goto fail_free; + } + rv_dd->class->devnode = rv_devnode; + +/* Allocate the dev_t */ + ret =alloc_chrdev_region (&rv_dd->dev,0 ,1 ,DRIVER_NAME ); +if (ret <0 ) { +pr_err ("Could not allocate chrdev region (err %d)\n" , -ret); + goto fail_destroy; + } + +/* Add the char device to both sysfs and devfs */ + device = &rv_dd->user_device; +device_initialize (device); + device->class = rv_dd->class; + device->parent =NULL ; + device->devt = rv_dd->dev; + device->release = rv_dev_release; +dev_set_name (device,"%s" ,DRIVER_NAME ); + cdev = &rv_dd->user_cdev; +cdev_init (cdev, &rv_file_ops); + cdev->owner =THIS_MODULE ; + ret =cdev_device_add (cdev, device); +if (ret <0 ) { +pr_err ("Could not add cdev for %s\n" ,DRIVER_NAME ); + goto fail_release; + } + +return 0 ; + +fail_release: +unregister_chrdev_region (rv_dd->dev,1 ); +fail_destroy: +class_destroy (rv_dd->class); +fail_free: +kfree (rv_dd); + rv_dd =NULL ; +fail_wq: +if (rv_wq3) { +destroy_workqueue (rv_wq3); + rv_wq3 =NULL ; + } +if (rv_wq2) { +destroy_workqueue (rv_wq2); + rv_wq2 =NULL ; + } +destroy_workqueue (rv_wq); + rv_wq =NULL ; + +return ret; +} + +/* +* We wait for all job devs to finish. At this time, there are no more +* users. Please be reminded that in RV_RDMA_MODE_USER mode, +* no job_dev is allocated. +*/ +void rv_file_uninit (void ) +{ +unsigned long timeout =msecs_to_jiffies (100 ); + +if (rv_dd) { +cdev_device_del (&rv_dd->user_cdev, &rv_dd->user_device); +unregister_chrdev_region (rv_dd->dev,1 ); +class_destroy (rv_dd->class); +put_device (&rv_dd->user_device); + } + +while (atomic_read (&rv_job_dev_cnt) >0 ) +schedule_timeout_interruptible (timeout); + +if (rv_wq3) { +flush_workqueue (rv_wq3); +destroy_workqueue (rv_wq3); + rv_wq3 =NULL ; + } +if (rv_wq2) { +flush_workqueue (rv_wq2); +destroy_workqueue (rv_wq2); + rv_wq2 =NULL ; + } +if (rv_wq) { +flush_workqueue (rv_wq); +destroy_workqueue (rv_wq); + rv_wq =NULL ; + } +} + +/* rv event reporting ring, allocated by mmap */ +static struct rv_user_ring *rv_user_ring_alloc (int rv_inx, +u32 num_entries, +struct vm_area_struct *vma) +{ +struct rv_user_ring *ring; +unsigned long len; +unsigned long pfn; +int ret; + + len =RV_RING_ALLOC_LEN (num_entries); + len =ALIGN (len,SMP_CACHE_BYTES ); +if (len > vma->vm_end - vma->vm_start) { +rv_err (rv_inx,"mmap too small for ring\n" ); +return ERR_PTR (-EINVAL ); + } + + ring =kzalloc (sizeof (*ring),GFP_KERNEL ); +if (!ring) +return ERR_PTR (-ENOMEM ); + + ring->rv_inx = rv_inx; + ring->num_entries = num_entries; + ring->order =get_order (len); + ring->page =__get_free_pages (GFP_KERNEL | __GFP_ZERO, ring->order); +if (!ring->page) { +rv_err (rv_inx,"ring alloc failure\n" ); + ret = -ENOMEM ; + goto fail; + } + pfn =virt_to_phys ((void *)ring->page) >>PAGE_SHIFT ; + +/* remap kernel memory to userspace */ + ret =remap_pfn_range (vma, vma->vm_start, pfn, len, vma->vm_page_prot); +if (ret <0 ) { +rv_err (rv_inx,"remap failed page:0x%lx pfn:0x%lx\n" , + (unsigned long )ring->page, pfn); + goto fail; + } +spin_lock_init (&ring->lock); + ring->hdr = (struct rv_ring_header *)ring->page; + ring->hdr->head =0 ; + ring->hdr->tail =0 ; + ring->hdr->overflow_cnt =0 ; +trace_rv_user_ring_alloc (ring->rv_inx, ring->num_entries, + ring->hdr->head, ring->hdr->tail); +return ring; + +fail: +if (ring->page) +free_pages (ring->page, ring->order); +kfree (ring); +return ERR_PTR (ret); +} + +/* We sanity check ring->page, should always be != 0 here */ +static void rv_user_ring_free (struct rv_user_ring *ring) +{ +trace_rv_user_ring_free (ring->rv_inx, ring->num_entries, + ring->hdr->head, ring->hdr->tail); +if (ring->page) +free_pages (ring->page, ring->order); +kfree (ring); +} + +/* +* we ignore offset, but if we decide we need multiple queues in future we +* can use that to identify which queue is being requested +*/ +static int rv_file_mmap (struct file *fp,struct vm_area_struct *vma) +{ +struct rv_user *rv = fp->private_data; +struct rv_user_ring *ring; +int ret = -EINVAL ; + +trace_rv_msg_mmap (rv->inx,"vma" , vma->vm_start, vma->vm_end); + +mutex_lock (&rv->mutex); +if (rv->cqr) { +rv_err (rv->inx,"ring already allocated\n" ); + goto unlock; + } +if (!rv->cq_entries) { +rv_err (rv->inx,"ring disabled at attach time\n" ); + goto unlock; + } + + ring =rv_user_ring_alloc (rv->inx, rv->cq_entries, vma); +if (IS_ERR (ring)) { + ret =PTR_ERR (ring); + goto unlock; + } + rv->cqr = ring; + ret =0 ; + +unlock: +mutex_unlock (&rv->mutex); +return ret; +}
drivers/infiniband/ulp/rv/rv_main.c +1 -1
@@ -178,7 +178,7 @@ static void rv_device_detach_users(struct rv_device *dev)list_del_init (&rv->user_entry);spin_unlock_irqrestore (&rv_dev_list_lock, flags); -/* Detach user here */ +rv_detach_user (rv);spin_lock_irqsave (&rv_dev_list_lock, flags); }spin_unlock_irqrestore (&rv_dev_list_lock, flags);
drivers/infiniband/ulp/rv/trace_conn.h +18 -0
@@ -233,6 +233,24 @@ DEFINE_EVENT(/* event */ jkey_len, jkey, sid, q_depth, ua_next, refcount) ); +DEFINE_EVENT (/* event */ + rv_jdev_template, rv_jdev_alloc, + TP_PROTO(void *ptr, const char *dev_name, u8 num_conn, u8 index_bits, + u16 loc_gid_index, u32 loc_addr, u8 jkey_len, u8 *jkey, + u64 sid, u32 q_depth, u32 ua_next, u32 refcount), +TP_ARGS (ptr, dev_name, num_conn, index_bits, loc_gid_index, loc_addr, + jkey_len, jkey, sid, q_depth, ua_next, refcount) +); + +DEFINE_EVENT (/* event */ + rv_jdev_template, rv_jdev_release, + TP_PROTO(void *ptr, const char *dev_name, u8 num_conn, u8 index_bits, + u16 loc_gid_index, u32 loc_addr, u8 jkey_len, u8 *jkey, + u64 sid, u32 q_depth, u32 ua_next, u32 refcount), +TP_ARGS (ptr, dev_name, num_conn, index_bits, loc_gid_index, loc_addr, + jkey_len, jkey, sid, q_depth, ua_next, refcount) +); + DECLARE_EVENT_CLASS(/* sconn */ rv_sconn_template, TP_PROTO(void *ptr, u8 index, u32 qp_num, void *conn, u32 flags,
drivers/infiniband/ulp/rv/trace_user.h +169 -0
208 lines not on this page · raw message
Signature · 1 line
2.18.1
RFC PATCH 9/9 RDMA/rv: Integrate the file operations into the rv module
In reply to [email protected]From: Kaike Wan <[email protected]>
Integrate the file operations into the module_init and module_exit functions so that user applications can access the rv module.
--- drivers/infiniband/ulp/rv/rv_main.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/infiniband/ulp/rv/rv_main.c +5 -0
@@ -253,11 +253,16 @@ static int __init rv_init_module(void)return -EINVAL ; } +if (rv_file_init ()) { +ib_unregister_client (&rv_client); +return -EINVAL ; + }return 0 ; } static void __exit rv_cleanup_module(void) { +rv_file_uninit ();ib_unregister_client (&rv_client);rv_deinit_devices (); }
Signature · 1 line
2.18.1
Jason Gunthorpe
maintainerQuoted 5 lines · On Fri, Mar 19, 2021 at 08:56:26AM -0400, [email protected] wrote:
- Basic mode of operations (PSM3 is used as an example for user applications):
- A middleware (like MPI) has out-of-band communication channels between any two nodes, which are used to establish high performance communications for providers such as PSM3.
Huh? Doesn't PSM3 already use it's own special non-verbs char devices that already have memory caches and other stuff? Now you want to throw that all away and do yet another char dev just for HFI? Why?
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
Jason
Quoted 15 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 9:53 AM
To: Wan, Kaike <[email protected]>
Cc: [email protected]; [email protected]; Rimmer, Todd <[email protected]>
Subject: Re: [PATCH RFC 0/9] A rendezvous moduleQuoted 5 lines · On Fri, Mar 19, 2021 at 08:56:26AM -0400, [email protected] wrote:
- Basic mode of operations (PSM3 is used as an example for user applications):
- A middleware (like MPI) has out-of-band communication channels between any two nodes, which are used to establish high performance communications for providers such as PSM3.
Quoted 3 lines
Huh? Doesn't PSM3 already use it's own special non-verbs char devices that already have memory caches and other stuff? Now you want to throw that all away and do yet another char dev just for HFI? Why?
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware.
PSM3 uses standard verbs drivers and supports standard RoCE. A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface. Rather it works with the hfi1 driver through standard verbs interface. The rv module was a new design to bring these concepts to standard transports and hardware.
Quoted 4 lines
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
[Wan, Kaike] While there are similarities in concepts, details are different. Quite frankly this could be viewed as an application accelerator much like RDS served that purpose for Oracle, which continues to be its main use case. The rv module is currently targeting to enables the MPI/OFI/PSM3 application.
The name "rv" is chosen simply because this module is designed to enable the rendezvous protocol of the MPI/OFI/PSM3 application stack for large messages. Short messages are handled by eager transfer through UDP in PSM3.
FYI, there is an OFA presentation from Thurs reviewing PSM3 and RV and covering much of the architecture and rationale.
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerQuoted 15 lines · Kaike
Quoted 15 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 9:53 AM
To: Wan, Kaike <[email protected]>
Cc: [email protected]; [email protected]; Rimmer, Todd <[email protected]>
Subject: Re: [PATCH RFC 0/9] A rendezvous moduleQuoted 5 lines · On Fri, Mar 19, 2021 at 08:56:26AM -0400, [email protected] wrote:
- Basic mode of operations (PSM3 is used as an example for user applications):
- A middleware (like MPI) has out-of-band communication channels between any two nodes, which are used to establish high performance communications for providers such as PSM3.
Quoted 7 lines
Quoted 3 lines
Huh? Doesn't PSM3 already use it's own special non-verbs char devices that already have memory caches and other stuff? Now you want to throw that all away and do yet another char dev just for HFI? Why?
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware. PSM3 uses standard verbs drivers and supports standard RoCE.
Uhhh.. "PSM" has always been about the ipath special char device, and if I recall properly the library was semi-discontinued and merged into libfabric.
So here you are talking about a libfabric verbs provider that doesn't use the ipath style char interface but uses verbs and this rv thing so we call it a libfabric PSM3 provider because thats not confusing to anyone at all..
Quoted 2 lines
A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface.
These are the drivers that aren't merged yet, I see. So why are you sending this now? I'm not interested to look at even more Intel code when their driver saga is still ongoing for years.
Quoted 2 lines
Rather it works with the hfi1 driver through standard verbs interface.
But nobody would do that right? You'd get better results using the hif1 native interfaces instead of their slow fake verbs stuff.
Quoted 6 lines
Quoted 3 lines
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
[Wan, Kaike] While there are similarities in concepts, details are different.
You should list these differences.
Quoted 3 lines
Quite frankly this could be viewed as an application accelerator much like RDS served that purpose for Oracle, which continues to be its main use case.
Obviously, except it seems to be doing the same basic acceleration technique as RDS.
Quoted 4 lines
The name "rv" is chosen simply because this module is designed to enable the rendezvous protocol of the MPI/OFI/PSM3 application stack for large messages. Short messages are handled by eager transfer through UDP in PSM3.
A bad name seems like it will further limit potential re-use of this code.
Jason
Quoted 20 lines · Jason Gunthorpe
Quoted 18 lines · Kaike
Quoted 18 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 9:53 AM
To: Wan, Kaike <[email protected]>
Cc: [email protected]; [email protected]; Rimmer, Todd <[email protected]>
Subject: Re: [PATCH RFC 0/9] A rendezvous moduleQuoted 5 lines · On Fri, Mar 19, 2021 at 08:56:26AM -0400, [email protected] wrote:
- Basic mode of operations (PSM3 is used as an example for user applications):
- A middleware (like MPI) has out-of-band communication channels between any two nodes, which are used to establish high performance communications for providers such as PSM3.Huh? Doesn't PSM3 already use it's own special non-verbs char devices that already have memory caches and other stuff? Now you want to throw that all away and do yet another char dev just for HFI? Why?
Quoted 7 lines
Quoted 3 lines
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware. PSM3 uses standard verbs drivers and supports standard RoCE.
Uhhh.. "PSM" has always been about the ipath special char device, and if I recall properly the library was semi-discontinued and merged into libfabric.
This driver is intended to work with a fork of the PSM2 library. The PSM2 library which is for Omni-Path is now maintained by Cornelis Networks on our GitHub. PSM3 is something from Intel for Ethernet. I know it's a bit confusing.
Quoted 12 lines
So here you are talking about a libfabric verbs provider that doesn't use the ipath style char interface but uses verbs and this rv thing so we call it a libfabric PSM3 provider because thats not confusing to anyone at all..
Quoted 2 lines
A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface.
These are the drivers that aren't merged yet, I see. So why are you sending this now? I'm not interested to look at even more Intel code when their driver saga is still ongoing for years.
Quoted 5 lines
Quoted 2 lines
Rather it works with the hfi1 driver through standard verbs interface.
But nobody would do that right? You'd get better results using the hif1 native interfaces instead of their slow fake verbs stuff.
I can't imagine why. I'm not sure what you mean by our slow fake verbs stuff? We support verbs just fine. It's certainly not fake.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
.
.
Quoted 7 lines
Quoted 4 lines
The name "rv" is chosen simply because this module is designed to enable the rendezvous protocol of the MPI/OFI/PSM3 application stack for large messages. Short messages are handled by eager transfer through UDP in PSM3.
A bad name seems like it will further limit potential re-use of this code.
As to the name, I started this driver when I was still at Intel before
handing the reins to Kaike when we spun out. I plucked the name rv out of the air without much consideration. While the code seems to have changed a lot, the name seems to have stuck. It was as Kaike mentions because it helps enable the rendezvous side of PSM3.
-Denny
Jason Gunthorpe
maintainerQuoted 4 lines · Dennis Dalessandro
Quoted 4 lines
Quoted 3 lines
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware. PSM3 uses standard verbs drivers and supports standard RoCE.
Quoted 8 lines
Quoted 3 lines
Uhhh.. "PSM" has always been about the ipath special char device, and if I recall properly the library was semi-discontinued and merged into libfabric.
This driver is intended to work with a fork of the PSM2 library. The PSM2 library which is for Omni-Path is now maintained by Cornelis Networks on our GitHub. PSM3 is something from Intel for Ethernet. I know it's a bit confusing.
"a bit" huh?
Quoted 12 lines
Quoted 12 lines
So here you are talking about a libfabric verbs provider that doesn't use the ipath style char interface but uses verbs and this rv thing so we call it a libfabric PSM3 provider because thats not confusing to anyone at all..
Quoted 2 lines
A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface.
These are the drivers that aren't merged yet, I see. So why are you sending this now? I'm not interested to look at even more Intel code when their driver saga is still ongoing for years.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Rather it works with the hfi1 driver through standard verbs interface.
But nobody would do that right? You'd get better results using the hif1 native interfaces instead of their slow fake verbs stuff.
I can't imagine why. I'm not sure what you mean by our slow fake verbs stuff? We support verbs just fine. It's certainly not fake.
hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash.
I didn't understand your answer, do you see using this with hfi1 or not?
It looks a lot copy&pasted from the hfi1 driver, so now we are on our third copy of this code :(
And why did it suddenly become a ULP that somehow shares uverbs resources?? I'm pretty skeptical that can be made to work correctly..
Jason
Quoted 1 line
hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. > So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash.
To be clear, different vendors have different priorities and hence different HW designs and approaches. hfi1 approached the HPC latency needs with a uniquely scalable approach with very low latency @scale. Ironically, other vendors have since replicated some of those mechanisms with their own proprietary mechanisms, such as UD-X. hfi1 approached storage needs and large message HPC transfers with a direct data placement mechanism (aka RDMA). It fully supported the verbs API and met the performance needs of it's customers with attractive price and power for its target markets.
Kaike sited hfi1 as just one example of a RDMA verbs device which rv can function for. Various network and server vendors will each make their own recommendations for software configs and various end users will each have their own preferences and requirements.
Todd
Jason Gunthorpe
maintainerQuoted 6 lines · Todd
Quoted 1 line
hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. > So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash.
To be clear, different vendors have different priorities and hence different HW designs and approaches. hfi1 approached the HPC latency needs with a uniquely scalable approach with very low latency @scale.
Yes, we all know the marketing spin the vendors like to use here. A kernel transition is a kernel transition, and the one in the HFI verbs path through all the common code is particularly expensive.
I'm suprirsed to hear someone advocate that is a good thing when we were all told that the hfi1 cdev *must* exist because the kernel transition through verbs was far to expensive.
Quoted 3 lines
Ironically, other vendors have since replicated some
of those mechanisms with their own proprietary mechanisms, such as UD-X.
What is a UD-X?
Quoted 2 lines
Kaike sited hfi1 as just one example of a RDMA verbs device which rv can function for.
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs (see what hns and mlx5 are doing), so I have doubt about this.
It would be better if you could get at least one other device to agree this is reasonable.
Or work with something that already exists for bulk messaging like RDS.
Jason
Quoted 3 lines
I'm suprirsed to hear someone advocate that is a good thing when we were all told that the hfi1 cdev *must* exist because the kernel transition through verbs was far to expensive.
It depends on the goal vendors have with verbs vs other APIs such as libfabric. hfi1's verbs goal was focused on storage bandwidth and the cdev was focused on HPC latency and bandwidth for MPI via PSM2 and libfabric. I'm unclear why we are debating hfi1 here, seems it should be in another thread.
Quoted 1 line
What is a UD-X?
UD-X is a vendor specific set of HW interfaces and wire protocols implemented in UCX for nVidia Connect-X series of network devices. Many of it's concepts are very similar to those which ipath and hfi1 HW and software implemented.
Quoted 2 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Different vendors have different approaches to performance and chose different design trade-offs.
Todd
Jason Gunthorpe
maintainerQuoted 9 lines · Todd
Quoted 3 lines
I'm suprirsed to hear someone advocate that is a good thing when we were all told that the hfi1 cdev *must* exist because the kernel transition through verbs was far to expensive.
It depends on the goal vendors have with verbs vs other APIs such as libfabric. hfi1's verbs goal was focused on storage bandwidth and the cdev was focused on HPC latency and bandwidth for MPI via PSM2 and libfabric. I'm unclear why we are debating hfi1 here, seems it should be in another thread.
Because this is copied from hfi1?
Quoted 5 lines
Quoted 1 line
What is a UD-X?
UD-X is a vendor specific set of HW interfaces and wire protocols implemented in UCX for nVidia Connect-X series of network devices. Many of it's concepts are very similar to those which ipath and hfi1 HW and software implemented.
Oh, there is lots of stuff in UCX, I'm not surprised you similarities to what PSM did since psm/libfabric/ucx are all solving the same problems.
Quoted 5 lines
Quoted 2 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Different vendors have different approaches to performance and chose different design trade-offs.
That isn't my point, by limiting the usability you also restrict the drivers where this would meaningfully be useful.
So far we now know that it is not useful for mlx5 or hfi1, that leaves only hns unknown and still in the HPC arena.
Jason
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 4:55 PM
To: Rimmer, Todd <[email protected]>
Cc: Dennis Dalessandro <[email protected]>; Wan, Oh, there is lots of stuff in UCX, I'm not surprised you similarities to what PSM did since psm/libfabric/ucx are all solving the same problems.Quoted 2 lines
Quoted 2 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Quoted 8 lines
Quoted 2 lines
Different vendors have different approaches to performance and chose different design trade-offs.
That isn't my point, by limiting the usability you also restrict the drivers where this would meaningfully be useful.
So far we now know that it is not useful for mlx5 or hfi1, that leaves only hns unknown and still in the HPC arena.
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Quoted 2 lines
Jason
Quoted 13 lines · Kaike
Quoted 13 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 4:55 PM
To: Rimmer, Todd <[email protected]>
Cc: Dennis Dalessandro <[email protected]>; Wan, Oh, there is lots of stuff in UCX, I'm not surprised you similarities to what PSM did since psm/libfabric/ucx are all solving the same problems.Quoted 5 lines
Quoted 2 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Different vendors have different approaches to performance and chose different design trade-offs.
Quoted 6 lines
Quoted 5 lines
That isn't my point, by limiting the usability you also restrict the drivers where this would meaningfully be useful.
So far we now know that it is not useful for mlx5 or hfi1, that leaves only hns unknown and still in the HPC arena.
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Interesting. I was thinking the opposite. So what's the benefit? When would someone want to do that?
-Denny
Quoted 26 lines
From: Dennis Dalessandro <[email protected]>
Sent: Friday, March 19, 2021 5:28 PM
To: Wan, Kaike <[email protected]>; Jason Gunthorpe <[email protected]>; Rimmer, Todd <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: [PATCH RFC 0/9] A rendezvous moduleQuoted 6 lines · Kaike
Quoted 6 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 4:55 PM
To: Rimmer, Todd <[email protected]>
Cc: Dennis Dalessandro <[email protected]>; Wan, Oh, there is lots of stuff in UCX, I'm not surprised you similarities to what PSM did since psm/libfabric/ucx are all solving thesame problems.
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Different vendors have different approaches to performance and chose different design trade-offs.
That isn't my point, by limiting the usability you also restrict the drivers where this would meaningfully be useful.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
So far we now know that it is not useful for mlx5 or hfi1, that leaves only hns unknown and still in the HPC arena.
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Interesting. I was thinking the opposite. So what's the benefit? When would someone want to do that?
[Wan, Kaike] This is only because rv works with the generic Verbs interface and hfi1 happens to be one of the devices that implements the Verbs interface.
Quoted 2 lines
-Denny
Jason Gunthorpe
maintainerQuoted 20 lines · Kaike
Quoted 20 lines · Kaike
Quoted 6 lines
Quoted 6 lines
From: Jason Gunthorpe <[email protected]>
Sent: Friday, March 19, 2021 4:55 PM
To: Rimmer, Todd <[email protected]>
Cc: Dennis Dalessandro <[email protected]>; Wan, Oh, there is lots of stuff in UCX, I'm not surprised you similarities to what PSM did since psm/libfabric/ucx are all solving thesame problems.
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
Quoted 3 lines
rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs
Different vendors have different approaches to performance and chose different design trade-offs.
That isn't my point, by limiting the usability you also restrict the drivers where this would meaningfully be useful.
So far we now know that it is not useful for mlx5 or hfi1, that leaves only hns unknown and still in the HPC arena.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Interesting. I was thinking the opposite. So what's the benefit? When would someone want to do that?
[Wan, Kaike] This is only because rv works with the generic Verbs interface and hfi1 happens to be one of the devices that implements the Verbs interface.
Well, it works with everything (except EFA), but that is not my point.
I said it is not useful.
Jason
Quoted 4 lines
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Interesting. I was thinking the opposite. So what's the benefit? When would someone want to do that?
The more interesting scenario is for customers who would like to run libfabric and other Open Fabrics Alliance software over various verbs capable hardware. Today PSM2 is a good choice for OPA hardware. However for some other devices without existing libfabric providers, rxm and rxd are the best choices. As was presented in Open Fabrics workshop today by James Erwin, PSM3 offers noticeable benefits over existing libfabric rxm and rxd providers and the rv module offers noticeable performance benefits when using PSM3.
Quoted 4 lines
This driver is intended to work with a fork of the PSM2 library. The PSM2 library which is for Omni-Path is now maintained by Cornelis Networks on our GitHub. PSM3 is something from Intel for Ethernet. I know it's a bit confusing.
Intel retains various IP and trademark rights. Intel's marketing team analyzed and chose the name PSM3. Obviously plusses and minuses to any name choice.
This is not unlike other industry software history where new major revisions often add and remove support for various HW generations. PSM(1) - supported infinipath IB adapters, was a standalone API (various forms).
PSM2 - dropped support for infinipath and IB and added support for Omni-Path, along with various features, also added libfabric support PSM3 - dropped support for Omni-Path, added support for RoCE and verbs capable devices, along with other features,
also dropped PSM2 API and standardized on libfabric.
All three have similar strategies of onload protocols for eager messages and shared kernel/HW resources for large messages and direct data placement (RDMA). So the name Performance Scaled Messaging is meant to reflect the concept and approach as opposed to reflecting a specific HW implementation or even API.
PSM3 is only available as a libfabric provider.
Quoted 2 lines
I haven't had a chance to look beyond the cover letter in depth at how things have changed. I really hope it's not that bad.
While a few stylistic elements got carried forward, as you noticed. This is much different from hfi1 as it doesn't directly access hardware and is hence smaller. We carefully looked at overlap with features in ib_core and the patch set contains a couple minor API additions to ib_core to simplify some operations which others may find useful.
Quoted 1 line
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol.
The focus of the design was to support the bulk transfer part of the MPI rendezvous protocol, hence the name rv. We'd welcome other name suggestions, wanted to keep the name simple and brief.
Quoted 2 lines
No pre-adding reserved stuff
Lots of alignment holes, don't do that either.
We'd like advise on a challenging situation. Some customers desire NICs to support nVidia GPUs in some environments. Unfortunately the nVidia GPU drivers are not upstream, and have not been for years. So we are forced to have both out of tree and upstream versions of the code. We need the same applications to be able to work over both, so we would like the GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things. We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
Todd
Jason Gunthorpe
maintainerQuoted 3 lines · Todd
We'd like advise on a challenging situation. Some customers desire NICs to support nVidia GPUs in some environments. Unfortunately the nVidia GPU drivers are not upstream, and have not been for years.
Yep.
Quoted 2 lines
So we are forced to have both out of tree and upstream versions of the code.
Sure
Quoted 3 lines
We need the same applications to be able to work over
both, so we would like the GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things.
The only answer is to get it upstream past community review before you ship anything to customers.
Quoted 3 lines
We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
Sorry, nope, you have to do something else. We will *not* have garbage in upstream or be restricted in any way based on out of tree code.
Jason
Quoted 5 lines · Todd
Quoted 4 lines
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
Interesting. I was thinking the opposite. So what's the benefit? When would someone want to do that?
The more interesting scenario is for customers who would like to run libfabric and other Open Fabrics Alliance software over various verbs capable hardware.
Ah ok that makes sense. Not that running it over hfi1 is the goal but being able to run over verbs devices. Makes sense to me now.
Quoted 3 lines
Today PSM2 is a good choice for OPA hardware. However for some other devices without existing libfabric providers, rxm and rxd are the best choices. As was presented in Open Fabrics workshop today by James Erwin, PSM3 offers noticeable benefits over existing libfabric rxm and rxd providers and the rv module offers noticeable performance benefits when using PSM3.
For those that haven't seen it the talks will be posted to YouTube and/or OpenFabrics.org web page. There are actually two talks on this stuff. The first of which is by Todd is available now [1], James' talk will be up soon I'm sure.
Quoted 5 lines
Quoted 2 lines
I haven't had a chance to look beyond the cover letter in depth at how things have changed. I really hope it's not that bad.
While a few stylistic elements got carried forward, as you noticed. This is much different from hfi1 as it doesn't directly access hardware and is hence smaller. We carefully looked at overlap with features in
ib_coreand the patch set contains a couple minor API additions toib_coreto simplify some operations which others may find useful.
Right, so if there is common functionality between hfi1 and rv then it might belong in the core. Especially considering if it's something that's common between a ULP and a HW driver.
Quoted 3 lines
Quoted 1 line
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol.
The focus of the design was to support the bulk transfer part of the MPI rendezvous protocol, hence the name rv. We'd welcome other name suggestions, wanted to keep the name simple and brief.
Like I said previously you can place the blame for the name on me. Kaike and Todd just carried it forward. I think Sean had an idea in one of the other replies. Let's hear some other suggestions too.
Quoted 8 lines
Quoted 2 lines
No pre-adding reserved stuff
Lots of alignment holes, don't do that either.We'd like advise on a challenging situation. Some customers desire NICs to support nVidia GPUs in some environments. Unfortunately the nVidia GPU drivers are not upstream, and have not been for years. So we are forced to have both out of tree and upstream versions of the code. We need the same applications to be able to work over both, so we would like the GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things. We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
This problem extends to other drivers as well. I'm also interested in advice on the situation. I don't particularly like this either, but we need a way to accomplish the goal. We owe it to users to be flexible. Please offer suggestions.
[1] https://www.youtube.com/watch?v=iOvt_Iqz0uU
-Denny
Quoted 2 lines · Dennis Dalessandro
Quoted 1 line · Todd
Quoted 1 line
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
<...>
Quoted 7 lines
Quoted 2 lines
We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
This problem extends to other drivers as well. I'm also interested in advice on the situation. I don't particularly like this either, but we need a way to accomplish the goal. We owe it to users to be flexible. Please offer suggestions.
Sorry to interrupt, but it seems that solution was said here [1]. It wasn't said directly, but between the lines it means that you need two things:
1. Upstream everything.
2. Find another vendor that jumps on this RV bandwagon.
[1] https://lore.kernel.org/linux-rdma/[email protected]
Thanks
Quoted 14 lines · Leon Romanovsky
Quoted 2 lines · Dennis Dalessandro
Quoted 1 line · Todd
Quoted 1 line
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
<...>
Quoted 7 lines
Quoted 2 lines
We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
This problem extends to other drivers as well. I'm also interested in advice on the situation. I don't particularly like this either, but we need a way to accomplish the goal. We owe it to users to be flexible. Please offer suggestions.
Quoted 4 lines
Sorry to interrupt, but it seems that solution was said here [1]. It wasn't said directly, but between the lines it means that you need two things:
1. Upstream everything.
Completely agree. However the GPU code from nvidia is not upstream. I don't see that issue getting resolved in this code review. Let's move on.
Quoted 1 line
2. Find another vendor that jumps on this RV bandwagon.
That's not a valid argument. Clearly this works over multiple vendors HW.
We should be trying to get things upstream, not putting up walls when people want to submit new drivers. Calling code "garbage" [1] is not productive.
Quoted 1 line
-Denny
Jason Gunthorpe
maintainerQuoted 20 lines · Dennis Dalessandro
Quoted 14 lines · Leon Romanovsky
Quoted 2 lines · Dennis Dalessandro
Quoted 1 line · Todd
Quoted 1 line
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
<...>
Quoted 7 lines
Quoted 2 lines
We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
This problem extends to other drivers as well. I'm also interested in advice on the situation. I don't particularly like this either, but we need a way to accomplish the goal. We owe it to users to be flexible. Please offer suggestions.
Quoted 4 lines
Sorry to interrupt, but it seems that solution was said here [1]. It wasn't said directly, but between the lines it means that you need two things:
1. Upstream everything.
Quoted 6 lines
Completely agree. However the GPU code from nvidia is not upstream. I don't see that issue getting resolved in this code review. Let's move on.
Quoted 1 line
2. Find another vendor that jumps on this RV bandwagon.
That's not a valid argument. Clearly this works over multiple vendors HW.
At a certain point we have to decide if this is a generic code of some kind or a driver-specific thing like HFI has.
There are also obvious technial problems designing it as a ULP, so it is a very important question to answer. If it will only ever be used by one Intel ethernet chip then maybe it isn't really generic code.
On the other hand it really looks like it overlaps in various ways with both RDS and the qib/hfi1 cdev, so why isn't there any effort to have some commonality??
Quoted 2 lines
We should be trying to get things upstream, not putting up walls when people want to submit new drivers. Calling code "garbage" [1] is not productive.
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
And you certainly can't say "we are already shipping this ABI so we won't change it" either.
You can't square this circle by compromising the upstream world in any way, it is simply not accepted by the overall community.
All of you should know this, I shouldn't have to lecture on this!
Also no, we should not be "trying to get things upstream" as some goal in itself. Upstream is not a trashcan to dump stuff into, someone has to maintain all of this long after it stops being interesting, so there better be good reasons to put it here in the first place.
If it isn't obvious, I'll repeat again: I'm highly annoyed that Intel is sending something like this RV, in the state it is in, to support their own out of tree driver, that they themselves have been dragging their feet on responding to review comments so it can be upstream for *years*.
Jason
Quoted 26 lines · Jason Gunthorpe
Quoted 25 lines · Dennis Dalessandro
Quoted 14 lines · Leon Romanovsky
Quoted 2 lines · Dennis Dalessandro
Quoted 1 line · Todd
Quoted 1 line
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
<...>
Quoted 7 lines
Quoted 2 lines
We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
This problem extends to other drivers as well. I'm also interested in advice on the situation. I don't particularly like this either, but we need a way to accomplish the goal. We owe it to users to be flexible. Please offer suggestions.
Quoted 4 lines
Sorry to interrupt, but it seems that solution was said here [1]. It wasn't said directly, but between the lines it means that you need two things:
1. Upstream everything.Completely agree. However the GPU code from nvidia is not upstream. I don't see that issue getting resolved in this code review. Let's move on.
Quoted 1 line
2. Find another vendor that jumps on this RV bandwagon.
Quoted 8 lines
Quoted 1 line
That's not a valid argument. Clearly this works over multiple vendors HW.
At a certain point we have to decide if this is a generic code of some kind or a driver-specific thing like HFI has.
There are also obvious technial problems designing it as a ULP, so it is a very important question to answer. If it will only ever be used by one Intel ethernet chip then maybe it isn't really generic code.
Todd/Kaike, is there something in here that is specific to the Intel ethernet chip?
Quoted 3 lines
On the other hand it really looks like it overlaps in various ways with both RDS and the qib/hfi1 cdev, so why isn't there any effort to have some commonality??
Maybe that's something that should be explored. Isn't this along the lines of stuff we talked about with the verbs 2.0 stuff, or whatever we ended up calling it.
Quoted 6 lines
Quoted 2 lines
We should be trying to get things upstream, not putting up walls when people want to submit new drivers. Calling code "garbage" [1] is not productive.
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
Not saying you should send this to Linus. I'm saying we should figure out a way to make it better and insulting people and their hard work isn't helping. This is the kind of culture we are trying to get away from in the kernel world.
Quoted 7 lines
And you certainly can't say "we are already shipping this ABI so we won't change it" either.
You can't square this circle by compromising the upstream world in any way, it is simply not accepted by the overall community.
All of you should know this, I shouldn't have to lecture on this!
No one is suggesting to compromise the upstream world. There is a bigger picture here. The answer for this driver may just be take out the reserved stuff. That's pretty simple. The bigger question is how do we deal with non-upstream code. It can't be a problem unique to the RDMA subsystem. How do others deal with it?
Quoted 4 lines
Also no, we should not be "trying to get things upstream" as some goal in itself. Upstream is not a trashcan to dump stuff into, someone has to maintain all of this long after it stops being interesting, so there better be good reasons to put it here in the first place.
That is completely not what I meant at all. I mean we should be trying to get rid of the proprietary, and out of tree stuff. It doesn't at all imply to fling crap against the wall and hope it sticks. We should be encouraging vendors to submit their code and work with them to get it in shape. We clearly have a problem with vendor proprietary code not being open. Let's not encourage that behavior. Vendors should say I want to submit my code to the Linux kernel. Not eh, it's too much of a hassle and kernel people are jerks, so we'll just post it on our website.
Quoted 5 lines
If it isn't obvious, I'll repeat again: I'm highly annoyed that Intel is sending something like this RV, in the state it is in, to support their own out of tree driver, that they themselves have been dragging their feet on responding to review comments so it can be upstream for *years*.
To be fair it is sent as RFC. So to me that says they know it's a ways off from being ready to be included and are starting the process early.
-Denny
Jason Gunthorpe
maintainerQuoted 3 lines · Dennis Dalessandro
Maybe that's something that should be explored. Isn't this along the lines of stuff we talked about with the verbs 2.0 stuff, or whatever we ended up calling it.
I think verbs 2.0 turned into the ioctl uapi stuff, I don't remember anymore.
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
We should be trying to get things upstream, not putting up walls when people want to submit new drivers. Calling code "garbage" [1] is not productive.
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
Not saying you should send this to Linus. I'm saying we should figure out a way to make it better and insulting people and their hard work isn't helping.
No - you've missed what happened here. Todd responded very fast and explained - Intel *knowingly* sent code that was sub-standard as some calculated attempt to make Intel's life maintaining their out of tree drivers easier.
This absoultely needs strong language as I can't catch everything and people need to understand there are real consequences to violating the community trust in this way!
Quoted 1 line
No one is suggesting to compromise the upstream world.
I'm not sure what you mean - what could upstream do to in any way change the situation other than compromising on what will be merged?
Quoted 4 lines
There is a bigger picture here. The answer for this driver may just be take out the reserved stuff. That's pretty simple. The bigger question is how do we deal with non-upstream code. It can't be a problem unique to the RDMA subsystem. How do others deal with it?
The kernel community consensus is that upstream code is on its own.
We don't change the kernel to accomodate out-of-tree code. If the kernel changes and out of tree breaks nobody cares.
Over a long time it has been proven that this methodology is a good way to effect business change to align with the community consensus development model - eventually the costs of being out of tree have bad ROI and companies align.
Quoted 5 lines
That is completely not what I meant at all. I mean we should be trying to get rid of the proprietary, and out of tree stuff. It doesn't at all imply to fling crap against the wall and hope it sticks. We should be encouraging vendors to submit their code and work with them to get it in shape.
Well, I am working on something like 4-5 Intel series right now, and it sometimes does feel like flinging. Have you seen Greg KH's remarks that he won't even look at Intel patches until they have internal expert sign off?
Quoted 3 lines
not encourage that behavior. Vendors should say I want to submit my code to the Linux kernel. Not eh, it's too much of a hassle and kernel people are jerks, so we'll just post it on our website.
There is a very, very fine line between "working with the community" and "the community will provide free expert engineering work to our staff".
Quoted 2 lines
To be fair it is sent as RFC. So to me that says they know it's a ways off from being ready to be included and are starting the process early.
I'm not sure why it is RFC. It sounds like it is much more than this if someone has already made a version with links to the NVIDIA GPU driver and has reached a point where they care about ABI stablility?
Jason
Quoted 1 line
Over a long time it has been proven that this methodology is a good way to effect business change to align with the community consensus development model - eventually the costs of being out of tree have bad ROI and companies align.
Agree. The key question is when will nVidia upstream it's drivers so companies don't have to endure the resulting "bad ROI" of being forced to have unique out of tree solutions.
Quoted 3 lines
Quoted 3 lines
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
Let's not overexaggerate this. The fields we're organized in a logical manner for end user consumption and understanding. A couple resv fields were used for alignment. I alluded to the fact we may have some ideas on how those fields or gaps could be used for GPU support in the future and this might help deal with the lack of upstream nVidia code and reduce the ROI challenges for other vendors which that causes. The discussion seemed centered around the attach parameters, attach is executed once per process at job start.
Todd
Jason Gunthorpe
maintainerQuoted 8 lines · Todd
Quoted 4 lines
Over a long time it has been proven that this methodology is a good way to effect business change to align with the community consensus development model - eventually the costs of being out of tree have bad ROI and companies align.
Agree. The key question is when will nVidia upstream it's drivers so companies don't have to endure the resulting "bad ROI" of being forced to have unique out of tree solutions.
If you are working with NVIDIA GPU and having inefficiencies then you need to take it through your buisness relationship, not here.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
Let's not overexaggerate this. The fields we're organized in a logical manner for end user consumption and understanding. A couple resv fields were used for alignment.
No, a couple of resv fields were randomly added and *a lot* of other mis-alignements were ignored. That is not our standard for ABI design.
Jason
Quoted 6 lines
Quoted 2 lines
To be fair it is sent as RFC. So to me that says they know it's a ways off from being ready to be included and are starting the process early.
I'm not sure why it is RFC. It sounds like it is much more than this if someone has already made a version with links to the NVIDIA GPU driver and has reached a point where they care about ABI stablility?
I can take some blame here. A couple of us were asked to look at this module. Because the functionality is intended to be device agnostic, we assumed there would be more scrutiny, and we weren't sure how acceptable some aspects would be (e.g. mr cache, ib cm data format). Rather than debate this internally for months, rework the code, and still miss, we asked Kaike to post an RFC to get community feedback. For *upstreaming* purposes it was intended as an RFC to gather feedback on the overall approach. That should have been made clearer.
The direct feedback that Kaike is examining is the difference between this approach and RDS.
- Sean
Jason Gunthorpe
maintainerQuoted 7 lines · Sean
Quoted 6 lines
Quoted 2 lines
To be fair it is sent as RFC. So to me that says they know it's a ways off from being ready to be included and are starting the process early.
I'm not sure why it is RFC. It sounds like it is much more than this if someone has already made a version with links to the NVIDIA GPU driver and has reached a point where they care about ABI stablility?
Quoted 9 lines
I can take some blame here. A couple of us were asked to look at this module. Because the functionality is intended to be device agnostic, we assumed there would be more scrutiny, and we weren't sure how acceptable some aspects would be (e.g. mr cache, ib cm data format). Rather than debate this internally for months, rework the code, and still miss, we asked Kaike to post an RFC to get community feedback. For *upstreaming* purposes it was intended as an RFC to gather feedback on the overall approach. That should have been made clearer.
Well, it is hard to even have that kind of conversation when all the details are wrong. The way this interfaces with uverbs is *just wrong*, it completely ignores the locking model for how disassociation works.
Something like this, that is trying to closely integrate with uverbs, really cannot exist without major surgery to uverbs. If you want to do something along these lines the uverbs parts cannot be in a ULP.
The mr cache could be moved into some kind of new uverb, that could be interesting if it is carefully designed.
The actual transport code.. That is going to be really hard. RDS doesn't integrate with uverbs for a reason, the kernel side owns the QPs and PDs.
How you create a QP owned by the kernel but linked to a PD owned by uverbs is going to need very delicate and careful work to be somehow compatible with our disassociation model.
Are you *sure* this needs to be in the kernel? You can't take a context switch to a user process and use the shared verbs FD stuff to create the de-duplicated QPs instead? It is a much simpler design.
Have you thought about an actual *shared verbs QP* ? We have a lot of other shared objects right now, it is not such a big step. It does require inter-process locking though - and that is non-trivial.
A shared QP with a kernel owned send/recv to avoid the locking issue could also be a very interesting solution.
Jason
Quoted 1 line
How you create a QP owned by the kernel but linked to a PD owned by uverbs is going to need very delicate and careful work to be somehow compatible with our disassociation model.
The primary usage mode is as follows:
The RC QP, PD and MR are all in the kernel. The buffer virtual address and len is supplied by the user process and then used to lookup a MR in the cache, upon miss, a kernel MR is created against the kernel PD. There are separate MR caches per user process.
The IOs are initiated by the user, matched to a MR in the cache, then a RDMA Write w/Immed is posted on the kernel RC QP.
In concept, this is not unlike other kernel ULPs which perform direct data placement into user memory but use kernel QPs for connections to remote resources, such as various RDMA storage and filesystem ULPs. The separation of the MR registration call from the IO allows the registration cost of a miss to be partially hidden behind the end to end RTS/CTS exchange which is occurring in user space.
There is a secondary usage mode where the MRs are cached, but created against a user PD and later used by the user process against QPs in the user. We found that usage mode offered some slight latency advantages over the primary mode for tiny jobs, but suffered significant scalability issues. Those latency advantages mainly manifested in microbenchmarks, but did help a few apps.
If it would simplify things, we could focus the discussion on the primary usage mode. Conceptually, the secondary usage mode may be a good candidate for an extension to uverbs (some form of register MR w/caching API where register MR checks a cache and deregister MR merely decrements a reference count in the cache).
Todd
Quoted 25 lines
From: Dennis Dalessandro <[email protected]>
Sent: Sunday, March 21, 2021 1:21 PM
To: Jason Gunthorpe <[email protected]>
Cc: Leon Romanovsky <[email protected]>; Rimmer, Todd <[email protected]>; Wan, Kaike <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH RFC 0/9] A rendezvous moduleQuoted 16 lines · Jason Gunthorpe
Quoted 11 lines · Dennis Dalessandro
Quoted 8 lines · Leon Romanovsky
Quoted 2 lines · Dennis Dalessandro
Quoted 1 line · Todd
Quoted 1 line
Quoted 1 line
[Wan, Kaike] Incorrect. The rv module works with hfi1.
<...>
2. Find another vendor that jumps on this RV bandwagon.
That's not a valid argument. Clearly this works over multiple vendors HW.
At a certain point we have to decide if this is a generic code of some kind or a driver-specific thing like HFI has.
Quoted 6 lines
Quoted 3 lines
There are also obvious technial problems designing it as a ULP, so it is a very important question to answer. If it will only ever be used by one Intel ethernet chip then maybe it isn't really generic code.
Todd/Kaike, is there something in here that is specific to the Intel ethernet chip?
[Wan, Kaike] No. The code is generic to Verbs/ROCE devices.
Quoted 5 lines · Dennis Dalessandro
No one is suggesting to compromise the upstream world. There is a bigger picture here. The answer for this driver may just be take out the reserved stuff. That's pretty simple. The bigger question is how do we deal with non-upstream code. It can't be a problem unique to the RDMA subsystem. How do others deal with it?
By ignoring it, or if it gets too annoying by making life for it so utterly painful that people give up on it.
Quoted 6 lines · Jason Gunthorpe
Quoted 2 lines
We should be trying to get things upstream, not putting up walls when people want to submit new drivers. Calling code "garbage" [1] is not productive.
Putting a bunch of misaligned structures and random reserved fields *is* garbage by the upstream standard and if I send that to Linus I'll get yelled at.
And when this is for out of tree crap it is per the very definition of the word garbage. If I was in your position I would not waste any time on such an utterly disrespectful submission that violates all the community rules.
Quoted 2 lines · Dennis Dalessandro
Completely agree. However the GPU code from nvidia is not upstream. I don't see that issue getting resolved in this code review. Let's move on.
In which case you need to stop wasting everyones time with you piece of crap code base, because it is completely and utterly irrelevant.
Quoted 6 lines · Todd
We'd like advise on a challenging situation. Some customers desire NICs to support nVidia GPUs in some environments. Unfortunately the nVidia GPU drivers are not upstream, and have not been for years. So we are forced to have both out of tree and upstream versions of the code. We need the same applications to be able to work over both, so we would like the GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things. We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
NVIDIA GPUs are supported by drivers/gpu/drm/nouveau/, and your are encourage to support them just like all the other in-tree GPU drivers. Not sure what support a network protocol would need for a specific GPU. You're probably trying to do something amazingly stupid here instead of relying on proper kernel subsystem use.
Jason Gunthorpe
maintainerQuoted 14 lines · Christoph Hellwig
Quoted 11 lines · Todd
We'd like advise on a challenging situation. Some customers desire NICs to support nVidia GPUs in some environments. Unfortunately the nVidia GPU drivers are not upstream, and have not been for years. So we are forced to have both out of tree and upstream versions of the code. We need the same applications to be able to work over both, so we would like the GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things. We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved" mechanisms to hold places for GPU specific fields which can't be upstreamed.
Quoted 5 lines
NVIDIA GPUs are supported by
drivers/gpu/drm/nouveau/, and your are encourage to support them just like all the other in-tree GPU drivers. Not sure what support a network protocol would need for a specific GPU. You're probably trying to do something amazingly stupid here instead of relying on proper kernel subsystem use.
The kernel building block for what they are trying to do with the GPU is the recently merged DMABUF MR support in the RDMA subsystem.
I'd like to think that since Daniel's team at Intel got the DMABUF stuff merged to support the applications Todd's Intel team is building that this RV stuff is already fully ready for dmabuf... (hint hint)
What Todd is alluding to here is the hacky DMABUF alternative that is in the NVIDIA GPU driver - which HPC networking companies must support if they want to interwork with the NVIDIA GPU.
Every RDMA vendor playing in the HPC space has some out-of-tree driver to enable this. :(
Jason
Quoted 6 lines · Jason Gunthorpe
What Todd is alluding to here is the hacky DMABUF alternative that is in the NVIDIA GPU driver - which HPC networking companies must support if they want to interwork with the NVIDIA GPU.
Every RDMA vendor playing in the HPC space has some out-of-tree driver to enable this. :(
I can only recommende everone to buy from a less fucked up GPU or accelerator vendor. We're certainly going to do everything we can to discourage making life easier for people who insist on doing this stupidest possible thing.
Quoted 2 lines
I can only recommende everone to buy from a less f***** up GPU or accelerator vendor.
I would certainly love that. This is not just a recent problem, it's been going on for at least 3-5 years with no end in sight. And the nvidia driver itself is closed-source in the kernel :-( Making tuning and debug even harder and continuing to add costs to NIC vendors other than nVidia themselves to support this.
Back to the topic at hand, yes, there are a few misalignments in the ABI. Most of the structures are carefully aligned. Below I summarize the major structures and their alignment characteristics. In a few places we chose readability for the application programmer by ordering fields in a logical order, such as for statistics.
In one place a superfluous resv field was used (rv_query_params_out) and when I alluded that might be able to be taken advantage in the future to enable a common ABI for GPUs, we went down this deep rat hole.
In studying all the related fields, in most cases if we shuffled everything for maximum packing, the structures would still end up being about the same size and this would all be for non-performance path ABIs.
Here is a summary:
13 structures all perfectly aligned with no gaps, plus 7 structures below.
rv_query_params_out - has one 4 byte reserved field to guarantee alignment for a u64 which follows.
rv_attach_params_in - organized logically as early fields influence how later fields are interpreted. Fair number of fields, two 1 byte gaps and one 7 byte gap. Shuffling this might save about 4-8 bytes tops
rv_cache_stats_params_out - ordered logically by statistics meanings. Two 4 byte gaps could be solved by having a less logical order. Of course, applications reporting these statistics will tend to do output in the original order, so packing this results in a harder to use ABI and more difficult code review for application writers wanting to make sure they report all stats but do so in a logical order.
rv_conn_get_stats_params_out - one 2 byte gap (so the 1 bytes field mimicking the input request can be 1st), three 4 byte gaps. Same explanation as rv_cache_stats_params_out
rv_conn_create_params_in - one 4 byte gap, easy enough to swap
rv_post_write_params_out - one 3 byte gap. Presented in logical order, shuffling would still yield the same size as compiler will round up size.
rv_event - carefully packed and aligned. Had to make this work on a wide range of compilers with a 1 byte common field defining which part of union was relevant. Could put the same field in all unions to get rid of packed attribute if that is preferred. We found other similar examples like this in an older 4.18 kernel, cited one below.
It should be noted, there are existing examples with small gaps or reserved fields in the existing kernel and RDMA stack. A few examples in ib_user_verbs.h include:
ib_uverbs_send_wr - 4 byte gap after ex for the rdma field in union.
ib_uverbs_flow_attr - 2 byte reserved field declared
ib_flow_spec - 2 byte gap after size field
rdma_netdev - 7 byte gap after port_num
./hw/mthca/mthca_eq.c - very similar use of packed for mthca_eqe to the one complained about in rv_event
Todd
Jason Gunthorpe
maintainerQuoted 4 lines · Todd
In studying all the related fields, in most cases if we shuffled everything for maximum packing, the structures would still end up being about the same size and this would all be for non-performance path ABIs.
It is not about size, it is about uABI compatability. Do not place packing holes in uABI structures as it is *very complicated* to guarentee those work across all the combinations of user/kernel we have to support.
Every uABI structure should have explicit padding, it should be organized to minimize padding, and "easy on the programmer" is not a top concern.
You must ensure that all compilers generate the same structure byte layout. On x86 this means a classic i386 compiler, a AMD64 x32 ABI, and the normal AMD64 compiler.
Quoted 3 lines
It should be noted, there are existing examples with small gaps or reserved fields in the existing kernel and RDMA stack. A few examples in
ib_user_verbs.hinclude:
Past mistakes do not excuse future errors. Most of those are not uABI structs.
Jason
Quoted 10 lines · Jason Gunthorpe
Quoted 8 lines · Dennis Dalessandro
Quoted 7 lines
Quoted 3 lines
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware. PSM3 uses standard verbs drivers and supports standard RoCE.
Uhhh.. "PSM" has always been about the ipath special char device, and if I recall properly the library was semi-discontinued and merged into libfabric.
Quoted 6 lines
Quoted 4 lines
This driver is intended to work with a fork of the PSM2 library. The PSM2 library which is for Omni-Path is now maintained by Cornelis Networks on our GitHub. PSM3 is something from Intel for Ethernet. I know it's a bit confusing.
"a bit" huh?
Just a bit. :)
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
So here you are talking about a libfabric verbs provider that doesn't use the ipath style char interface but uses verbs and this rv thing so we call it a libfabric PSM3 provider because thats not confusing to anyone at all..
Quoted 2 lines
A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface.
These are the drivers that aren't merged yet, I see. So why are you sending this now? I'm not interested to look at even more Intel code when their driver saga is still ongoing for years.
Quoted 2 lines
Rather it works with the hfi1 driver through standard verbs interface.
But nobody would do that right? You'd get better results using the hif1 native interfaces instead of their slow fake verbs stuff.
Quoted 7 lines
Quoted 2 lines
I can't imagine why. I'm not sure what you mean by our slow fake verbs stuff? We support verbs just fine. It's certainly not fake.
hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash.
Probably better to argue that in another thread I guess.
Quoted 2 lines
I didn't understand your answer, do you see using this with hfi1 or not?
I don't see how this could ever use hfi1. So no.
Quoted 2 lines
It looks a lot copy&pasted from the hfi1 driver, so now we are on our third copy of this code :(
I haven't had a chance to look beyond the cover letter in depth at how things have changed. I really hope it's not that bad.
Quoted 2 lines
And why did it suddenly become a ULP that somehow shares uverbs resources?? I'm pretty skeptical that can be made to work correctly..
I see your point.
I was just providing some background/clarification. Bottom line this has nothing to do with hfi1 or OPA or PSM2.
-Denny
Jason Gunthorpe
maintainerQuoted 2 lines · Dennis Dalessandro
I was just providing some background/clarification. Bottom line this has nothing to do with hfi1 or OPA or PSM2.
Well, I hope your two companies can agree on naming if Cornelis someday needs a PSM3 for it's HW.
Jason
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
[Wan, Kaike] While there are similarities in concepts, details are different.
Quoted 8 lines
You should list these differences.
Quoted 3 lines
Quite frankly this could be viewed as an application accelerator much like RDS served that purpose for Oracle, which continues to be its main use case.
Obviously, except it seems to be doing the same basic acceleration technique as RDS.
A better name for this might be "scalable RDMA service", with RDMA meaning the transport operation. My understanding is this is intended to be usable over any IB/RoCE device.
I'm not familiar enough with the details of RDS or this service to know the differences.
- Sean
Jason Gunthorpe
maintainerQuoted 13 lines · Sean
Quoted 13 lines
Quoted 6 lines
Quoted 3 lines
I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol. This is all about bulk transfers. It is actually a lot like RDS. Maybe you should be using RDS?
[Wan, Kaike] While there are similarities in concepts, details are different.
You should list these differences.
Quoted 3 lines
Quite frankly this could be viewed as an application accelerator much like RDS served that purpose for Oracle, which continues to be its main use case.
Quoted 6 lines
Quoted 2 lines
Obviously, except it seems to be doing the same basic acceleration technique as RDS.
A better name for this might be "scalable RDMA service", with RDMA meaning the transport operation. My understanding is this is intended to be usable over any IB/RoCE device.
It looks like it is really all about bulk zero-copy transfers with kernel involvment.
Jason