RTRS update for 5.14
Subsystems (2)
Jack Wang
maintainerv3 cover RTRS update for 5.14
Hi Jason, hi Doug,
Please consider to include following changes to the next merge window. It contains:
- Patch 01 ~ 10: Typical code refactoring patches
- Patch 11: Requested by Jason
https://www.spinics.net/lists/linux-rdma/msg102009.html
- Patch 12 RDMA/rtrs-srv: Replace atomic_t with percpu_ref for ids_inflight, we split the sysfs_emit conversion to a seperate patch 13.
- Patch 14 ~ 20: Bug fixes
V3->V2:
- ratelimit error message for first_con check. (Jason)
- split the sysfs_emit conversion to a new patch 13.
V2->V1:
- drop one patch "RDMA/rtrs-clt: No need to check queue_depth when receiving" as requested by Leon
- (void) casting will be removed by next patch set
as requested by Leon.
https://www.spinics.net/lists/linux-rdma/msg102200.html
Dima Stepanov (1):
RDMA/rtrs: Use strscpy instead of strlcpy
Gioh Kim (7):
RDMA/rtrs-clt: Remove MAX_SESS_QUEUE_DEPTH from rtrs_send_sess_info RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
RDMA/rtrs: Define MIN_CHUNK_SIZE
RDMA/rtrs: Do not reset hb_missed_max after re-connection RDMA/rtrs-srv: Duplicated session name is not allowed RDMA/rtrs-srv: Fix memory leak of unfreed rtrs_srv_stats object RDMA/rtrs-clt: Fix memory leak of not-freed sess->stats and
stats->pcpu_stats
Guoqing Jiang (6):
RDMA/rtrs-srv: Kill reject_w_econnreset label RDMA/rtrs-srv: Clean up the code in __rtrs_srv_change_state RDMA/rtrs-clt: Kill rtrs_clt_{start,stop}_hb RDMA/rtrs-clt: Kill rtrs_clt_disconnect_from_sysfs RDMA/rtrs-srv: Kill __rtrs_srv_change_state RDMA/rtrs-clt: Remove redundant 'break'
Jack Wang (2):
RDMA/rtrs-srv: convert scnprintf to sysfs_emit RDMA/rtrs-srv: Fix memory leak when having multiple sessions
Md Haris Iqbal (4):
RDMA/rtrs-srv: Add error messages for cases when failing RDMA
connection
RDMA/rtrs-clt: Check state of the rtrs_clt_sess before reading its
stats
RDMA/rtrs-srv: Replace atomic_t with percpu_ref for ids_inflight RDMA/rtrs-clt: Check if the queue_depth has changed during a
reconnection
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 5 +- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 58 +++---- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 2 +- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 14 +- drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 12 +- drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 163 +++++++++++-------- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 4 +- drivers/infiniband/ulp/rtrs/rtrs.c | 1 - 9 files changed, 146 insertions(+), 114 deletions(-)
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 1/20 RDMA/rtrs-srv: Kill reject_w_econnreset label
From: Guoqing Jiang <[email protected]>
We can goto reject_w_err label after initialize err with -ECONNRESET.
--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +9 -12
@@ -1780,33 +1780,33 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,u16 version, con_num, cid;u16 recon_cnt; -int err; +int err = -ECONNRESET ;if (len <sizeof (*msg)) {pr_err ("Invalid RTRS connection request\n" ); - goto reject_w_econnreset; + goto reject_w_err; }if (le16_to_cpu (msg->magic) !=RTRS_MAGIC ) {pr_err ("Invalid RTRS magic\n" ); - goto reject_w_econnreset; + goto reject_w_err; } version =le16_to_cpu (msg->version);if (version >>8 !=RTRS_PROTO_VER_MAJOR ) {pr_err ("Unsupported major RTRS version: %d, expected %d\n" , version >>8 ,RTRS_PROTO_VER_MAJOR ); - goto reject_w_econnreset; + goto reject_w_err; } con_num =le16_to_cpu (msg->cid_num);if (con_num >4096 ) {/* Sanity check */ pr_err ("Too many connections requested: %d\n" , con_num); - goto reject_w_econnreset; + goto reject_w_err; } cid =le16_to_cpu (msg->cid);if (cid >= con_num) {/* Sanity check */ pr_err ("Incorrect cid: %d >= %d\n" , cid, con_num); - goto reject_w_econnreset; + goto reject_w_err; } recon_cnt =le16_to_cpu (msg->recon_cnt); srv =get_or_create_srv (ctx, &msg->paths_uuid, msg->first_conn);
@@ -1826,7 +1826,7 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,rtrs_err (s,"Session in wrong state: %s\n" ,rtrs_srv_state_str (sess->state));mutex_unlock (&srv->paths_mutex); - goto reject_w_econnreset; + goto reject_w_err; }/* * Sanity checks
@@ -1835,13 +1835,13 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,rtrs_err (s,"Incorrect request: %d, %d\n" , cid, con_num);mutex_unlock (&srv->paths_mutex); - goto reject_w_econnreset; + goto reject_w_err; }if (s->con[cid]) {rtrs_err (s,"Connection already exists: %d\n" , cid);mutex_unlock (&srv->paths_mutex); - goto reject_w_econnreset; + goto reject_w_err; } }else { sess = __alloc_sess(srv, cm_id, con_num, recon_cnt,
@@ -1882,9 +1882,6 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id, reject_w_err:return rtrs_rdma_do_reject (cm_id, err); -reject_w_econnreset: -return rtrs_rdma_do_reject (cm_id, -ECONNRESET ); - close_and_return_err:mutex_unlock (&srv->paths_mutex);close_sess (sess);
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 2/20 RDMA/rtrs-clt: Remove MAX_SESS_QUEUE_DEPTH from rtrs_send_sess_info
In reply to Jack WangFrom: Gioh Kim <[email protected]>
Client receives queue_depth value from server. There is no need to use MAX_SESS_QUEUE_DEPTH value.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +1 -1
@@ -2465,7 +2465,7 @@ static int rtrs_send_sess_info(struct rtrs_clt_sess *sess)int err; rx_sz =sizeof (struct rtrs_msg_info_rsp ); - rx_sz +=sizeof (u64) *MAX_SESS_QUEUE_DEPTH ; + rx_sz +=sizeof (struct rtrs_sg_desc ) * sess->queue_depth; tx_iu = rtrs_iu_alloc(1, sizeof(struct rtrs_msg_info_req), GFP_KERNEL, sess->s.dev->ib_dev, DMA_TO_DEVICE,
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 3/20 RDMA/rtrs-srv: Add error messages for cases when failing RDMA connection
In reply to Jack WangFrom: Md Haris Iqbal <[email protected]>
It was difficult to find out why it failed to establish RDMA connection. This patch adds some messages to show which function has failed why.
--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +7 -1
@@ -1356,8 +1356,10 @@ static struct rtrs_srv *get_or_create_srv(struct rtrs_srv_ctx *ctx,* If this request is not the first connection request from the * client for this session then fail and return error. */ -if (!first_conn) +if (!first_conn) { +pr_err_ratelimited ("Error: Not the first connection request for this session\n" );return ERR_PTR (-ENXIO ); + }/* need to allocate a new srv */ srv =kzalloc (sizeof (*srv),GFP_KERNEL );
@@ -1812,6 +1814,7 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id, srv =get_or_create_srv (ctx, &msg->paths_uuid, msg->first_conn);if (IS_ERR (srv)) { err =PTR_ERR (srv); +pr_err ("get_or_create_srv(), error %d\n" , err); goto reject_w_err; }mutex_lock (&srv->paths_mutex);
@@ -1850,11 +1853,13 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,mutex_unlock (&srv->paths_mutex);put_srv (srv); err =PTR_ERR (sess); +pr_err ("RTRS server session allocation failed: %d\n" , err); goto reject_w_err; } } err =create_con (sess, cm_id, cid);if (err) { +rtrs_err ((&sess->s),"create_con(), error %d\n" , err); (void )rtrs_rdma_do_reject (cm_id, err);/* * Since session has other connections we follow normal way
@@ -1865,6 +1870,7 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id, } err =rtrs_rdma_do_accept (sess, cm_id);if (err) { +rtrs_err ((&sess->s),"rtrs_rdma_do_accept(), error %d\n" , err); (void )rtrs_rdma_do_reject (cm_id, err);/* * Since current connection was successfully added to the
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 4/20 RDMA/rtrs-srv: Clean up the code in __rtrs_srv_change_state
In reply to Jack WangFrom: Guoqing Jiang <[email protected]>
No need to use double switch to check the change of state everywhere, let's change them to "if" to reduce size.
--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +4 -19
@@ -77,32 +77,17 @@ static bool __rtrs_srv_change_state(struct rtrs_srv_sess *sess, old_state = sess->state;switch (new_state) {case RTRS_SRV_CONNECTED : -switch (old_state) { -case RTRS_SRV_CONNECTING : +if (old_state ==RTRS_SRV_CONNECTING ) changed = true; - fallthrough; -default : -break ; - }break ;case RTRS_SRV_CLOSING : -switch (old_state) { -case RTRS_SRV_CONNECTING : -case RTRS_SRV_CONNECTED : +if (old_state ==RTRS_SRV_CONNECTING || + old_state ==RTRS_SRV_CONNECTED ) changed = true; - fallthrough; -default : -break ; - }break ;case RTRS_SRV_CLOSED : -switch (old_state) { -case RTRS_SRV_CLOSING : +if (old_state ==RTRS_SRV_CLOSING ) changed = true; - fallthrough; -default : -break ; - }break ;default :break ;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 5/20 RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
In reply to Jack WangFrom: Gioh Kim <[email protected]>
Max IB immediate data size is 2^28 (MAX_IMM_PAYL_BITS) and the minimum chunk size is 4096 (2^12). Therefore the maximum sess_queue_depth is 65536 (2^16).
--- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-pri.h +8 -5
@@ -47,12 +47,15 @@ enum {MAX_PATHS_NUM =128 ,/* -* With the size of struct rtrs_permit allocated on the client, 4K -* is the maximum number of rtrs_permits we can allocate. This number is -* also used on the client to allocate the IU for the user connection -* to receive the RDMA addresses from the server. +* Max IB immediate data size is 2^28 (MAX_IMM_PAYL_BITS) +* and the minimum chunk size is 4096 (2^12). +* So the maximum sess_queue_depth is 65536 (2^16) in theory. +* But mempool_create, create_qp and ib_post_send fail with +* "cannot allocate memory" error if sess_queue_depth is too big. +* Therefore the pratical max value of sess_queue_depth is +* somewhere between 1 and 65536 and it depends on the system. */ -MAX_SESS_QUEUE_DEPTH =4096 , +MAX_SESS_QUEUE_DEPTH =65536 ,RTRS_HB_INTERVAL_MS =5000 , RTRS_HB_MISSED_MAX = 5,
Signature · 1 line
2.25.1
kernel test robot
build botRe: [PATCHv3 for-next 05/20] RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
Thank you for the patch! Perhaps something to improve:
Hi Jack,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on rdma/for-next]
[also build test WARNING on v5.13-rc3 next-20210528]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jack-Wang/RTRS-update-for-5-14/20210528-193313 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: x86_64-randconfig-a012-20210526 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 6505c630407c5feec818f0bb1c284f9eeebf2071)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/66f95f659060028d1f0f91473ad1c16a6595fcac
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jack-Wang/RTRS-update-for-5-14/20210528-193313 git checkout 66f95f659060028d1f0f91473ad1c16a6595fcac
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]
if (queue_depth > MAX_SESS_QUEUE_DEPTH) {
~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~
include/linux/compiler.h:56:47: note: expanded from macro 'if'
#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
^~~~
include/linux/compiler.h:58:86: note: expanded from macro '__trace_if_var'
#define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
^~~~
include/linux/compiler.h:69:3: note: expanded from macro '__trace_if_value'
(cond) ? \
^~~~
>> drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]
if (queue_depth > MAX_SESS_QUEUE_DEPTH) {
~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~
include/linux/compiler.h:56:47: note: expanded from macro 'if'
#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
^~~~
include/linux/compiler.h:58:52: note: expanded from macro '__trace_if_var'
#define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
^~~~
>> drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]
if (queue_depth > MAX_SESS_QUEUE_DEPTH) {
~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~
include/linux/compiler.h:56:47: note: expanded from macro 'if'
#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
^~~~
include/linux/compiler.h:58:61: note: expanded from macro '__trace_if_var'
#define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
^~~~3 warnings generated.
vim +1786 drivers/infiniband/ulp/rtrs/rtrs-clt.c
6a98d71daea186 Jack Wang 2020-05-11 1750
6a98d71daea186 Jack Wang 2020-05-11 1751 static int rtrs_rdma_conn_established(struct rtrs_clt_con *con,
6a98d71daea186 Jack Wang 2020-05-11 1752 struct rdma_cm_event *ev)
6a98d71daea186 Jack Wang 2020-05-11 1753 {
6a98d71daea186 Jack Wang 2020-05-11 1754 struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
6a98d71daea186 Jack Wang 2020-05-11 1755 struct rtrs_clt *clt = sess->clt;
6a98d71daea186 Jack Wang 2020-05-11 1756 const struct rtrs_msg_conn_rsp *msg;
6a98d71daea186 Jack Wang 2020-05-11 1757 u16 version, queue_depth;
6a98d71daea186 Jack Wang 2020-05-11 1758 int errno;
6a98d71daea186 Jack Wang 2020-05-11 1759 u8 len;
6a98d71daea186 Jack Wang 2020-05-11 1760
6a98d71daea186 Jack Wang 2020-05-11 1761 msg = ev->param.conn.private_data;
6a98d71daea186 Jack Wang 2020-05-11 1762 len = ev->param.conn.private_data_len;
6a98d71daea186 Jack Wang 2020-05-11 1763 if (len < sizeof(*msg)) {
6a98d71daea186 Jack Wang 2020-05-11 1764 rtrs_err(clt, "Invalid RTRS connection response\n");
6a98d71daea186 Jack Wang 2020-05-11 1765 return -ECONNRESET;
6a98d71daea186 Jack Wang 2020-05-11 1766 }
6a98d71daea186 Jack Wang 2020-05-11 1767 if (le16_to_cpu(msg->magic) != RTRS_MAGIC) {
6a98d71daea186 Jack Wang 2020-05-11 1768 rtrs_err(clt, "Invalid RTRS magic\n");
6a98d71daea186 Jack Wang 2020-05-11 1769 return -ECONNRESET;
6a98d71daea186 Jack Wang 2020-05-11 1770 }
6a98d71daea186 Jack Wang 2020-05-11 1771 version = le16_to_cpu(msg->version);
6a98d71daea186 Jack Wang 2020-05-11 1772 if (version >> 8 != RTRS_PROTO_VER_MAJOR) {
6a98d71daea186 Jack Wang 2020-05-11 1773 rtrs_err(clt, "Unsupported major RTRS version: %d, expected %d\n",
6a98d71daea186 Jack Wang 2020-05-11 1774 version >> 8, RTRS_PROTO_VER_MAJOR);
6a98d71daea186 Jack Wang 2020-05-11 1775 return -ECONNRESET;
6a98d71daea186 Jack Wang 2020-05-11 1776 }
6a98d71daea186 Jack Wang 2020-05-11 1777 errno = le16_to_cpu(msg->errno);
6a98d71daea186 Jack Wang 2020-05-11 1778 if (errno) {
6a98d71daea186 Jack Wang 2020-05-11 1779 rtrs_err(clt, "Invalid RTRS message: errno %d\n",
6a98d71daea186 Jack Wang 2020-05-11 1780 errno);
6a98d71daea186 Jack Wang 2020-05-11 1781 return -ECONNRESET;
6a98d71daea186 Jack Wang 2020-05-11 1782 }
6a98d71daea186 Jack Wang 2020-05-11 1783 if (con->c.cid == 0) {
6a98d71daea186 Jack Wang 2020-05-11 1784 queue_depth = le16_to_cpu(msg->queue_depth);
6a98d71daea186 Jack Wang 2020-05-11 1785
6a98d71daea186 Jack Wang 2020-05-11 @1786 if (queue_depth > MAX_SESS_QUEUE_DEPTH) {
6a98d71daea186 Jack Wang 2020-05-11 1787 rtrs_err(clt, "Invalid RTRS message: queue=%d\n",
6a98d71daea186 Jack Wang 2020-05-11 1788 queue_depth);
6a98d71daea186 Jack Wang 2020-05-11 1789 return -ECONNRESET;
6a98d71daea186 Jack Wang 2020-05-11 1790 }
6a98d71daea186 Jack Wang 2020-05-11 1791 if (!sess->rbufs || sess->queue_depth < queue_depth) {
6a98d71daea186 Jack Wang 2020-05-11 1792 kfree(sess->rbufs);
6a98d71daea186 Jack Wang 2020-05-11 1793 sess->rbufs = kcalloc(queue_depth, sizeof(*sess->rbufs),
6a98d71daea186 Jack Wang 2020-05-11 1794 GFP_KERNEL);
6a98d71daea186 Jack Wang 2020-05-11 1795 if (!sess->rbufs)
6a98d71daea186 Jack Wang 2020-05-11 1796 return -ENOMEM;
6a98d71daea186 Jack Wang 2020-05-11 1797 }
6a98d71daea186 Jack Wang 2020-05-11 1798 sess->queue_depth = queue_depth;
6a98d71daea186 Jack Wang 2020-05-11 1799 sess->max_hdr_size = le32_to_cpu(msg->max_hdr_size);
6a98d71daea186 Jack Wang 2020-05-11 1800 sess->max_io_size = le32_to_cpu(msg->max_io_size);
6a98d71daea186 Jack Wang 2020-05-11 1801 sess->flags = le32_to_cpu(msg->flags);
6a98d71daea186 Jack Wang 2020-05-11 1802 sess->chunk_size = sess->max_io_size + sess->max_hdr_size;
6a98d71daea186 Jack Wang 2020-05-11 1803
6a98d71daea186 Jack Wang 2020-05-11 1804 /*
6a98d71daea186 Jack Wang 2020-05-11 1805 * Global queue depth and IO size is always a minimum.
6a98d71daea186 Jack Wang 2020-05-11 1806 * If while a reconnection server sends us a value a bit
6a98d71daea186 Jack Wang 2020-05-11 1807 * higher - client does not care and uses cached minimum.
6a98d71daea186 Jack Wang 2020-05-11 1808 *
6a98d71daea186 Jack Wang 2020-05-11 1809 * Since we can have several sessions (paths) restablishing
6a98d71daea186 Jack Wang 2020-05-11 1810 * connections in parallel, use lock.
6a98d71daea186 Jack Wang 2020-05-11 1811 */
6a98d71daea186 Jack Wang 2020-05-11 1812 mutex_lock(&clt->paths_mutex);
6a98d71daea186 Jack Wang 2020-05-11 1813 clt->queue_depth = min_not_zero(sess->queue_depth,
6a98d71daea186 Jack Wang 2020-05-11 1814 clt->queue_depth);
6a98d71daea186 Jack Wang 2020-05-11 1815 clt->max_io_size = min_not_zero(sess->max_io_size,
6a98d71daea186 Jack Wang 2020-05-11 1816 clt->max_io_size);
6a98d71daea186 Jack Wang 2020-05-11 1817 mutex_unlock(&clt->paths_mutex);
6a98d71daea186 Jack Wang 2020-05-11 1818
6a98d71daea186 Jack Wang 2020-05-11 1819 /*
6a98d71daea186 Jack Wang 2020-05-11 1820 * Cache the hca_port and hca_name for sysfs
6a98d71daea186 Jack Wang 2020-05-11 1821 */
6a98d71daea186 Jack Wang 2020-05-11 1822 sess->hca_port = con->c.cm_id->port_num;
6a98d71daea186 Jack Wang 2020-05-11 1823 scnprintf(sess->hca_name, sizeof(sess->hca_name),
6a98d71daea186 Jack Wang 2020-05-11 1824 sess->s.dev->ib_dev->name);
6a98d71daea186 Jack Wang 2020-05-11 1825 sess->s.src_addr = con->c.cm_id->route.addr.src_addr;
03e9b33a0fd677 Md Haris Iqbal 2021-02-12 1826 /* set for_new_clt, to allow future reconnect on any path */
03e9b33a0fd677 Md Haris Iqbal 2021-02-12 1827 sess->for_new_clt = 1;
6a98d71daea186 Jack Wang 2020-05-11 1828 }
6a98d71daea186 Jack Wang 2020-05-11 1829
6a98d71daea186 Jack Wang 2020-05-11 1830 return 0;
6a98d71daea186 Jack Wang 2020-05-11 1831 }
6a98d71daea186 Jack Wang 2020-05-11 1832 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Jinpu Wang
maintainerRe: [PATCHv3 for-next 05/20] RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
Quoted 27 lines · kernel test robot
Hi Jack,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on rdma/for-next]
[also build test WARNING on v5.13-rc3 next-20210528]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]url: https://github.com/0day-ci/linux/commits/Jack-Wang/RTRS-update-for-5-14/20210528-193313 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: x86_64-randconfig-a012-20210526 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 6505c630407c5feec818f0bb1c284f9eeebf2071)reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# installx86_64cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/66f95f659060028d1f0f91473ad1c16a6595fcac
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jack-Wang/RTRS-update-for-5-14/20210528-193313 git checkout66f95f659060028d1f0f91473ad1c16a6595fcac
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
Quoted 8 lines
If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]if (queue_depth > MAX_SESS_QUEUE_DEPTH) { ~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~
Thanks for the reporting.
As the check is checking against u16 max,I think we should reduce MAX_SESS_QUEUE_DEPTH to 65535, and drop the check in line rtrs-clt:
1786
Jason, you mentioned v3 is applied in for-next, I guess I'll wait when you push it out, and send the patch to fix this. is it ok?
Thanks!
Re: [PATCHv3 for-next 05/20] RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
Quoted 36 lines · Jinpu Wang
Quoted 27 lines · kernel test robot
Hi Jack,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on rdma/for-next]
[also build test WARNING on v5.13-rc3 next-20210528]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]url: https://github.com/0day-ci/linux/commits/Jack-Wang/RTRS-update-for-5-14/20210528-193313 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: x86_64-randconfig-a012-20210526 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 6505c630407c5feec818f0bb1c284f9eeebf2071)reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# installx86_64cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/66f95f659060028d1f0f91473ad1c16a6595fcac
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jack-Wang/RTRS-update-for-5-14/20210528-193313 git checkout66f95f659060028d1f0f91473ad1c16a6595fcac
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64Quoted 8 lines
If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]if (queue_depth > MAX_SESS_QUEUE_DEPTH) { ~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~
Quoted 8 lines
Thanks for the reporting.
As the check is checking against u16 max,I think we should reduce
MAX_SESS_QUEUE_DEPTHto 65535, and drop the check in line rtrs-clt:
1786Jason, you mentioned v3 is applied in for-next, I guess I'll wait when you push it out, and send the patch to fix this. is it ok?
Send me a fix right away and I'll fix the original patch
Jason
Jinpu Wang
maintainerRe: [PATCHv3 for-next 05/20] RDMA/rtrs: Change MAX_SESS_QUEUE_DEPTH
Quoted 40 lines · Jason Gunthorpe
Quoted 38 lines · Jinpu Wang
Quoted 27 lines · kernel test robot
Hi Jack,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on rdma/for-next]
[also build test WARNING on v5.13-rc3 next-20210528]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]url: https://github.com/0day-ci/linux/commits/Jack-Wang/RTRS-update-for-5-14/20210528-193313 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: x86_64-randconfig-a012-20210526 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 6505c630407c5feec818f0bb1c284f9eeebf2071)reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# installx86_64cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/66f95f659060028d1f0f91473ad1c16a6595fcac
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jack-Wang/RTRS-update-for-5-14/20210528-193313 git checkout66f95f659060028d1f0f91473ad1c16a6595fcac
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64Quoted 8 lines
If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
drivers/infiniband/ulp/rtrs/rtrs-clt.c:1786:19: warning: result of comparison of constant 'MAX_SESS_QUEUE_DEPTH' (65536) with expression of type 'u16' (aka 'unsigned short') is always false [-Wtautological-constant-out-of-range-compare]if (queue_depth > MAX_SESS_QUEUE_DEPTH) { ~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~Thanks for the reporting.
Quoted 8 lines
Quoted 6 lines
As the check is checking against u16 max,I think we should reduce
MAX_SESS_QUEUE_DEPTHto 65535, and drop the check in line rtrs-clt:
1786Jason, you mentioned v3 is applied in for-next, I guess I'll wait when you push it out, and send the patch to fix this. is it ok?
Send me a fix right away and I'll fix the original patch
ok, it's here:
https://lore.kernel.org/linux-rdma/20210531122835.58329-1-jinpu.wang(a)ionos.com/T/#u
Quoted 2 lines
Jason
Thanks!
Jack Wang
maintainerPATCH v3 6/20 RDMA/rtrs: Define MIN_CHUNK_SIZE
In reply to Jack WangFrom: Gioh Kim <[email protected]>
Define MIN_CHUNK_SIZE to replace the hard-coding number. We need 4k for metadata, so MIN_CHUNK_SIZE should be at least 8k.
--- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-pri.h +1 -0
@@ -56,6 +56,7 @@ enum {* somewhere between 1 and 65536 and it depends on the system. */ MAX_SESS_QUEUE_DEPTH =65536 , +MIN_CHUNK_SIZE =8192 ,RTRS_HB_INTERVAL_MS =5000 , RTRS_HB_MISSED_MAX = 5,
drivers/infiniband/ulp/rtrs/rtrs-srv.c +2 -2
@@ -2165,9 +2165,9 @@ static int check_module_params(void) sess_queue_depth, 1, MAX_SESS_QUEUE_DEPTH);return -EINVAL ; } -if (max_chunk_size <4096 || !is_power_of_2 (max_chunk_size)) { +if (max_chunk_size <MIN_CHUNK_SIZE || !is_power_of_2 (max_chunk_size)) {pr_err ("Invalid max_chunk_size value %d, has to be >= %d and should be power of two.\n" , - max_chunk_size,4096 ); + max_chunk_size,MIN_CHUNK_SIZE );return -EINVAL ; }
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 7/20 RDMA/rtrs: Use strscpy instead of strlcpy
In reply to Jack WangFrom: Dima Stepanov <[email protected]>
During checkpatch analyzing the following warning message was found:
WARNING:STRLCPY: Prefer strscpy over strlcpy - see:
https://lore.kernel.org/r/CAHk-=wgfRnXz0W3D37d01q3JFkr_i_uTL=V6A6G1oUZcprmknw@mail.gmail.com/
Fix it by using strscpy calls instead of strlcpy.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 4 ++-- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +2 -2
@@ -1505,7 +1505,7 @@ static struct rtrs_clt_sess *alloc_sess(struct rtrs_clt *clt,if (path->src)memcpy (&sess->s.src_addr, path->src,rdma_addr_size ((struct sockaddr *)path->src)); -strlcpy (sess->s.sessname, clt->sessname,sizeof (sess->s.sessname)); +strscpy (sess->s.sessname, clt->sessname,sizeof (sess->s.sessname)); sess->clt = clt; sess->max_pages_per_mr = max_segments;init_waitqueue_head (&sess->state_wq);
@@ -2652,7 +2652,7 @@ static struct rtrs_clt *alloc_clt(const char *sessname, size_t paths_num, clt->priv = priv; clt->link_ev = link_ev; clt->mp_policy =MP_POLICY_MIN_INFLIGHT ; -strlcpy (clt->sessname, sessname,sizeof (clt->sessname)); +strscpy (clt->sessname, sessname,sizeof (clt->sessname));init_waitqueue_head (&clt->permits_wait);mutex_init (&clt->paths_ev_mutex);mutex_init (&clt->paths_mutex);
drivers/infiniband/ulp/rtrs/rtrs-srv.c +3 -3
@@ -780,7 +780,7 @@ static int process_info_req(struct rtrs_srv_con *con, rwr =kcalloc (sess->mrs_num,sizeof (*rwr),GFP_KERNEL );if (unlikely (!rwr))return -ENOMEM ; -strlcpy (sess->s.sessname, msg->sessname,sizeof (sess->s.sessname)); +strscpy (sess->s.sessname, msg->sessname,sizeof (sess->s.sessname)); tx_sz =sizeof (*rsp); tx_sz +=sizeof (rsp->desc[0 ]) * sess->mrs_num;
@@ -1261,7 +1261,7 @@ int rtrs_srv_get_sess_name(struct rtrs_srv *srv, char *sessname, size_t len)list_for_each_entry (sess, &srv->paths_list, s.entry) {if (sess->state !=RTRS_SRV_CONNECTED )continue ; -strlcpy (sessname, sess->s.sessname, +strscpy (sessname, sess->s.sessname,min_t (size_t,sizeof (sess->s.sessname), len)); err =0 ;break ;
@@ -1715,7 +1715,7 @@ static struct rtrs_srv_sess *__alloc_sess(struct rtrs_srv *srv, path.src = &sess->s.src_addr; path.dst = &sess->s.dst_addr;rtrs_addr_to_str (&path, str,sizeof (str)); -strlcpy (sess->s.sessname, str,sizeof (sess->s.sessname)); +strscpy (sess->s.sessname, str,sizeof (sess->s.sessname)); sess->s.con_num = con_num; sess->s.recon_cnt = recon_cnt;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 8/20 RDMA/rtrs-clt: Kill rtrs_clt_{start,stop}_hb
In reply to Jack WangFrom: Guoqing Jiang <[email protected]>
The two wrappers are not needed since we can call rtrs_{start,stop}_hb directly.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +2 -12
@@ -1449,16 +1449,6 @@ static void rtrs_clt_init_hb(struct rtrs_clt_sess *sess) rtrs_wq); } -static void rtrs_clt_start_hb (struct rtrs_clt_sess *sess) -{ -rtrs_start_hb (&sess->s); -} - -static void rtrs_clt_stop_hb (struct rtrs_clt_sess *sess) -{ -rtrs_stop_hb (&sess->s); -} -static void rtrs_clt_reconnect_work (struct work_struct *work);static void rtrs_clt_close_work (struct work_struct *work);
@@ -2098,7 +2088,7 @@ static void rtrs_clt_stop_and_destroy_conns(struct rtrs_clt_sess *sess)*/ synchronize_rcu (); -rtrs_clt_stop_hb (sess); +rtrs_stop_hb (&sess->s);/* * The order it utterly crucial: firstly disconnect and complete all
@@ -2291,7 +2281,7 @@ static int init_conns(struct rtrs_clt_sess *sess)if (err) goto destroy; -rtrs_clt_start_hb (sess); +rtrs_start_hb (&sess->s);return 0 ;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 9/20 RDMA/rtrs-clt: Kill rtrs_clt_disconnect_from_sysfs
In reply to Jack WangFrom: Guoqing Jiang <[email protected]>
The function is just a wrapper of rtrs_clt_close_conns, let's call rtrs_clt_close_conns directly.
--- drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 5 +---- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 9 +-------- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 2 +- 3 files changed, 3 insertions(+), 13 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c +1 -4
@@ -250,7 +250,6 @@ static ssize_t rtrs_clt_disconnect_store(struct kobject *kobj, const char *buf, size_t count) {struct rtrs_clt_sess *sess; -int ret; sess = container_of(kobj, struct rtrs_clt_sess, kobj);if (!sysfs_streq (buf,"1" )) {
@@ -258,9 +257,7 @@ static ssize_t rtrs_clt_disconnect_store(struct kobject *kobj, attr->attr.name, buf);return -EINVAL ; } - ret =rtrs_clt_disconnect_from_sysfs (sess); -if (ret) -return ret; +rtrs_clt_close_conns (sess, true);return count; }
drivers/infiniband/ulp/rtrs/rtrs-clt.c +1 -8
@@ -1859,7 +1859,7 @@ static int rtrs_rdma_conn_rejected(struct rtrs_clt_con *con,return -ECONNRESET ; } -static void rtrs_clt_close_conns (struct rtrs_clt_sess *sess,bool wait) +void rtrs_clt_close_conns (struct rtrs_clt_sess *sess,bool wait) {if (rtrs_clt_change_state_get_old (sess,RTRS_CLT_CLOSING ,NULL ))queue_work (rtrs_wq, &sess->close_work);
@@ -2831,13 +2831,6 @@ int rtrs_clt_reconnect_from_sysfs(struct rtrs_clt_sess *sess)return err; } -int rtrs_clt_disconnect_from_sysfs (struct rtrs_clt_sess *sess) -{ -rtrs_clt_close_conns (sess, true); - -return 0 ; -} -int rtrs_clt_remove_path_from_sysfs (struct rtrs_clt_sess *sess,const struct attribute *sysfs_self) {
drivers/infiniband/ulp/rtrs/rtrs-clt.h +1 -1
@@ -202,7 +202,7 @@ static inline struct rtrs_permit *get_permit(struct rtrs_clt *clt, int idx) }int rtrs_clt_reconnect_from_sysfs (struct rtrs_clt_sess *sess); -int rtrs_clt_disconnect_from_sysfs (struct rtrs_clt_sess *sess); +void rtrs_clt_close_conns (struct rtrs_clt_sess *sess,bool wait);int rtrs_clt_create_path_from_sysfs (struct rtrs_clt *clt,struct rtrs_addr *addr); int rtrs_clt_remove_path_from_sysfs(struct rtrs_clt_sess *sess,
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 10/20 RDMA/rtrs-srv: Kill __rtrs_srv_change_state
In reply to Jack WangFrom: Guoqing Jiang <[email protected]>
No need since the only user is rtrs_srv_change_state.
--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +3 -14
@@ -67,13 +67,13 @@ static inline struct rtrs_srv_sess *to_srv_sess(struct rtrs_sess *s) return container_of(s, struct rtrs_srv_sess, s); } -static bool __rtrs_srv_change_state (struct rtrs_srv_sess *sess, -enum rtrs_srv_state new_state) +static bool rtrs_srv_change_state (struct rtrs_srv_sess *sess, +enum rtrs_srv_state new_state) {enum rtrs_srv_state old_state;bool changed = false; -lockdep_assert_held (&sess->state_lock); +spin_lock_irq (&sess->state_lock); old_state = sess->state;switch (new_state) {case RTRS_SRV_CONNECTED :
@@ -94,17 +94,6 @@ static bool __rtrs_srv_change_state(struct rtrs_srv_sess *sess, }if (changed) sess->state = new_state; - -return changed; -} - -static bool rtrs_srv_change_state (struct rtrs_srv_sess *sess, -enum rtrs_srv_state new_state) -{ -bool changed; - -spin_lock_irq (&sess->state_lock); - changed =__rtrs_srv_change_state (sess, new_state);spin_unlock_irq (&sess->state_lock);return changed;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 11/20 RDMA/rtrs-clt: Remove redundant 'break'
In reply to Jack WangFrom: Guoqing Jiang <[email protected]>
It is duplicated with the very next line
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0 -1
@@ -655,7 +655,6 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc)rtrs_err (con->c.sess,"rtrs_post_recv_empty(): %d\n" , err);rtrs_rdma_error_recovery (con); -break ; }break ;case IB_WC_RECV :
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 12/20 RDMA/rtrs-clt: Check state of the rtrs_clt_sess before reading its stats
In reply to Jack WangFrom: Md Haris Iqbal <[email protected]>
When get_next_path_min_inflight is called to select the next path, it iterates over the list of available rtrs_clt_sess (paths). It then reads the number of inflight IOs for that path to select one which has the least inflight IO.
But it may so happen that rtrs_clt_sess (path) is no longer in the connected state because closing or error recovery paths can change the status of the rtrs_clt_Sess.
For example, the client sent the heart-beat and did not get the response, it would change the session status and stop IO processing. The added checking of this patch can prevent accessing the broken path and generating duplicated error messages.
It is ok if the status is changed after checking the status because the error recovery path does not free memory and only tries to reconnection. And also it is ok if the session is closed after checking the status because closing the session changes the session status and flush all IO beforing free memory. If the session is being accessed for IO processing, the closing session will wait.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +3 -0
@@ -813,6 +813,9 @@ static struct rtrs_clt_sess *get_next_path_min_inflight(struct path_it *it)int inflight;list_for_each_entry_rcu (sess, &clt->paths_list, s.entry) { +if (unlikely (READ_ONCE (sess->state) !=RTRS_CLT_CONNECTED )) +continue ; +if (unlikely (!list_empty (raw_cpu_ptr (sess->mp_skip_entry))))continue ;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 13/20 RDMA/rtrs-srv: Replace atomic_t with percpu_ref for ids_inflight
In reply to Jack WangFrom: Md Haris Iqbal <[email protected]>
ids_inflight is used to track the inflight IOs. But the use of atomic_t variable can cause performance drops and can also become a performance bottleneck.
This commit replaces the use of atomic_t with a percpu_ref structure. The advantage it offers is, it doesn't check if the reference has fallen to 0, until the user explicitly signals it to; and that is done by the percpu_ref_kill() function call. After that, the percpu_ref structure behaves like an atomic_t and for every put call, checks whether the reference has fallen to 0 or not.
rtrs_srv_stats_rdma_to_str shows the count of ids_inflight as 0 for user-mode tools not to be confused.
--- drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 12 +++--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 43 +++++++++++++------- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 4 +- 3 files changed, 35 insertions(+), 24 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c +5 -7
@@ -27,12 +27,10 @@ ssize_t rtrs_srv_stats_rdma_to_str(struct rtrs_srv_stats *stats, char *page, size_t len) {struct rtrs_srv_stats_rdma_stats *r = &stats->rdma_stats; -struct rtrs_srv_sess *sess = stats->sess; -return scnprintf (page, len,"%lld %lld %lld %lld %u\n" , - (s64 )atomic64_read (&r->dir[READ ].cnt), - (s64 )atomic64_read (&r->dir[READ ].size_total), - (s64 )atomic64_read (&r->dir[WRITE ].cnt), - (s64 )atomic64_read (&r->dir[WRITE ].size_total), -atomic_read (&sess->ids_inflight)); +return scnprintf (page, len,"%lld %lld %lld %lldn %u\n" , + (s64 )atomic64_read (&r->dir[READ ].cnt), + (s64 )atomic64_read (&r->dir[READ ].size_total), + (s64 )atomic64_read (&r->dir[WRITE ].cnt), + (s64 )atomic64_read (&r->dir[WRITE ].size_total),0 ); }
drivers/infiniband/ulp/rtrs/rtrs-srv.c +28 -15
83 lines not on this page · raw message
drivers/infiniband/ulp/rtrs/rtrs-srv.h +2 -2
@@ -81,8 +81,8 @@ struct rtrs_srv_sess {spinlock_t state_lock;int cur_cq_vector;struct rtrs_srv_op **ops_ids; -atomic_t ids_inflight; -wait_queue_head_t ids_waitq; +struct percpu_ref ids_inflight_ref; +struct completion complete_done;struct rtrs_srv_mr *mrs;unsigned int mrs_num;dma_addr_t *dma_addr;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 14/20 RDMA/rtrs-srv: convert scnprintf to sysfs_emit
In reply to Jack Wang--- drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c +1 -1
@@ -28,7 +28,7 @@ ssize_t rtrs_srv_stats_rdma_to_str(struct rtrs_srv_stats *stats, {struct rtrs_srv_stats_rdma_stats *r = &stats->rdma_stats; - return scnprintf(page, len, "%lld %lld %lld %lldn %u\n", + return sysfs_emit(page, "%lld %lld %lld %lldn %u\n", (s64)atomic64_read(&r->dir[READ].cnt), (s64)atomic64_read(&r->dir[READ].size_total), (s64)atomic64_read(&r->dir[WRITE].cnt),
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 15/20 RDMA/rtrs: Do not reset hb_missed_max after re-connection
In reply to Jack WangFrom: Gioh Kim <[email protected]>
When re-connecting, it resets hb_missed_max to 0. Before the first re-connecting, client will trigger re-connection when it gets hb-ack more than 5 times. But after the first re-connecting, clients will do re-connection whenever it does not get hb-ack because hb_missed_max is 0.
There is no need to reset hb_missed_max when re-connecting. hb_missed_max should be kept until closing the session.
--- drivers/infiniband/ulp/rtrs/rtrs.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs.c +0 -1
@@ -376,7 +376,6 @@ void rtrs_stop_hb(struct rtrs_sess *sess) {cancel_delayed_work_sync (&sess->hb_dwork); sess->hb_missed_cnt =0 ; - sess->hb_missed_max =0 ; }EXPORT_SYMBOL_GPL (rtrs_stop_hb);
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 16/20 RDMA/rtrs-srv: Duplicated session name is not allowed
In reply to Jack WangFrom: Gioh Kim <[email protected]>
If two clients try to use the same session name, rtrs-server generates a kernel error that it failed to create the sysfs because the filename is duplicated.
This patch adds code to check if there already exists the same session name with the different UUID. If a client tries to add more session, it sends the UUID and the session name. Therefore it is ok if there is already same session name with the same UUID. The rtrs-server must fail only-if there is the same session name with the different UUID.
--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 42 +++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +41 -1
@@ -753,7 +753,40 @@ static void rtrs_srv_sess_down(struct rtrs_srv_sess *sess)mutex_unlock (&srv->paths_ev_mutex); } +static bool exist_sessname (struct rtrs_srv_ctx *ctx, +const char *sessname,const uuid_t *path_uuid) +{ +struct rtrs_srv *srv; +struct rtrs_srv_sess *sess; +bool found = false; + +mutex_lock (&ctx->srv_mutex); +list_for_each_entry (srv, &ctx->srv_list, ctx_list) { +mutex_lock (&srv->paths_mutex); + +/* when a client with same uuid and same sessname tried to add a path */ +if (uuid_equal (&srv->paths_uuid, path_uuid)) { +mutex_unlock (&srv->paths_mutex); +continue ; + } + +list_for_each_entry (sess, &srv->paths_list, s.entry) { +if (strlen (sess->s.sessname) ==strlen (sessname) && + !strcmp (sess->s.sessname, sessname)) { + found = true; +break ; + } + } +mutex_unlock (&srv->paths_mutex); +if (found) +break ; + } +mutex_unlock (&ctx->srv_mutex); +return found; +} +static int post_recv_sess (struct rtrs_srv_sess *sess); +static int rtrs_rdma_do_reject (struct rdma_cm_id *cm_id,int errno);static int process_info_req (struct rtrs_srv_con *con,struct rtrs_msg_info_req *msg)
@@ -772,10 +805,17 @@ static int process_info_req(struct rtrs_srv_con *con,rtrs_err (s,"post_recv_sess(), err: %d\n" , err);return err; } + +if (exist_sessname (sess->srv->ctx, + msg->sessname, &sess->srv->paths_uuid)) { +rtrs_err (s,"sessname is duplicated: %s\n" , msg->sessname); +return -EPERM ; + } +strscpy (sess->s.sessname, msg->sessname,sizeof (sess->s.sessname)); + rwr =kcalloc (sess->mrs_num,sizeof (*rwr),GFP_KERNEL );if (unlikely (!rwr))return -ENOMEM ; -strscpy (sess->s.sessname, msg->sessname,sizeof (sess->s.sessname)); tx_sz =sizeof (*rsp); tx_sz +=sizeof (rsp->desc[0 ]) * sess->mrs_num;
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 17/20 RDMA/rtrs-srv: Fix memory leak of unfreed rtrs_srv_stats object
In reply to Jack WangFrom: Gioh Kim <[email protected]>
When closing a session, currently the rtrs_srv_stats object in the closing session is freed by kobject release. But if it failed to create a session by various reasons, it must free the rtrs_srv_stats object directly because kobject is not created yet.
This problem is found by kmemleak as below:
1. One client machine maps /dev/nullb0 with session name 'bla':
root@test1:~# echo "sessname=bla path=ip:192.168.122.190 \ device_path=/dev/nullb0" > /sys/devices/virtual/rnbd-client/ctl/map_device
2. Another machine failed to create a session with the same name 'bla':
root@test2:~# echo "sessname=bla path=ip:192.168.122.190 \ device_path=/dev/nullb1" > /sys/devices/virtual/rnbd-client/ctl/map_device -bash: echo: write error: Connection reset by peer
3. The kmemleak on server machine reported an error:
unreferenced object 0xffff888033cdc800 (size 128):
comm "kworker/2:1", pid 83, jiffies 4295086585 (age 2508.680s) hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<00000000a72903b2>] __alloc_sess+0x1d4/0x1250 [rtrs_server]
[<00000000d1e5321e>] rtrs_srv_rdma_cm_handler+0xc31/0xde0 [rtrs_server]
[<00000000bb2f6e7e>] cma_ib_req_handler+0xdc5/0x2b50 [rdma_cm]
[<00000000e896235d>] cm_process_work+0x2d/0x100 [ib_cm]
[<00000000b6866c5f>] cm_req_handler+0x11bc/0x1c40 [ib_cm]
[<000000005f5dd9aa>] cm_work_handler+0xe65/0x3cf2 [ib_cm]
[<00000000610151e7>] process_one_work+0x4bc/0x980
[<00000000541e0f77>] worker_thread+0x78/0x5c0
[<00000000423898ca>] kthread+0x191/0x1e0
[<000000005a24b239>] ret_from_fork+0x3a/0x50--- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +1 -0
@@ -1503,6 +1503,7 @@ static void free_sess(struct rtrs_srv_sess *sess)kobject_del (&sess->kobj);kobject_put (&sess->kobj); }else { +kfree (sess->stats);kfree (sess); } }
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 18/20 RDMA/rtrs-srv: Fix memory leak when having multiple sessions
In reply to Jack WangFrom: Jack Wang <[email protected]>
Gioh notice memory leak below
unreferenced object 0xffff8880acda2000 (size 2048):
comm "kworker/4:1", pid 77, jiffies 4295062871 (age 1270.730s) hex dump (first 32 bytes):
00 20 da ac 80 88 ff ff 00 20 da ac 80 88 ff ff . ....... ......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<00000000e85d85b5>] rtrs_srv_rdma_cm_handler+0x8e5/0xa90 [rtrs_server]
[<00000000e31a988a>] cma_ib_req_handler+0xdc5/0x2b50 [rdma_cm]
[<000000000eb02c5b>] cm_process_work+0x2d/0x100 [ib_cm]
[<00000000e1650ca9>] cm_req_handler+0x11bc/0x1c40 [ib_cm]
[<000000009c28818b>] cm_work_handler+0xe65/0x3cf2 [ib_cm]
[<000000002b53eaa1>] process_one_work+0x4bc/0x980
[<00000000da3499fb>] worker_thread+0x78/0x5c0
[<00000000167127a4>] kthread+0x191/0x1e0
[<0000000060802104>] ret_from_fork+0x3a/0x50unreferenced object 0xffff88806d595d90 (size 8):
comm "kworker/4:1H", pid 131, jiffies 4295062972 (age 1269.720s) hex dump (first 8 bytes):
62 6c 61 00 6b 6b 6b a5 bla.kkk.
backtrace:
[<000000004447d253>] kstrdup+0x2e/0x60
[<0000000047259793>] kobject_set_name_vargs+0x2f/0xb0
[<00000000c2ee3bc8>] dev_set_name+0xab/0xe0
[<000000002b6bdfb1>] rtrs_srv_create_sess_files+0x260/0x290 [rtrs_server]
[<0000000075d87bd7>] rtrs_srv_info_req_done+0x71b/0x960 [rtrs_server]
[<00000000ccdf1bb5>] __ib_process_cq+0x94/0x100 [ib_core]
[<00000000cbcb60cb>] ib_cq_poll_work+0x32/0xc0 [ib_core]
[<000000002b53eaa1>] process_one_work+0x4bc/0x980
[<00000000da3499fb>] worker_thread+0x78/0x5c0
[<00000000167127a4>] kthread+0x191/0x1e0
[<0000000060802104>] ret_from_fork+0x3a/0x50
unreferenced object 0xffff88806d6bb100 (size 256):
comm "kworker/4:1H", pid 131, jiffies 4295062972 (age 1269.720s)
hex dump (first 32 bytes):
00 00 00 00 ad 4e ad de ff ff ff ff 00 00 00 00 .....N..........
ff ff ff ff ff ff ff ff 00 59 4d 86 ff ff ff ff .........YM.....
backtrace:
[<00000000a18a11e4>] device_add+0x74d/0xa00
[<00000000a915b95f>] rtrs_srv_create_sess_files.cold+0x49/0x1fe [rtrs_server]
[<0000000075d87bd7>] rtrs_srv_info_req_done+0x71b/0x960 [rtrs_server]
[<00000000ccdf1bb5>] __ib_process_cq+0x94/0x100 [ib_core]
[<00000000cbcb60cb>] ib_cq_poll_work+0x32/0xc0 [ib_core]
[<000000002b53eaa1>] process_one_work+0x4bc/0x980
[<00000000da3499fb>] worker_thread+0x78/0x5c0
[<00000000167127a4>] kthread+0x191/0x1e0
[<0000000060802104>] ret_from_fork+0x3a/0x50The problem is we increase device refcount by get_device in process_info_req for each path, but only does put_deice for last path, which lead to memory leak.
To fix it, it also calls put_device when dev_ref is not 0.
--- drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c +1 -0
@@ -208,6 +208,7 @@ rtrs_srv_destroy_once_sysfs_root_folders(struct rtrs_srv_sess *sess)device_del (&srv->dev);put_device (&srv->dev); }else { +put_device (&srv->dev);mutex_unlock (&srv->paths_mutex); } }
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 19/20 RDMA/rtrs-clt: Check if the queue_depth has changed during a reconnection
In reply to Jack WangFrom: Md Haris Iqbal <[email protected]>
The queue_depth is a module parameter for rtrs_server. It is used on the client side to determing the queue_depth of the request queue for the RNBD virtual block device.
During a reconnection event for an already mapped device, in case the rtrs_server module queue_depth has changed, fail the reconnect attempt.
Also stop further auto reconnection attempts. A manual reconnect via sysfs has to be triggerred.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +15 -4
@@ -1780,7 +1780,19 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con, queue_depth);return -ECONNRESET ; } -if (!sess->rbufs || sess->queue_depth < queue_depth) { +if (sess->queue_depth >0 && queue_depth != sess->queue_depth) { +rtrs_err (clt,"Error: queue depth changed\n" ); + +/* +* Stop any more reconnection attempts +*/ + sess->reconnect_attempts =-1 ; +rtrs_err (clt, +"Disabling auto-reconnect. Trigger a manual reconnect after issue is resolved\n" ); +return -ECONNRESET ; + } + +if (!sess->rbufs) {kfree (sess->rbufs); sess->rbufs =kcalloc (queue_depth,sizeof (*sess->rbufs),GFP_KERNEL );
@@ -1794,7 +1806,7 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con, sess->chunk_size = sess->max_io_size + sess->max_hdr_size;/* -* Global queue depth and IO size is always a minimum. +* Global IO size is always a minimum. * If while a reconnection server sends us a value a bit * higher - client does not care and uses cached minimum. *
@@ -1802,8 +1814,7 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con,* connections in parallel, use lock. */ mutex_lock (&clt->paths_mutex); - clt->queue_depth =min_not_zero (sess->queue_depth, - clt->queue_depth); + clt->queue_depth = sess->queue_depth; clt->max_io_size =min_not_zero (sess->max_io_size, clt->max_io_size);mutex_unlock (&clt->paths_mutex);
Signature · 1 line
2.25.1
Jack Wang
maintainerPATCH v3 20/20 RDMA/rtrs-clt: Fix memory leak of not-freed sess->stats and stats->pcpu_stats
In reply to Jack WangFrom: Gioh Kim <[email protected]>
sess->stats and sess->stats->pcpu_stats objects are freed when sysfs entry is removed. If something wrong happens and session is closed before sysfs entry is created, sess->stats and sess->stats->pcpu_stats objects are not freed.
This patch adds freeing of them at three places:
1. When client uses wrong address and session creation fails.
2. When client fails to create a sysfs entry.
3. When client adds wrong address via sysfs add_path.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +6 -0
@@ -2765,6 +2765,8 @@ struct rtrs_clt *rtrs_clt_open(struct rtrs_clt_ops *ops,if (err) {list_del_rcu (&sess->s.entry);rtrs_clt_close_conns (sess, true); +free_percpu (sess->stats->pcpu_stats); +kfree (sess->stats);free_sess (sess); goto close_all_sess; }
@@ -2773,6 +2775,8 @@ struct rtrs_clt *rtrs_clt_open(struct rtrs_clt_ops *ops,if (err) {list_del_rcu (&sess->s.entry);rtrs_clt_close_conns (sess, true); +free_percpu (sess->stats->pcpu_stats); +kfree (sess->stats);free_sess (sess); goto close_all_sess; }
@@ -3048,6 +3052,8 @@ int rtrs_clt_create_path_from_sysfs(struct rtrs_clt *clt, close_sess:rtrs_clt_remove_path_from_arr (sess);rtrs_clt_close_conns (sess, true); +free_percpu (sess->stats->pcpu_stats); +kfree (sess->stats);free_sess (sess);return err;
Signature · 1 line
2.25.1
Quoted 48 lines · Jack Wang
Hi Jason, hi Doug,
Please consider to include following changes to the next merge window. It contains:
- Patch 01 ~ 10: Typical code refactoring patches
- Patch 11: Requested by Jason
https://www.spinics.net/lists/linux-rdma/msg102009.html
- Patch 12 RDMA/rtrs-srv: Replaceatomic_twithpercpu_refforids_inflight, we split thesysfs_emitconversion to a seperate patch 13.
- Patch 14 ~ 20: Bug fixes
V3->V2:
- ratelimit error message forfirst_concheck. (Jason)
- split thesysfs_emitconversion to a new patch 13.
V2->V1:
- drop one patch "RDMA/rtrs-clt: No need to checkqueue_depthwhen receiving" as requested by Leon
- (void) casting will be removed by next patch set
as requested by Leon.
https://www.spinics.net/lists/linux-rdma/msg102200.htmlDima Stepanov (1):
RDMA/rtrs: Use strscpy instead of strlcpyGioh Kim (7):
RDMA/rtrs-clt: RemoveMAX_SESS_QUEUE_DEPTHfromrtrs_send_sess_infoRDMA/rtrs: ChangeMAX_SESS_QUEUE_DEPTH
RDMA/rtrs: DefineMIN_CHUNK_SIZE
RDMA/rtrs: Do not resethb_missed_maxafter re-connection RDMA/rtrs-srv: Duplicated session name is not allowed RDMA/rtrs-srv: Fix memory leak of unfreedrtrs_srv_statsobject RDMA/rtrs-clt: Fix memory leak of not-freedsess->statsandstats->pcpu_statsGuoqing Jiang (6):
RDMA/rtrs-srv: Killreject_w_econnresetlabel RDMA/rtrs-srv: Clean up the code in__rtrs_srv_change_stateRDMA/rtrs-clt: Kill rtrs_clt_{start,stop}_hb RDMA/rtrs-clt: Killrtrs_clt_disconnect_from_sysfsRDMA/rtrs-srv: Kill__rtrs_srv_change_stateRDMA/rtrs-clt: Remove redundant 'break'Jack Wang (2):
RDMA/rtrs-srv: convert scnprintf tosysfs_emitRDMA/rtrs-srv: Fix memory leak when having multiple sessions
Quoted 7 lines
Md Haris Iqbal (4):
RDMA/rtrs-srv: Add error messages for cases when failing RDMA
connection
RDMA/rtrs-clt: Check state of thertrs_clt_sessbefore reading its
stats
RDMA/rtrs-srv: Replaceatomic_twithpercpu_refforids_inflightRDMA/rtrs-clt: Check if thequeue_depthhas changed during a
Applied to for-next, thanks
Jason