Misc update for RTRS
Subsystems (2)
Md Haris Iqbal
maintainercover Misc update for RTRS
Hi Jason, hi Doug,
Please consider to include following changes to the next merge window.
The patchset is organized as:
- patch1, patch2, patch3 fixes warnings generated from checkpatch
- patch4 updates a comment.
- patch5 In case of error, performs failover earlier & avoids deadlock
Gioh Kim (3):
RDMA/rtrs: fix CHECK:BRACES type warning
RDMA/rtrs-clt: fix CHECK type warnings
RDMA/rtrs-clt: fix CHECK type warnings
Jack Wang (2):
RDMA/rtrs-clt: Update one outdated comment in path_it_deinit RDMA/rtrs-clt: Do stop and failover outside reconnect work.
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 3 +- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 91 ++++++++++---------- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 + drivers/infiniband/ulp/rtrs/rtrs.c | 1 - 4 files changed, 47 insertions(+), 49 deletions(-)
Signature · 1 line
2.25.1
Md Haris Iqbal
maintainerPATCH 1/5 RDMA/rtrs: fix CHECK:BRACES type warning
From: Gioh Kim <[email protected]>
fix "CHECK:BRACES: Blank lines aren't necessary after an open brace '{'" warning from checkpatch.pl
--- drivers/infiniband/ulp/rtrs/rtrs.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs.c +0 -1
@@ -479,7 +479,6 @@ static int rtrs_str_to_sockaddr(const char *addr, size_t len,*/ int sockaddr_to_str (const struct sockaddr *addr,char *buf,size_t len) { -switch (addr->sa_family) {case AF_IB : return scnprintf(buf, len, "gid:%pI6",
Signature · 1 line
2.25.1
Md Haris Iqbal
maintainerPATCH 2/5 RDMA/rtrs-clt: fix CHECK type warnings
In reply to Md Haris IqbalFrom: Gioh Kim <[email protected]>
First this patch removes list_next_or_null_rr_rcu macro to
fix below warnings. That macro is used only twice. CHECK:MACRO_ARG_REUSE: Macro argument reuse 'head' - possible side-effects? CHECK:MACRO_ARG_REUSE: Macro argument reuse 'ptr' - possible side-effects? CHECK:MACRO_ARG_REUSE: Macro argument reuse 'memb' - possible side-effects?
This patch also fixes below warning:
CHECK:SPACING: No space is necessary after a cast
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 49 ++++++++++++-------------- 1 file changed, 22 insertions(+), 27 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +22 -27
@@ -748,26 +748,6 @@ struct path_it {struct rtrs_clt_path *(*next_path)(struct path_it *it); }; -/** -* list_next_or_null_rr_rcu - get next list element in round-robin fashion. -* @head: the head for the list. -* @ptr: the list head to take the next element from. -* @type: the type of the struct this is embedded in. -* @memb: the name of the list_head within the struct. -* -* Next element returned in round-robin fashion, i.e. head will be skipped, -* but if list is observed as empty, NULL will be returned. -* -* This primitive may safely run concurrently with the _rcu list-mutation -* primitives such as list_add_rcu() as long as it's guarded by rcu_read_lock(). -*/ -#define list_next_or_null_rr_rcu (head, ptr, type, memb) \ -({ \ - list_next_or_null_rcu(head, ptr, type, memb) ?: \ - list_next_or_null_rcu(head, READ_ONCE((ptr)->next), \ - type, memb); \ -}) -/** * get_next_path_rr() - Returns path in round-robin fashion. * @it: the path pointer
@@ -797,10 +777,20 @@ static struct rtrs_clt_path *get_next_path_rr(struct path_it *it) path =list_first_or_null_rcu (&clt->paths_list,typeof (*path), s.entry); else - path =list_next_or_null_rr_rcu (&clt->paths_list, - &path->s.entry, -typeof (*path), - s.entry); +/* +* Next element returned in round-robin fashion, i.e. head will be skipped, +* but if list is observed as empty, NULL will be returned. +* +* This primitive may safely run concurrently with the _rcu list-mutation +* primitives such as list_add_rcu() as long as it's guarded by rcu_read_lock(). +*/ + path =list_next_or_null_rcu (&clt->paths_list, + &path->s.entry, +typeof (*path), + s.entry) ?: +list_next_or_null_rcu (&clt->paths_list, +READ_ONCE ((&path->s.entry)->next), +typeof (*path), s.entry);rcu_assign_pointer (*ppcpu_path, path);return path;
@@ -1863,7 +1853,7 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con, } clt_path->queue_depth = queue_depth; clt_path->s.signal_interval =min_not_zero (queue_depth, - (unsigned short )SERVICE_CON_QUEUE_DEPTH ); + (unsigned short )SERVICE_CON_QUEUE_DEPTH ); clt_path->max_hdr_size =le32_to_cpu (msg->max_hdr_size); clt_path->max_io_size =le32_to_cpu (msg->max_io_size); clt_path->flags =le32_to_cpu (msg->flags);
@@ -2268,8 +2258,13 @@ static void rtrs_clt_remove_path_from_arr(struct rtrs_clt_path *clt_path)* removed. If @sess is the last element, then @next is NULL. */ rcu_read_lock (); - next =list_next_or_null_rr_rcu (&clt->paths_list, &clt_path->s.entry, -typeof (*next), s.entry); + next =list_next_or_null_rcu (&clt->paths_list, + &clt_path->s.entry, +typeof (*next), + s.entry) ?: +list_next_or_null_rcu (&clt->paths_list, +READ_ONCE ((&clt_path->s.entry)->next), +typeof (*next), s.entry);rcu_read_unlock ();/*
Signature · 1 line
2.25.1
Jason Gunthorpe
maintainerRe: [PATCH for-next 2/5] RDMA/rtrs-clt: fix CHECK type warnings
Quoted 9 lines · Md Haris Iqbal
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0
-9-/** -* list_next_or_null_rr_rcu - get next list element in round-robin fashion. -* @head: the head for the list. -* @ptr: the list head to take the next element from. -* @type: the type of the struct this is embedded in. -* @memb: the name of the list_head within the struct. -* -* Next element returned in round-robin fashion, i.e. head will be skipped, -* but if list is observed as empty, NULL will be returned.
drivers/infiniband/ulp/rtrs/rtrs-clt.c struct path_it { · line 769
Quoted 10 lines
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0
-10-* -* This primitive may safely run concurrently with the _rcu list-mutation -* primitives such as list_add_rcu() as long as it's guarded by rcu_read_lock(). -*/ -#define list_next_or_null_rr_rcu (head, ptr, type, memb) \ -({ \ - list_next_or_null_rcu(head, ptr, type, memb) ?: \ - list_next_or_null_rcu(head, READ_ONCE((ptr)->next), \ - type, memb); \ -})
Why not put this in a static inline instead of open coding it? Type is always the same for both usages, right?
Jason
Haris Iqbal
maintainerRe: [PATCH for-next 2/5] RDMA/rtrs-clt: fix CHECK type warnings
Quoted 17 lines · Jason Gunthorpe
Quoted 14 lines · Md Haris Iqbal
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0
-14-/** -* list_next_or_null_rr_rcu - get next list element in round-robin fashion. -* @head: the head for the list. -* @ptr: the list head to take the next element from. -* @type: the type of the struct this is embedded in. -* @memb: the name of the list_head within the struct. -* -* Next element returned in round-robin fashion, i.e. head will be skipped, -* but if list is observed as empty, NULL will be returned. -* -* This primitive may safely run concurrently with the _rcu list-mutation -* primitives such as list_add_rcu() as long as it's guarded by rcu_read_lock(). -*/ -#define list_next_or_null_rr_rcu (head, ptr, type, memb) \
Quoted 8 lines
drivers/infiniband/ulp/rtrs/rtrs-clt.c struct path_it { · line 769
Quoted 5 lines
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0
-5-({ \ -list_next_or_null_rcu (head, ptr, type, memb) ?: \ -list_next_or_null_rcu (head,READ_ONCE ((ptr)->next), \ - type, memb); \ -})Why not put this in a static inline instead of open coding it? Type is always the same for both usages, right?
Yes. Makes sense. I will send this change along with the next patchset.
Thanks for the review and comment.
Quoted 2 lines
Jason
Md Haris Iqbal
maintainerPATCH 3/5 RDMA/rtrs-clt: fix CHECK type warnings
In reply to Md Haris IqbalFrom: Gioh Kim <[email protected]>
fix below warning
CHECK:OPEN_ENDED_LINE: Lines should not end with a '('
--- drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c +1 -2
@@ -156,8 +156,7 @@ static DEVICE_ATTR_RW(mpath_policy);static ssize_t add_path_show (struct device *dev,struct device_attribute *attr,char *page) { -return sysfs_emit ( - page, +return sysfs_emit (page,"Usage: echo [<source addr>@]<destination addr> > %s\n\n*addr ::= [ ip:<ipv4|ipv6> | gid:<gid> ]\n" , attr->attr.name); }
Signature · 1 line
2.25.1
Md Haris Iqbal
maintainerPATCH 4/5 RDMA/rtrs-clt: Update one outdated comment in path_it_deinit
In reply to Md Haris IqbalFrom: Jack Wang <[email protected]>
The skip_list is used for both MIN_INFLIGHT and MIN_LATENCY.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +1 -1
@@ -907,7 +907,7 @@ static inline void path_it_deinit(struct path_it *it) {struct list_head *skip, *tmp;/* -* The skip_list is used only for the MIN_INFLIGHT policy. +* The skip_list is used only for the MIN_INFLIGHT and MIN_LATENCY policies. * We need to remove paths from it, so that next IO can insert * paths (->mp_skip_entry) into a skip_list again. */
Signature · 1 line
2.25.1
Md Haris Iqbal
maintainerPATCH 5/5 RDMA/rtrs-clt: Do stop and failover outside reconnect work.
In reply to Md Haris IqbalFrom: Jack Wang <[email protected]>
We can't do instant reconnect, not to DDoS server, but we should stop and failover earlier, so there is less service interruption.
To avoid deadlock, as error_recovery is called from different callback like rdma event or hb error handler, add a new err recovery_work.
--- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 40 ++++++++++++++------------ drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 + 2 files changed, 23 insertions(+), 18 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +22 -18
@@ -297,6 +297,7 @@ static bool rtrs_clt_change_state_from_to(struct rtrs_clt_path *clt_path,return changed; } +static void rtrs_clt_stop_and_destroy_conns (struct rtrs_clt_path *clt_path);static void rtrs_rdma_error_recovery (struct rtrs_clt_con *con) {struct rtrs_clt_path *clt_path =to_clt_path (con->c.path);
@@ -304,16 +305,7 @@ static void rtrs_rdma_error_recovery(struct rtrs_clt_con *con)if (rtrs_clt_change_state_from_to (clt_path,RTRS_CLT_CONNECTED ,RTRS_CLT_RECONNECTING )) { -struct rtrs_clt_sess *clt = clt_path->clt; -unsigned int delay_ms; - -/* -* Normal scenario, reconnect if we were successfully connected -*/ - delay_ms = clt->reconnect_delay_sec *1000 ; -queue_delayed_work (rtrs_wq, &clt_path->reconnect_dwork, -msecs_to_jiffies (delay_ms + -prandom_u32 () %RTRS_RECONNECT_SEED )); +queue_work (rtrs_wq, &clt_path->err_recovery_work); }else {/* * Error can happen just on establishing new connection,
@@ -1501,6 +1493,22 @@ static void rtrs_clt_init_hb(struct rtrs_clt_path *clt_path)static void rtrs_clt_reconnect_work (struct work_struct *work);static void rtrs_clt_close_work (struct work_struct *work); +static void rtrs_clt_err_recovery_work (struct work_struct *work) +{ +struct rtrs_clt_path *clt_path; +struct rtrs_clt_sess *clt; +int delay_ms; + + clt_path = container_of(work, struct rtrs_clt_path, err_recovery_work); + clt = clt_path->clt; + delay_ms = clt->reconnect_delay_sec *1000 ; +rtrs_clt_stop_and_destroy_conns (clt_path); +queue_delayed_work (rtrs_wq, &clt_path->reconnect_dwork, +msecs_to_jiffies (delay_ms + +prandom_u32 () % +RTRS_RECONNECT_SEED )); +} +static struct rtrs_clt_path *alloc_path (struct rtrs_clt_sess *clt,const struct rtrs_addr *path,size_t con_num,u32 nr_poll_queues)
@@ -1552,6 +1560,7 @@ static struct rtrs_clt_path *alloc_path(struct rtrs_clt_sess *clt, clt_path->state =RTRS_CLT_CONNECTING ;atomic_set (&clt_path->connected_cnt,0 );INIT_WORK (&clt_path->close_work, rtrs_clt_close_work); +INIT_WORK (&clt_path->err_recovery_work, rtrs_clt_err_recovery_work);INIT_DELAYED_WORK (&clt_path->reconnect_dwork, rtrs_clt_reconnect_work);rtrs_clt_init_hb (clt_path);
@@ -2321,6 +2330,7 @@ static void rtrs_clt_close_work(struct work_struct *work) clt_path = container_of(work, struct rtrs_clt_path, close_work); +cancel_work_sync (&clt_path->err_recovery_work);cancel_delayed_work_sync (&clt_path->reconnect_dwork);rtrs_clt_stop_and_destroy_conns (clt_path);rtrs_clt_change_state_get_old (clt_path,RTRS_CLT_CLOSED ,NULL );
@@ -2633,7 +2643,6 @@ static void rtrs_clt_reconnect_work(struct work_struct *work) {struct rtrs_clt_path *clt_path;struct rtrs_clt_sess *clt; -unsigned int delay_ms;int err; clt_path = container_of(to_delayed_work(work), struct rtrs_clt_path,
@@ -2650,8 +2659,6 @@ static void rtrs_clt_reconnect_work(struct work_struct *work) } clt_path->reconnect_attempts++; -/* Stop everything */ -rtrs_clt_stop_and_destroy_conns (clt_path);msleep (RTRS_RECONNECT_BACKOFF );if (rtrs_clt_change_state_get_old (clt_path,RTRS_CLT_CONNECTING ,NULL )) { err =init_path (clt_path);
@@ -2664,11 +2671,7 @@ static void rtrs_clt_reconnect_work(struct work_struct *work) reconnect_again:if (rtrs_clt_change_state_get_old (clt_path,RTRS_CLT_RECONNECTING ,NULL )) { clt_path->stats->reconnects.fail_cnt++; - delay_ms = clt->reconnect_delay_sec *1000 ; -queue_delayed_work (rtrs_wq, &clt_path->reconnect_dwork, -msecs_to_jiffies (delay_ms + -prandom_u32 () % -RTRS_RECONNECT_SEED )); +queue_work (rtrs_wq, &clt_path->err_recovery_work); } }
@@ -2900,6 +2903,7 @@ int rtrs_clt_reconnect_from_sysfs(struct rtrs_clt_path *clt_path) &old_state);if (changed) { clt_path->reconnect_attempts =0 ; +rtrs_clt_stop_and_destroy_conns (clt_path);queue_delayed_work (rtrs_wq, &clt_path->reconnect_dwork,0 ); }if (changed || old_state ==RTRS_CLT_RECONNECTING ) {
drivers/infiniband/ulp/rtrs/rtrs-clt.h +1 -0
@@ -134,6 +134,7 @@ struct rtrs_clt_path {struct rtrs_clt_io_req *reqs;struct delayed_work reconnect_dwork;struct work_struct close_work; +struct work_struct err_recovery_work;unsigned int reconnect_attempts;bool established;struct rtrs_rbuf *rbufs;
Signature · 1 line
2.25.1
Haris Iqbal
maintainerQuoted 4 lines · Md Haris Iqbal
Hi Jason, hi Doug,
Please consider to include following changes to the next merge window.
Ping.
Quoted 24 lines
The patchset is organized as:
- patch1, patch2, patch3 fixes warnings generated from checkpatch
- patch4 updates a comment.
- patch5 In case of error, performs failover earlier & avoids deadlockGioh Kim (3):
RDMA/rtrs: fix CHECK:BRACES type warning
RDMA/rtrs-clt: fix CHECK type warnings
RDMA/rtrs-clt: fix CHECK type warningsJack Wang (2):
RDMA/rtrs-clt: Update one outdated comment inpath_it_deinitRDMA/rtrs-clt: Do stop and failover outside reconnect work.drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 3 +- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 91 ++++++++++---------- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 + drivers/infiniband/ulp/rtrs/rtrs.c | 1 - 4 files changed, 47 insertions(+), 49 deletions(-)--
2.25.1
Jason Gunthorpe
maintainerQuoted 7 lines · Haris Iqbal
Quoted 4 lines · Md Haris Iqbal
Hi Jason, hi Doug,
Please consider to include following changes to the next merge window.
Ping.
I don't look at most patches during the merge window.
Quoted 6 lines
Quoted 6 lines
The patchset is organized as:
- patch1, patch2, patch3 fixes warnings generated from checkpatch
- patch4 updates a comment.
- patch5 In case of error, performs failover earlier & avoids deadlock
Quoted 7 lines
Quoted 7 lines
Gioh Kim (3):
RDMA/rtrs: fix CHECK:BRACES type warning
RDMA/rtrs-clt: fix CHECK type warningsJack Wang (2):
RDMA/rtrs-clt: Update one outdated comment inpath_it_deinitRDMA/rtrs-clt: Do stop and failover outside reconnect work.
I took these ones to for-next
Quoted 1 line
Quoted 1 line
RDMA/rtrs-clt: fix CHECK type warnings
This one needs some more thinking
I fixed all the commit messages too
Thanks,
Jason