fix unexpected SMC_CLC_DECL_ERR_REGRMB error

From: "D. Wythe" <[email protected]>

We can easily trigger the SMC_CLC_DECL_ERR_REGRMB exception within following script:

server: smc_run nginx
client: smc_run ./wrk -c 2000 -t 8 -d 20 http://smc-server

And we can clearly see that this error is also divided into two types:

1. 0x09990003
2. 0x05000000/0x09990003

Which has the same root causes, but the immediate causes vary.

The root cause of this issues is that remove connections from link group is not synchronous with add/delete rtoken entry, which means that even the number of connections is less that SMC_RMBS_PER_LGR_MAX, it does not mean that the connection can register rtoken successfully later. In other words, the rtoken entry may released, This will cause an unexpected SMC_CLC_DECL_ERR_REGRMB to be reported, and then this SMC connections have to fallback to TCP.

This patch set handles two types of SMC_CLC_DECL_ERR_REGRMB exceptions from different perspectives.

Patch 1: fix the 0x05000000/0x09990003 error.
Patch 2: fix the 0x09990003 error.

After those patches, there is no SMC_CLC_DECL_ERR_REGRMB exceptions in my
test case any more.

v1 -> v2:
- add bugfix patch for SMC_CLC_DECL_ERR_REGRMB cause by server side v2 -> v3:
- fix incorrect mail thread

D. Wythe (2):
net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error generated by
client
net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error cause by server

 net/smc/smc_core.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

1.8.3.1

From: "D. Wythe" <[email protected]>

The main reason for this unexpected SMC_CLC_DECL_ERR_REGRMB in client dues to following execution sequence:

Server Conn A: Server Conn B: Client Conn B:

smc_lgr_unregister_conn

                        smc_lgr_register_conn
                        smc_clc_send_accept     ->
                                                        smc_rtoken_add
smcr_buf_unuse
		->		Client Conn A:
				smc_rtoken_delete

smc_lgr_unregister_conn() makes current link available to assigned to new incoming connection, while smcr_buf_unuse() has not executed yet, which means that smc_rtoken_add may fail because of insufficient rtoken_entry, reversing their execution order will avoid this problem.

---
 net/smc/smc_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 29525d0..f8c9675 100644
@@ -1161,8 +1161,8 @@ void smc_conn_free(struct smc_connection *conn)
 			cancel_work_sync(&conn->abort_work);
 	}
 	if (!list_empty(&lgr->list)) {
-		smc_lgr_unregister_conn(conn);
 		smc_buf_unuse(conn, lgr); /* allow buffer reuse */
+		smc_lgr_unregister_conn(conn);
 	}
 
 	if (!lgr->conns_num)

1.8.3.1

From: "D. Wythe" <[email protected]>

The problem of SMC_CLC_DECL_ERR_REGRMB on the server is very clear. Based on the fact that whether a new SMC connection can be accepted or not depends on not only the limit of conn nums, but also the available entries of rtoken. Since the rtoken release is trigger by peer, while the conn nums is decrease by local, tons of thing can happen in this time difference.

This only thing that needs to be mentioned is that now all connection creations are completely protected by smc_server_lgr_pending lock, it's enough to check only the available entries in rtokens_used_mask.

---
 net/smc/smc_core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index f8c9675..be7d704 100644
@@ -1864,7 +1864,8 @@ int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
 		    (ini->smcd_version == SMC_V2 ||
 		     lgr->vlan_id == ini->vlan_id) &&
 		    (role == SMC_CLNT || ini->is_smcd ||
-		     lgr->conns_num < SMC_RMBS_PER_LGR_MAX)) {
+		    (lgr->conns_num < SMC_RMBS_PER_LGR_MAX &&
+		      !bitmap_full(lgr->rtokens_used_mask, SMC_RMBS_PER_LGR_MAX)))) {
 			/* link group found */
 			ini->first_contact_local = 0;
 			conn->lgr = lgr;

1.8.3.1