Skip to contentSkip to conversation

net/smc: Reset conn->lgr when link group registration fails

PATCH Superseded Wen Gu 3 participants 5 messages Jan 5, 2022 – Jan 6, 2022

Subsystems (4)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Wen Gu

author reviewer
#0 lore raw

PATCH v3 1/1 net/smc: Reset conn->lgr when link group registration fails

SMC connections might fail to be registered to a link group due to things like unable to find a link to assign to in its creation. As a result, connection creation will return a failure and most resources related to the connection won't be applied or initialized, such as conn->abort_work or conn->lnk.

If smc_conn_free() is invoked later, it will try to access the resources related to the connection, which wasn't initialized, thus causing a panic.

Here is an example, a SMC-R connection failed to be registered to a link group and conn->lnk is NULL. The following crash will happen if smc_conn_free() tries to access conn->lnk in smc_cdc_tx_dismiss_slots().

 BUG: kernel NULL pointer dereference, address: 0000000000000168
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page
 PGD 0 P4D 0
 Oops: 0000 [#1] PREEMPT SMP PTI
 CPU: 4 PID: 68 Comm: kworker/4:1 Kdump: loaded Tainted: G E     5.16.0-rc5+ #52
 Workqueue: smc_hs_wq smc_listen_work [smc]
 RIP: 0010:smc_wr_tx_dismiss_slots+0x1e/0xc0 [smc]
 Call Trace:
  <TASK>
  smc_conn_free+0xd8/0x100 [smc]
  smc_lgr_cleanup_early+0x15/0x90 [smc]
  smc_listen_work+0x302/0x1230 [smc]
  ? process_one_work+0x25c/0x600
  process_one_work+0x25c/0x600
  worker_thread+0x4f/0x3a0
  ? process_one_work+0x600/0x600
  kthread+0x15d/0x1a0
  ? set_kthread_struct+0x40/0x40
  ret_from_fork+0x1f/0x30
  </TASK>

This patch tries to fix this by resetting conn->lgr to NULL if an abnormal exit occurs in smc_lgr_register_conn(), thus avoiding the crash caused by accessing the uninitialized resources in smc_conn_free(). And the new created link group will be terminated if smc connections can't be registered to it.

Fixes 56bc3b2094b4 ("net/smc: assign link to a new connection")

---
v1->v2:
- Reset conn->lgr to NULL in smc_lgr_register_conn().
- Only free new created link group.
v2->v3:
- Using __smc_lgr_terminate() instead of smc_lgr_schedule_free_work() for an immediate free.

---
 net/smc/smc_core.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
net/smc/smc_core.c +10 -2
index 412bc85..0201f99 100644
@@ -171,8 +171,10 @@ static int smc_lgr_register_conn(struct smc_connection *conn, bool first)
 
 	if (!conn->lgr->is_smcd) {
 		rc = smcr_lgr_conn_assign_link(conn, first);
-		if (rc)
+		if (rc) {
+			conn->lgr = NULL;
 			return rc;
+		}
 	}
 	/* find a new alert_token_local value not yet used by some connection
 	 * in this link group
@@ -1835,8 +1837,14 @@ int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
 		write_lock_bh(&lgr->conns_lock);
 		rc = smc_lgr_register_conn(conn, true);
 		write_unlock_bh(&lgr->conns_lock);
-		if (rc)
+		if (rc) {
+			spin_lock_bh(lgr_lock);
+			if (!list_empty(&lgr->list))
+				list_del_init(&lgr->list);
+			spin_unlock_bh(lgr_lock);
+			__smc_lgr_terminate(lgr, true);
 			goto out;
+		}
 	}
 	conn->local_tx_ctrl.common.type = SMC_CDC_MSG_TYPE;
 	conn->local_tx_ctrl.len = SMC_WR_TX_SIZE;
Signature · 1 line
1.8.3.1
maintainer
#1 lore raw
Quoted 73 lines · Wen Gu

SMC connections might fail to be registered to a link group due to things like unable to find a link to assign to in its creation. As a result, connection creation will return a failure and most resources related to the connection won't be applied or initialized, such as conn->abort_work or conn->lnk.

If smc_conn_free() is invoked later, it will try to access the resources related to the connection, which wasn't initialized, thus causing a panic.

Here is an example, a SMC-R connection failed to be registered to a link group and conn->lnk is NULL. The following crash will happen if smc_conn_free() tries to access conn->lnk in smc_cdc_tx_dismiss_slots().

BUG: kernel NULL pointer dereference, address: 0000000000000168
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 4 PID: 68 Comm: kworker/4:1 Kdump: loaded Tainted: G E     5.16.0-rc5+ #52
Workqueue: smc_hs_wq smc_listen_work [smc]
RIP: 0010:smc_wr_tx_dismiss_slots+0x1e/0xc0 [smc]
Call Trace:
 <TASK>
 smc_conn_free+0xd8/0x100 [smc]
 smc_lgr_cleanup_early+0x15/0x90 [smc]
 smc_listen_work+0x302/0x1230 [smc]
 ? process_one_work+0x25c/0x600
 process_one_work+0x25c/0x600
 worker_thread+0x4f/0x3a0
 ? process_one_work+0x600/0x600
 kthread+0x15d/0x1a0
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x1f/0x30
 </TASK>

This patch tries to fix this by resetting conn->lgr to NULL if an abnormal exit occurs in smc_lgr_register_conn(), thus avoiding the crash caused by accessing the uninitialized resources in smc_conn_free(). And the new created link group will be terminated if smc connections can't be registered to it.

Fixes 56bc3b2094b4 ("net/smc: assign link to a new connection")

---
v1->v2:
- Reset conn->lgr to NULL in smc_lgr_register_conn().
- Only free new created link group.
v2->v3:
- Using __smc_lgr_terminate() instead of smc_lgr_schedule_free_work() for an immediate free.
---
net/smc/smc_core.c | 12 ++++++++++--

1 file changed, 10 insertions(+), 2 deletions(-)
net/smc/smc_core.c +3 -1
index 412bc85..0201f99 100644
@@ -171,8 +171,10 @@ static int smc_lgr_register_conn(struct smc_connection *conn, bool first)
 
 	if (!conn->lgr->is_smcd) {
 		rc = smcr_lgr_conn_assign_link(conn, first);
-		if (rc)
+		if (rc) {
+			conn->lgr = NULL;
 			return rc;
+		}
 	}
 	/* find a new alert_token_local value not yet used by some connection
 	 * in this link group

net/smc/smc_core.c int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini) · line 1845

Quoted 10 lines
net/smc/smc_core.c +6 -1
@@ -1835,8 +1837,14 @@ int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
 		write_lock_bh(&lgr->conns_lock);
 		rc = smc_lgr_register_conn(conn, true);
 		write_unlock_bh(&lgr->conns_lock);
-		if (rc)
+		if (rc) {
+			spin_lock_bh(lgr_lock);
+			if (!list_empty(&lgr->list))
+				list_del_init(&lgr->list);
+			spin_unlock_bh(lgr_lock);
+			__smc_lgr_terminate(lgr, true);

What about adding a smc_lgr_terminate() wrapper and put list_del_init() and __smc_lgr_terminate() into it ?

Quoted 7 lines
net/smc/smc_core.c +1 -0
 			goto out;
+		}
 	}
 	conn->local_tx_ctrl.common.type = SMC_CDC_MSG_TYPE;
 	conn->local_tx_ctrl.len = SMC_WR_TX_SIZE;
Signature · 1 line
1.8.3.1
author reviewer
#2 lore raw

Thanks for your suggestion.

Quoted 10 lines · dust.li

net/smc/smc_core.c int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini) · line 1845

Quoted 7 lines
net/smc/smc_core.c +6 -1
-		if (rc)
+		if (rc) {
+			spin_lock_bh(lgr_lock);
+			if (!list_empty(&lgr->list))
+				list_del_init(&lgr->list);
+			spin_unlock_bh(lgr_lock);
+			__smc_lgr_terminate(lgr, true);

What about adding a smc_lgr_terminate() wrapper and put list_del_init() and __smc_lgr_terminate() into it ?

Adding a new wrapper is a good idea. But I think the logic here is relatively simple. So instead of wrapping them, I coded them like what smc_lgr_cleanup_early() does.

Thanks,
Wen Gu

Quoted 8 lines
Quoted 8 lines
			goto out;
(quoted diff) +1 -0
+		}
	}
	conn->local_tx_ctrl.common.type = SMC_CDC_MSG_TYPE;
	conn->local_tx_ctrl.len = SMC_WR_TX_SIZE;
Signature · 1 line
1.8.3.1
#3 lore raw
Quoted 10 lines · Wen Gu
Quoted 8 lines · dust.li
Quoted 7 lines

-        if (rc)
+        if (rc) {
+            spin_lock_bh(lgr_lock);
+            if (!list_empty(&lgr->list))
+                list_del_init(&lgr->list);
+            spin_unlock_bh(lgr_lock);
+            __smc_lgr_terminate(lgr, true);

Quoted 5 lines
Quoted 2 lines

What about adding a smc_lgr_terminate() wrapper and put list_del_init() and __smc_lgr_terminate() into it ?

Adding a new wrapper is a good idea. But I think the logic here is relatively simple. So instead of wrapping them, I coded them like what smc_lgr_cleanup_early() does.

It might look cleaner with the following changes:
- adopt smc_lgr_cleanup_early() to take only an lgr as parameter and remove the call to smc_conn_free()
- change smc_conn_abort() (which is the only caller of smc_lgr_cleanup_early() right now), always call smc_conn_free() and if (local_first) additionally call smc_lgr_cleanup_early() (hold a local copy of the lgr for this call)
- finally call smc_lgr_cleanup_early(lgr) from smc_conn_create()

This should be the same processing, but the smc_conn_free() is moved to smc_conn_abort() where it looks to be a better place for this call. And smc_lgr_cleanup_early() takes only care of an lgr.

What do you think? Did I miss something?

author reviewer
#4 lore raw

Thanks for your suggestion.

Quoted 10 lines · Karsten Graul

It might look cleaner with the following changes:
- adopt smc_lgr_cleanup_early() to take only an lgr as parameter and remove the call to smc_conn_free()
- change smc_conn_abort() (which is the only caller of smc_lgr_cleanup_early() right now), always call smc_conn_free() and if (local_first) additionally call smc_lgr_cleanup_early() (hold a local copy of the lgr for this call)
- finally call smc_lgr_cleanup_early(lgr) from smc_conn_create()

This should be the same processing, but the smc_conn_free() is moved to smc_conn_abort() where it looks to be a better place for this call. And smc_lgr_cleanup_early() takes only care of an lgr.

I think those are very good changes, making smc_lgr_cleanup_early() processing link group only and more reusable.

Quoted 1 line

What do you think? Did I miss something?

I think it is better and complete. I will improve the patch and test it, then send a v4.

Thanks,
Wen Gu