Skip to contentSkip to conversation

RDMA/cm: Read the remote_id_table root under cm.lock

PATCH Applied Changyul Lee 3 participants 4 messages Nov 11 – Nov 16 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/cm: Read the remote_id_table root under cm.lock

cm_find_remote_id() reads the root of cm.remote_id_table before it takes cm.lock. While it waits for the lock, cm_destroy_id() or cm_reset_to_idle() on another CPU can remove that cm_timewait_info from the tree and free it, and the lookup then starts from freed memory. This lookup is done for a REJ with reason IB_CM_REJ_TIMEOUT.

Read the root after taking cm.lock.

Fixes cfa68b0d0440 ("RDMA/cm: Make find_remote_id() return a cm_id_private")

---
Found with a static analysis tool we are developing.

Reproduced in QEMU (x86_64, KASAN) on rdma_rxe over virtio-net, with a test module that connects to a listener on the same device through rdma_cm and destroys each active cm_id while it is in IB_CM_REQ_SENT, so a REJ with IB_CM_REJ_TIMEOUT is sent. Another kthread destroys the passive cm_ids. The window was widened for the test with an msleep(20) between reading the root and taking cm.lock. Without this patch, KASAN reported 46 to 48 slab-use-after-free reads in cm_work_handler() per 20 second run in three runs, on a cm_timewait_info allocated by cm_create_timewait_info() and freed by cm_destroy_id() through cm_reset_to_idle(). With it, there was no KASAN report in three runs.

 drivers/infiniband/core/cm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cm.c +2 -1
index 1a2c2775b14d..5347b7519a8d 100644
@@ -734,11 +734,12 @@ cm_insert_remote_id(struct cm_timewait_info *timewait_info)
 static struct cm_id_private *cm_find_remote_id(__be64 remote_ca_guid,
 					       __be32 remote_id)
 {
-	struct rb_node *node = cm.remote_id_table.rb_node;
+	struct rb_node *node;
 	struct cm_timewait_info *timewait_info;
 	struct cm_id_private *res = NULL;
 
 	spin_lock_irq(&cm.lock);
+	node = cm.remote_id_table.rb_node;
 	while (node) {
 		timewait_info = rb_entry(node, struct cm_timewait_info,
 					 remote_id_node);

base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb

Signature · 1 line
2.43.0
#1 lore raw
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

cm_find_remote_id() reads the root of cm.remote_id_table before it takes cm.lock. While it waits for the lock, cm_destroy_id() or cm_reset_to_idle() on another CPU can remove that cm_timewait_info from the tree and free it, and the lookup then starts from freed memory. This lookup is done for a REJ with reason IB_CM_REJ_TIMEOUT.

Read the root after taking cm.lock.

Fixes cfa68b0d0440 ("RDMA/cm: Make find_remote_id() return a cm_id_private")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Changyul Lee
Quoted 13 lines · Changyul Lee

cm_find_remote_id() reads the root of cm.remote_id_table before it takes cm.lock. While it waits for the lock, cm_destroy_id() or cm_reset_to_idle() on another CPU can remove that cm_timewait_info from the tree and free it, and the lookup then starts from freed memory. This lookup is done for a REJ with reason IB_CM_REJ_TIMEOUT.

Read the root after taking cm.lock.

Fixes cfa68b0d0440 ("RDMA/cm: Make find_remote_id() return a cm_id_private")

---
Found with a static analysis tool we are developing.

Quoted 10 lines

Reproduced in QEMU (x86_64, KASAN) on rdma_rxe over virtio-net, with a test module that connects to a listener on the same device through rdma_cm and destroys each active cm_id while it is in IB_CM_REQ_SENT, so a REJ with IB_CM_REJ_TIMEOUT is sent. Another kthread destroys the passive cm_ids. The window was widened for the test with an msleep(20) between reading the root and taking cm.lock. Without this patch, KASAN reported 46 to 48 slab-use-after-free reads in cm_work_handler() per 20 second run in three runs, on a cm_timewait_info allocated by cm_create_timewait_info() and freed by cm_destroy_id() through cm_reset_to_idle(). With it, there was no KASAN report in three runs.

I'll take this patch, since it's simple enough. However, using a test module that misuses an in-kernel API is not the right approach.

Thanks

Quoted 27 lines
 drivers/infiniband/core/cm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cm.c +2 -1
index 1a2c2775b14d..5347b7519a8d 100644
@@ -734,11 +734,12 @@ cm_insert_remote_id(struct cm_timewait_info *timewait_info)
 static struct cm_id_private *cm_find_remote_id(__be64 remote_ca_guid,
 					       __be32 remote_id)
 {
-	struct rb_node *node = cm.remote_id_table.rb_node;
+	struct rb_node *node;
 	struct cm_timewait_info *timewait_info;
 	struct cm_id_private *res = NULL;
 
 	spin_lock_irq(&cm.lock);
+	node = cm.remote_id_table.rb_node;
 	while (node) {
 		timewait_info = rb_entry(node, struct cm_timewait_info,
 					 remote_id_node);

base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb

Signature · 1 line
2.43.0
#3 lore raw
In reply to Changyul Lee
Quoted 9 lines · Changyul Lee

cm_find_remote_id() reads the root of cm.remote_id_table before it takes cm.lock. While it waits for the lock, cm_destroy_id() or cm_reset_to_idle() on another CPU can remove that cm_timewait_info from the tree and free it, and the lookup then starts from freed memory. This lookup is done for a REJ with reason IB_CM_REJ_TIMEOUT.

Read the root after taking cm.lock.

[...]

Applied, thanks!

[1/1] RDMA/cm: Read the remote_id_table root under cm.lock

      https://git.kernel.org/rdma/rdma/c/a6b0e712b0fccc

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>