Skip to contentSkip to conversation

RDMA/rxe: Fix mr->map double free

PATCH In review Li Zhijian 5 participants 14 messages Oct 30, 2022 – Nov 23, 2022

Subsystems (2)
lore
Share & tools
lore

Li Zhijian

author
#0 lore raw

PATCH v2 1/1 RDMA/rxe: Fix mr->map double free

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

This issue was fistrly exposed since
commit: b18c7da63fcb ("RDMA/rxe: Fix memory leak in error path code") and then we fixed it in
commit: 8ff5f5d9d8cf ("RDMA/rxe: Prevent double freeing rxe_map_set()") but this fix was reverted together at last by
commit: 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")

Fixes 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")
---
 drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index d4f10c2d1aa7..7c99d1591580 100644
@@ -99,6 +99,7 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
 		kfree(mr->map[i]);
 
 	kfree(mr->map);
+	mr->map = NULL;
 err1:
 	return -ENOMEM;
 }
@@ -122,7 +123,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 	int			num_buf;
 	void			*vaddr;
 	int err;
-	int i;
 
 	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
 	if (IS_ERR(umem)) {
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 				pr_warn("%s: Unable to get virtual address\n",
 						__func__);
 				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;
 			}
-
 			buf->addr = (uintptr_t)vaddr;
 			buf->size = PAGE_SIZE;
 			num_buf++;
@@ -182,10 +181,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 
 	return 0;
 
-err_cleanup_map:
-	for (i = 0; i < mr->num_map; i++)
-		kfree(mr->map[i]);
-	kfree(mr->map);
 err_release_umem:
 	ib_umem_release(umem);
 err_out:
Signature · 1 line
1.8.3.1
#1 lore raw

ping...

Hi Yanjun

I hope you could take a look to this bug fix more earlier.

Thanks
Zhijian

Quoted 86 lines · Li Zhijian

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

This issue was fistrly exposed since
commit: b18c7da63fcb ("RDMA/rxe: Fix memory leak in error path code") and then we fixed it in
commit: 8ff5f5d9d8cf ("RDMA/rxe: Prevent double freeing rxe_map_set()") but this fix was reverted together at last by
commit: 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")

Fixes 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")
---
  drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
  1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index d4f10c2d1aa7..7c99d1591580 100644
@@ -99,6 +99,7 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
  		kfree(mr->map[i]);
 
  	kfree(mr->map);
+	mr->map = NULL;
  err1:
  	return -ENOMEM;
  }
@@ -122,7 +123,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  	int			num_buf;
  	void			*vaddr;
  	int err;
-	int i;
 
  	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
  	if (IS_ERR(umem)) {
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  				pr_warn("%s: Unable to get virtual address\n",
  						__func__);
  				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;
  			}
-
  			buf->addr = (uintptr_t)vaddr;
  			buf->size = PAGE_SIZE;
  			num_buf++;
@@ -182,10 +181,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 
  	return 0;
 
-err_cleanup_map:
-	for (i = 0; i < mr->num_map; i++)
-		kfree(mr->map[i]);
-	kfree(mr->map);
  err_release_umem:
  	ib_umem_release(umem);
  err_out:
#2 lore raw
Quoted 1 line · 在 2022/11/12 11:29, [email protected] 写道:

ping...

Hi, Zhijian

The changes are too much. I need time to delve into it.

Zhu Yanjun

Quoted 99 lines

Hi Yanjun

I hope you could take a look to this bug fix more earlier.

Thanks
Zhijian

Quoted 86 lines · Li Zhijian

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

This issue was fistrly exposed since
commit: b18c7da63fcb ("RDMA/rxe: Fix memory leak in error path code") and then we fixed it in
commit: 8ff5f5d9d8cf ("RDMA/rxe: Prevent double freeing rxe_map_set()") but this fix was reverted together at last by
commit: 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")

Fixes 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")
---
   drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
   1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index d4f10c2d1aa7..7c99d1591580 100644
@@ -99,6 +99,7 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
   		kfree(mr->map[i]);
 
   	kfree(mr->map);
+	mr->map = NULL;
   err1:
   	return -ENOMEM;
   }
@@ -122,7 +123,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
   	int			num_buf;
   	void			*vaddr;
   	int err;
-	int i;
 
   	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
   	if (IS_ERR(umem)) {
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
   				pr_warn("%s: Unable to get virtual address\n",
   						__func__);
   				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;
   			}
-
   			buf->addr = (uintptr_t)vaddr;
   			buf->size = PAGE_SIZE;
   			num_buf++;
@@ -182,10 +181,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 
   	return 0;
 
-err_cleanup_map:
-	for (i = 0; i < mr->num_map; i++)
-		kfree(mr->map[i]);
-	kfree(mr->map);
   err_release_umem:
   	ib_umem_release(umem);
   err_out:
#3 lore raw
In reply to Li Zhijian
Quoted 65 lines · Li Zhijian

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

This issue was fistrly exposed since
commit: b18c7da63fcb ("RDMA/rxe: Fix memory leak in error path code") and then we fixed it in
commit: 8ff5f5d9d8cf ("RDMA/rxe: Prevent double freeing rxe_map_set()") but this fix was reverted together at last by
commit: 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")

Fixes 1e75550648da (Revert "RDMA/rxe: Create duplicate mapping tables for FMRs")
---
  drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
  1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
index d4f10c2d1aa7..7c99d1591580 100644
@@ -99,6 +99,7 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
  		kfree(mr->map[i]);
 
  	kfree(mr->map);
+	mr->map = NULL;
  err1:
  	return -ENOMEM;
  }
@@ -122,7 +123,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  	int			num_buf;
  	void			*vaddr;
  	int err;
-	int i;
 
  	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
  	if (IS_ERR(umem)) {

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, · line 166

Quoted 5 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  				pr_warn("%s: Unable to get virtual address\n",
  						__func__);
  				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;

This call trace results from page_address's returning NULL, then goto err_cleanup_map where mr->map[i] and mr->map are freed.

And finally rxe_reg_user_mr gets an error from rxe_mr_init_user, the function rxe_mr_cleanup is called to handle mr to free mr->map[i] and mr->map again.

So mr->map[i] and mr->map are double freed.

As such, this commit is reasonable.

But why page_address will return NULL?

Zhu Yanjun

Quoted 16 lines
drivers/infiniband/sw/rxe/rxe_mr.c +0 -5
  			}
-
  			buf->addr = (uintptr_t)vaddr;
  			buf->size = PAGE_SIZE;
  			num_buf++;
@@ -182,10 +181,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 
  	return 0;
 
-err_cleanup_map:
-	for (i = 0; i < mr->num_map; i++)
-		kfree(mr->map[i]);
-	kfree(mr->map);
  err_release_umem:
  	ib_umem_release(umem);
  err_out:
#4 lore raw
Quoted 34 lines · Yanjun Zhu
Quoted 20 lines

index d4f10c2d1aa7..7c99d1591580 100644

drivers/infiniband/sw/rxe/rxe_mr.c +1 -0
@@ -99,6 +99,7 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
           kfree(mr->map[i]);
       kfree(mr->map);
+    mr->map = NULL;

  err1:
      return -ENOMEM;
  }

drivers/infiniband/sw/rxe/rxe_mr.c +0 -1
@@ -122,7 +123,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 
 start, u64 length, u64 iova,
       int            num_buf;
       void            *vaddr;
       int err;
-    int i;
       umem = ib_umem_get(&rxe->ib_dev, start, length, access);
       if (IS_ERR(umem)) {

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_mr_init_user(struct rxe_dev *rxe, u64 · line 167

Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 
 start, u64 length, u64 iova,
                   pr_warn("%s: Unable to get virtual address\n",
                           __func__);
                   err = -ENOMEM;
-                goto err_cleanup_map;
+                goto err_release_umem;

This call trace results from page_address's returning NULL, then goto err_cleanup_map where mr->map[i] and mr->map are freed.

And finally rxe_reg_user_mr gets an error from rxe_mr_init_user, the function rxe_mr_cleanup is called to handle mr to free mr->map[i] and mr->map again.

Quoted 5 lines

So mr->map[i] and mr->map are double freed.

As such, this commit is reasonable.

But why page_address will return NULL?

ENOMEM? but I don't think we need taking too much care upon the reason.

this patch is most likely porting the reverted back, commit:
8ff5f5d9d8cf ("RDMA/rxe: Prevent double freeing rxe_map_set()")

Actually, the double free can be triggered by below error path too.

149 err = rxe_mr_alloc(mr, num_buf);

150 if (err) {

151 pr_warn("%s: Unable to allocate memory for map\n",

152 __func__);

153 goto err_release_umem;

154 }

where rxe_mr_alloc() freed the memory but don't set 'mr->map = NULL'

Thanks
Zhijian

Quoted 3 lines

Zhu Yanjun

maintainer
#5 lore raw
In reply to Li Zhijian
Quoted 22 lines · Li Zhijian

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
Quoted 8 lines
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

Please dont include timestamps in commit messages

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, · line 168

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -2
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 				pr_warn("%s: Unable to get virtual address\n",
 						__func__);
 				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;
 			}
-

page_address() fails if this is a highmem system and the page hasn't been kmap'd yet. So the right thing to do is to use kmap..

But this looks right, so applied to for-next

Thanks,
Jason

#6 lore raw
Quoted 34 lines · Jason Gunthorpe
Quoted 22 lines · Li Zhijian

rxe_mr_cleanup() which tries to free mr->map again will be called when rxe_mr_init_user() fails.

[43895.939883] CPU: 0 PID: 4917 Comm: rdma_flush_serv Kdump: loaded Not tainted 6.1.0-rc1-roce-flush+ #25
[43895.942341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[43895.945208] Call Trace:
[43895.946130]  <TASK>
[43895.946931]  dump_stack_lvl+0x45/0x5d
[43895.948049]  panic+0x19e/0x349
[43895.949010]  ? panic_print_sys_info.part.0+0x77/0x77
[43895.950356]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[43895.952589]  ? preempt_count_sub+0x14/0xc0
[43895.953809]  end_report.part.0+0x54/0x7c
[43895.954993]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.956406]  kasan_report.cold+0xa/0xf
[43895.957668]  ? rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.959090]  rxe_mr_cleanup+0x9d/0xf0 [rdma_rxe]
[43895.960502]  __rxe_cleanup+0x10a/0x1e0 [rdma_rxe]
[43895.961983]  rxe_reg_user_mr+0xb7/0xd0 [rdma_rxe]
[43895.963456]  ib_uverbs_reg_mr+0x26a/0x480 [ib_uverbs]
[43895.964921]  ? __lock_acquire+0x876/0x31e0
[43895.966182]  ? ib_uverbs_ex_create_wq+0x630/0x630 [ib_uverbs]
Quoted 8 lines
[43895.967739]  ? uverbs_fill_udata+0x1c6/0x330 [ib_uverbs]
[43895.969204]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x1a2/0x250 [ib_uverbs]
[43895.971126]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.973094]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.975096]  ? uverbs_fill_udata+0x25f/0x330 [ib_uverbs]
[43895.976466]  ib_uverbs_cmd_verbs+0x1397/0x15a0 [ib_uverbs]
[43895.977930]  ? ib_uverbs_handler_UVERBS_METHOD_QUERY_CONTEXT+0x1a0/0x1a0 [ib_uverbs]
[43895.979937]  ? uverbs_fill_udata+0x330/0x330 [ib_uverbs]

Please dont include timestamps in commit messages

Quoted 11 lines

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, · line 168

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -2
@@ -163,9 +163,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  				pr_warn("%s: Unable to get virtual address\n",
  						__func__);
  				err = -ENOMEM;
-				goto err_cleanup_map;
+				goto err_release_umem;
  			}
-

page_address() fails if this is a highmem system and the page hasn't been kmap'd yet. So the right thing to do is to use kmap..

Sure.

sgt_append.sgt is allocated in this function ib_umem_get. And the function sg_alloc_append_table_from_pages is called to allocate memory.

147 struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr,
148 size_t size, int access)
149 {
...
230 ret = sg_alloc_append_table_from_pages(

231                         &umem->sgt_append, page_list, pinned, 0,
232                         pinned << PAGE_SHIFT, ib_dma_max_seg_size(device),
233                         npages, GFP_KERNEL);

...

And it seems that it is not highmem.

So page_address will not be NULL?

As such, it is not necessary to test the return vaue of page_address?

If so, can we add a new commit to avoid testing of the return value of page_address?

Zhu Yanjun

Quoted 5 lines

But this looks right, so applied to for-next

Thanks,
Jason

#7 lore raw

PATCH 1/1 RDMA/rxe: sgt_append from ib_umem_get is not highmem

From: Zhu Yanjun <[email protected]>

In ib_umem_get, sgt_append is allocated from the function sg_alloc_append_table_from_pages. And it is not from highmem.

As such, the return value of page_address will not be NULL.

---
 drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index b1423000e4bc..3f33a4cdef24 100644
@@ -119,7 +119,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 	struct ib_umem		*umem;
 	struct sg_page_iter	sg_iter;
 	int			num_buf;
-	void			*vaddr;
 	int err;
 
 	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
@@ -149,6 +148,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 		buf = map[0]->buf;
 
 		for_each_sgtable_page (&umem->sgt_append.sgt, &sg_iter, 0) {
+			void *vaddr;
+
 			if (num_buf >= RXE_BUF_PER_MAP) {
 				map++;
 				buf = map[0]->buf;
@@ -156,16 +157,10 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 			}
 
 			vaddr = page_address(sg_page_iter_page(&sg_iter));
-			if (!vaddr) {
-				rxe_dbg_mr(mr, "Unable to get virtual address\n");
-				err = -ENOMEM;
-				goto err_release_umem;
-			}
 			buf->addr = (uintptr_t)vaddr;
 			buf->size = PAGE_SIZE;
 			num_buf++;
 			buf++;
-
 		}
 	}
 
Signature · 1 line
2.27.0
#8 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

There was a thread that tries to refactor iova_to_vaddr[1][2], where page_address will be drop. if so, your changes will be no longer needed.

[1]https://lore.kernel.org/lkml/[email protected]/T/ [2]https://www.spinics.net/lists/linux-rdma/msg114053.html

Thanks
Zhijian

Quoted 50 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

In ib_umem_get, sgt_append is allocated from the function sg_alloc_append_table_from_pages. And it is not from highmem.

As such, the return value of page_address will not be NULL.

---
  drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
  1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index b1423000e4bc..3f33a4cdef24 100644
@@ -119,7 +119,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  	struct ib_umem		*umem;
  	struct sg_page_iter	sg_iter;
  	int			num_buf;
-	void			*vaddr;
  	int err;
 
  	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
@@ -149,6 +148,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  		buf = map[0]->buf;
 
  		for_each_sgtable_page (&umem->sgt_append.sgt, &sg_iter, 0) {
+			void *vaddr;
+
  			if (num_buf >= RXE_BUF_PER_MAP) {
  				map++;
  				buf = map[0]->buf;
@@ -156,16 +157,10 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
  			}
 
  			vaddr = page_address(sg_page_iter_page(&sg_iter));
-			if (!vaddr) {
-				rxe_dbg_mr(mr, "Unable to get virtual address\n");
-				err = -ENOMEM;
-				goto err_release_umem;
-			}
  			buf->addr = (uintptr_t)vaddr;
  			buf->size = PAGE_SIZE;
  			num_buf++;
  			buf++;
-
  		}
  	}
 
#9 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

Quoted 3 lines · 在 2022/11/19 18:30, [email protected] 写道:

There was a thread that tries to refactor iova_to_vaddr[1][2], where page_address will be drop. if so, your changes will be no longer needed.

My commit is to indicate that your commit is not good enough.

And you should make all the related commits in a patch series.

Zhu Yanjun

Quoted 59 lines

[1]https://lore.kernel.org/lkml/[email protected]/T/ [2]https://www.spinics.net/lists/linux-rdma/msg114053.html

Thanks
Zhijian

Quoted 50 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

In ib_umem_get, sgt_append is allocated from the function sg_alloc_append_table_from_pages. And it is not from highmem.

As such, the return value of page_address will not be NULL.

---
   drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++-------
   1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +2 -7
index b1423000e4bc..3f33a4cdef24 100644
@@ -119,7 +119,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
   	struct ib_umem		*umem;
   	struct sg_page_iter	sg_iter;
   	int			num_buf;
-	void			*vaddr;
   	int err;
 
   	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
@@ -149,6 +148,8 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
   		buf = map[0]->buf;
 
   		for_each_sgtable_page (&umem->sgt_append.sgt, &sg_iter, 0) {
+			void *vaddr;
+
   			if (num_buf >= RXE_BUF_PER_MAP) {
   				map++;
   				buf = map[0]->buf;
@@ -156,16 +157,10 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
   			}
 
   			vaddr = page_address(sg_page_iter_page(&sg_iter));
-			if (!vaddr) {
-				rxe_dbg_mr(mr, "Unable to get virtual address\n");
-				err = -ENOMEM;
-				goto err_release_umem;
-			}
   			buf->addr = (uintptr_t)vaddr;
   			buf->size = PAGE_SIZE;
   			num_buf++;
   			buf++;
-
   		}
   	}
 
#10 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

In reply to Zhu Yanjun
Quoted 4 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

In ib_umem_get, sgt_append is allocated from the function sg_alloc_append_table_from_pages. And it is not from highmem.

You've confused the allocation of the SGL table itself with the page_address called on the struct page * stored inside the SGL table.

Think of the SGL as an array of 'struct page *'

The page_address() can return NULL because the 'struct page *' it contains came from userspace and could very will be highmem.

Jason

#11 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

Quoted 11 lines · Jason Gunthorpe
Quoted 4 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

In ib_umem_get, sgt_append is allocated from the function sg_alloc_append_table_from_pages. And it is not from highmem.

You've confused the allocation of the SGL table itself with the page_address called on the struct page * stored inside the SGL table.

Think of the SGL as an array of 'struct page *'

About "The page_address() can return NULL because the 'struct page *' it
contains came from userspace and could very will be highmem.",

From the function ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr,size_t size, int access), I agree with you that struct page comes from user space.

But from "Understanding the Linux Kernel", third edition - sections "8.1.3. Memory Zones" and "8.1.6. Kernel Mappings of High-Memory Page Frames".

In the process' virtual address space, the user space occupies the first 3GB, and the kernel space the 4th GB of this linear address space.

The first 896MB of the kernel space (not only kernel code, but its data also) is "directly" mapped to the first 896 MB of the physical memory.

The last 128MB part of the virtual kernel space is where are mapped some pieces of the physical "high memory" (> 896MB) : thus it can only map no more than 128MB of "high memory" at a time.

It seems that page_address of these "128MB high memory" will return NULL.

But can user space access these high memory? From "Understanding the Linux Kernel", third edition, it seems that it is in kernel space.

Thanks and Regards,
Zhu Yanjun

Quoted 2 lines

Jason

#12 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

Quoted 2 lines · Yanjun Zhu

But can user space access these high memory? From "Understanding the Linux Kernel", third edition, it seems that it is in kernel space.

Yes, "highmem" is effecitvely the ability to have a struct page * that is not mapped into the kernel address space, but is mapped into a userspace process address sspace.

Jason

#13 lore raw

Re: [for-next PATCH 1/1] RDMA/rxe: sgt_append from ib_umem_get is not highmem

Quoted 7 lines · Jason Gunthorpe
Quoted 2 lines · Yanjun Zhu

But can user space access these high memory? From "Understanding the Linux Kernel", third edition, it seems that it is in kernel space.

Yes, "highmem" is effecitvely the ability to have a struct page * that is not mapped into the kernel address space, but is mapped into a userspace process address sspace.

Got it. I am interested in this "highmem is mapped into a

userspace process address space.".

Would you like to share some documents, links or source code about this with me?

I want to delve into this "highmem is mapped into a

userspace process address space."

Thanks and Regards,

Zhu Yanjun

Quoted 2 lines

Jason