Skip to contentSkip to conversation

net/smc: buffer allocation and registration improvements

PATCH In review D. Wythe 6 participants 30 messages Jan 23 – Feb 3

Subsystems (7)
lore
Share & tools
lore

D. Wythe

author maintainer
#0 lore raw

cover net/smc: buffer allocation and registration improvements

This series improves SMC-R buffer management by refining the allocation logic and reducing hardware resource overhead during registration.

The primary improvement is the significant reduction in MTTE consumption. By aligning IB registration with the actual physical block sizes, we can reduce the entry count from one per 4KB page to just one per contiguous block. This is especially beneficial for large buffers, preventing hardware resource exhaustion on RDMA NICs.

D. Wythe (3):
net/smc: cap allocation order for SMC-R physically contiguous buffers
mm: vmalloc: export find_vm_area()
net/smc: optimize MTTE consumption for SMC-R buffers

 mm/vmalloc.c       |  1 +
 net/smc/smc_core.c | 31 ++++++++++++++++++-------------
 net/smc/smc_ib.c   | 23 ++++++++++++++++++++---
 3 files changed, 39 insertions(+), 16 deletions(-)
Signature · 1 line
2.45.0
author maintainer
#4 lore raw

PATCH 2/3 mm: vmalloc: export find_vm_area()

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +1 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 
 /**
  * remove_vm_area - find and remove a continuous kernel virtual area
Signature · 1 line
2.45.0
author maintainer
#27 lore raw

PATCH 3/3 net/smc: optimize MTTE consumption for SMC-R buffers

In reply to D. Wythe

SMC-R buffers currently use 4KB page mapping for IB registration. Each page consumes one MTTE, which is inefficient and quickly depletes limited IB hardware resources for large buffers.

For virtual contiguous buffer, switch to vmalloc_huge() to leverage huge page support. By using larger page sizes during IB MR registration, we can drastically reduce MTTE consumption.

For physically contiguous buffer, the entire buffer now requires only one single MTTE.

---
 net/smc/smc_core.c |  3 ++-
 net/smc/smc_ib.c   | 23 ++++++++++++++++++++---
 2 files changed, 22 insertions(+), 4 deletions(-)
net/smc/smc_core.c +2 -1
index 6219db498976..8aca5dc54be7 100644
@@ -2348,7 +2348,8 @@ static struct smc_buf_desc *smcr_new_buf_create(struct smc_link_group *lgr,
 			goto out;
 		fallthrough;	// try virtually contiguous buf
 	case SMCR_VIRT_CONT_BUFS:
-		buf_desc->cpu_addr = vzalloc(PAGE_SIZE << buf_desc->order);
+		buf_desc->cpu_addr = vmalloc_huge(PAGE_SIZE << buf_desc->order,
+						  GFP_KERNEL | __GFP_ZERO);
 		if (!buf_desc->cpu_addr)
 			goto out;
 		buf_desc->pages = NULL;
net/smc/smc_ib.c +20 -3
index 1154907c5c05..67211d44a1db 100644
@@ -20,6 +20,7 @@
 #include <linux/wait.h>
 #include <linux/mutex.h>
 #include <linux/inetdevice.h>
+#include <linux/vmalloc.h>
 #include <rdma/ib_verbs.h>
 #include <rdma/ib_cache.h>
 
@@ -697,6 +698,18 @@ void smc_ib_put_memory_region(struct ib_mr *mr)
 	ib_dereg_mr(mr);
 }
 
+static inline int smc_buf_get_vm_page_order(struct smc_buf_desc *buf_slot)
+{
+#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
+	struct vm_struct *vm;
+
+	vm = find_vm_area(buf_slot->cpu_addr);
+	return vm ? vm->page_order : 0;
+#else
+	return 0;
+#endif
+}
+
 static int smc_ib_map_mr_sg(struct smc_buf_desc *buf_slot, u8 link_idx)
 {
 	unsigned int offset = 0;
@@ -706,8 +719,9 @@ static int smc_ib_map_mr_sg(struct smc_buf_desc *buf_slot, u8 link_idx)
 	sg_num = ib_map_mr_sg(buf_slot->mr[link_idx],
 			      buf_slot->sgt[link_idx].sgl,
 			      buf_slot->sgt[link_idx].orig_nents,
-			      &offset, PAGE_SIZE);
-
+			      &offset,
+			      buf_slot->is_vm ? PAGE_SIZE << smc_buf_get_vm_page_order(buf_slot) :
+			      PAGE_SIZE << buf_slot->order);
 	return sg_num;
 }
 
@@ -719,7 +733,10 @@ int smc_ib_get_memory_region(struct ib_pd *pd, int access_flags,
 		return 0; /* already done */
 
 	buf_slot->mr[link_idx] =
-		ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG, 1 << buf_slot->order);
+		ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+			    buf_slot->is_vm ?
+			    1 << (buf_slot->order - smc_buf_get_vm_page_order(buf_slot)) : 1);
+
 	if (IS_ERR(buf_slot->mr[link_idx])) {
 		int rc;
 
Signature · 1 line
2.45.0
author maintainer
#1 lore raw

PATCH 1/3 net/smc: cap allocation order for SMC-R physically contiguous buffers

In reply to D. Wythe

The alloc_page() cannot satisfy requests exceeding MAX_PAGE_ORDER, and attempting such allocations will lead to guaranteed failures and potential kernel warnings.

For SMCR_PHYS_CONT_BUFS, the allocation order is now capped to MAX_PAGE_ORDER, ensures the attempts to allocate the largest possible physically contiguous chunk instead of failing with an invalid order, which also avoid redundant "try-fail-degrade" cycles in __smc_buf_create().

For SMCR_MIXED_BUFS, If it's order exceeds MAX_PAGE_ORDER, skips the doomed physical allocation attempt and fallback to virtual memory immediately.

---
 net/smc/smc_core.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
net/smc/smc_core.c +16 -12
index e4eabc83719e..6219db498976 100644
@@ -2324,26 +2324,30 @@ static struct smc_buf_desc *smcr_new_buf_create(struct smc_link_group *lgr,
 	if (!buf_desc)
 		return ERR_PTR(-ENOMEM);
 
+	buf_desc->order = get_order(bufsize);
+
 	switch (lgr->buf_type) {
 	case SMCR_PHYS_CONT_BUFS:
+		buf_desc->order = min(buf_desc->order, MAX_PAGE_ORDER);
+		fallthrough;
 	case SMCR_MIXED_BUFS:
-		buf_desc->order = get_order(bufsize);
-		buf_desc->pages = alloc_pages(GFP_KERNEL | __GFP_NOWARN |
-					      __GFP_NOMEMALLOC | __GFP_COMP |
-					      __GFP_NORETRY | __GFP_ZERO,
-					      buf_desc->order);
-		if (buf_desc->pages) {
-			buf_desc->cpu_addr =
-				(void *)page_address(buf_desc->pages);
-			buf_desc->len = bufsize;
-			buf_desc->is_vm = false;
-			break;
+		if (buf_desc->order <= MAX_PAGE_ORDER) {
+			buf_desc->pages = alloc_pages(GFP_KERNEL | __GFP_NOWARN |
+						      __GFP_NOMEMALLOC | __GFP_COMP |
+						      __GFP_NORETRY | __GFP_ZERO,
+						      buf_desc->order);
+			if (buf_desc->pages) {
+				buf_desc->cpu_addr =
+					(void *)page_address(buf_desc->pages);
+				buf_desc->len = bufsize;
+				buf_desc->is_vm = false;
+				break;
+			}
 		}
 		if (lgr->buf_type == SMCR_PHYS_CONT_BUFS)
 			goto out;
 		fallthrough;	// try virtually contiguous buf
 	case SMCR_VIRT_CONT_BUFS:
-		buf_desc->order = get_order(bufsize);
 		buf_desc->cpu_addr = vzalloc(PAGE_SIZE << buf_desc->order);
 		if (!buf_desc->cpu_addr)
 			goto out;
Signature · 1 line
2.45.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH net-next 1/3] net/smc: cap allocation order for SMC-R physically contiguous buffers

On 23.01.26 09:23, D. Wythe wrote:
Describe your changes in imperative mood.
https://docs.kernel.org/process/submitting-patches.html#describe-your-changes

Quoted 9 lines

For SMCR_PHYS_CONT_BUFS, the allocation order is now capped to MAX_PAGE_ORDER, ensures the attempts to allocate the largest possible physically contiguous chunk instead of failing with an invalid order, which also avoid redundant "try-fail-degrade" cycles in __smc_buf_create().

For SMCR_MIXED_BUFS, If it's order exceeds MAX_PAGE_ORDER, skips the doomed physical allocation attempt and fallback to virtual memory immediately.

Proposal for a version in imperative mood (iiuc):
"
For SMCR_PHYS_CONT_BUFS, cap the allocation order to MAX_PAGE_ORDER. This ensures the attempts to allocate the largest possible physically contiguous chunk succeed, instead of failing with an invalid order. This also avoids redundant "try-fail-degrade" cycles in __smc_buf_create().

For SMCR_MIXED_BUFS, if its order exceeds MAX_PAGE_ORDER, skip the doomed physical allocation attempt and fallback to virtual memory immediately. "

Quoted 2 lines

Other than that: LGTM

#5 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 3 lines · D. Wythe

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

No, they have absolutely no business doing that. This functionality is very intentionally kept private.

#28 lore raw

Re: [PATCH net-next 3/3] net/smc: optimize MTTE consumption for SMC-R buffers

In reply to D. Wythe

net/smc/smc_ib.c void smc_ib_put_memory_region(struct ib_mr *mr) · line 710

Quoted 10 lines · D. Wythe
net/smc/smc_ib.c +10 -0
+static inline int smc_buf_get_vm_page_order(struct smc_buf_desc *buf_slot)
+{
+#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
+	struct vm_struct *vm;
+
+	vm = find_vm_area(buf_slot->cpu_addr);
+	return vm ? vm->page_order : 0;
+#else
+	return 0;
+#endif

You might want to encapsulate this logic in a vmalloc_order or similar helper in vmalloc.c.

maintainer
#6 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 14 lines · D. Wythe

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 5 lines
mm/vmalloc.c +1 -0
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

author maintainer
#3 lore raw

Re: [PATCH net-next 1/3] net/smc: cap allocation order for SMC-R physically contiguous buffers

In reply to Alexandra Winter
Quoted 29 lines · Alexandra Winter

On 23.01.26 09:23, D. Wythe wrote:
Describe your changes in imperative mood.
https://docs.kernel.org/process/submitting-patches.html#describe-your-changes

Quoted 9 lines

For SMCR_PHYS_CONT_BUFS, the allocation order is now capped to MAX_PAGE_ORDER, ensures the attempts to allocate the largest possible physically contiguous chunk instead of failing with an invalid order, which also avoid redundant "try-fail-degrade" cycles in __smc_buf_create().

For SMCR_MIXED_BUFS, If it's order exceeds MAX_PAGE_ORDER, skips the doomed physical allocation attempt and fallback to virtual memory immediately.

Proposal for a version in imperative mood (iiuc):
"
For SMCR_PHYS_CONT_BUFS, cap the allocation order to MAX_PAGE_ORDER. This ensures the attempts to allocate the largest possible physically contiguous chunk succeed, instead of failing with an invalid order. This also avoids redundant "try-fail-degrade" cycles in __smc_buf_create().

For SMCR_MIXED_BUFS, if its order exceeds MAX_PAGE_ORDER, skip the doomed physical allocation attempt and fallback to virtual memory immediately. "

Quoted 5 lines
Quoted 2 lines

Other than that: LGTM

Hi Alexandra,

Thank you for your review and for providing the refined description. I will use your suggested wording for the commit message in V2.

Best regards,
D. Wythe

author maintainer
#29 lore raw

Re: [PATCH net-next 3/3] net/smc: optimize MTTE consumption for SMC-R buffers

In reply to Christoph Hellwig
Quoted 14 lines · Christoph Hellwig

net/smc/smc_ib.c void smc_ib_put_memory_region(struct ib_mr *mr) · line 710

Quoted 10 lines · D. Wythe
net/smc/smc_ib.c +10 -0
+static inline int smc_buf_get_vm_page_order(struct smc_buf_desc *buf_slot)
+{
+#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
+	struct vm_struct *vm;
+
+	vm = find_vm_area(buf_slot->cpu_addr);
+	return vm ? vm->page_order : 0;
+#else
+	return 0;
+#endif

You might want to encapsulate this logic in a vmalloc_order or similar helper in vmalloc.c.

Hi Christoph,

That's a great suggestion. Encapsulating this logic into a helper like vmalloc_page_order() (or similar) within vmalloc.c is indeed much cleaner than exporting find_vm_area().

I'll implement this helper in V2 and use it in the SMC code. Thanks for pointing this out!

Thanks,
D. Wythe

author maintainer
#7 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to Uladzislau Rezki
Quoted 16 lines · Uladzislau Rezki
Quoted 15 lines · D. Wythe

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Thanks,
D. Wythe

maintainer
#8 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

author maintainer
#9 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

To address this, we intend to use vmalloc_huge(). When it successfully allocates high-order pages, the vmalloc area is backed by a sequence of physically contiguous chunks (e.g., 2MB each). If we know this page_order, we can register these larger physical blocks instead of individual 4KB pages, reducing MTT consumption from 512 entries down to 1 for every 2MB of memory (with page_order == 9).

However, the result of vmalloc_huge() is currently opaque to the caller. We cannot determine whether it successfully allocated huge pages or fell back to 4KB pages based solely on the returned pointer. Therefore, we need a helper function to query the actual page order, enabling SMC-R to adapt its registration logic to the underlying physical layout.

I hope this clarifies our design motivation!

Best regards,
D. Wythe

maintainer
#10 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Hello, D. Wythe!

Quoted 71 lines
Quoted 45 lines
Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

To address this, we intend to use vmalloc_huge(). When it successfully allocates high-order pages, the vmalloc area is backed by a sequence of physically contiguous chunks (e.g., 2MB each). If we know this page_order, we can register these larger physical blocks instead of individual 4KB pages, reducing MTT consumption from 512 entries down to 1 for every 2MB of memory (with page_order == 9).

Quoted 8 lines

However, the result of vmalloc_huge() is currently opaque to the caller. We cannot determine whether it successfully allocated huge pages or fell back to 4KB pages based solely on the returned pointer. Therefore, we need a helper function to query the actual page order, enabling SMC-R to adapt its registration logic to the underlying physical layout.

I hope this clarifies our design motivation!

Appreciate for the explanation. Yes it clarifies an intention.

As for proposed patch above:

- A page_order is available if CONFIG_HAVE_ARCH_HUGE_VMALLOC is defined;
- It makes sense to get a node, grab a spin-lock and find VM, save page_order and release the lock.

You can have a look at the vmalloc_dump_obj(void *object) function. We try-spinlock there whereas you need just spin-lock. But the idea is the same.

--
Uladzislau Rezki

author maintainer
#11 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 89 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 71 lines
Quoted 45 lines
Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

To address this, we intend to use vmalloc_huge(). When it successfully allocates high-order pages, the vmalloc area is backed by a sequence of physically contiguous chunks (e.g., 2MB each). If we know this page_order, we can register these larger physical blocks instead of individual 4KB pages, reducing MTT consumption from 512 entries down to 1 for every 2MB of memory (with page_order == 9).

Quoted 8 lines

However, the result of vmalloc_huge() is currently opaque to the caller. We cannot determine whether it successfully allocated huge pages or fell back to 4KB pages based solely on the returned pointer. Therefore, we need a helper function to query the actual page order, enabling SMC-R to adapt its registration logic to the underlying physical layout.

I hope this clarifies our design motivation!

Appreciate for the explanation. Yes it clarifies an intention.

As for proposed patch above:

- A page_order is available if CONFIG_HAVE_ARCH_HUGE_VMALLOC is defined;
- It makes sense to get a node, grab a spin-lock and find VM, save page_order and release the lock.

Quoted 6 lines

You can have a look at the vmalloc_dump_obj(void *object) function. We try-spinlock there whereas you need just spin-lock. But the idea is the same.

--
Uladzislau Rezki

Hi Uladzislau,

Thanks very much for the detailed guidance, especially on the correct locking pattern. This is extremely helpful.I will follow it and send a v2 patch series with the new helper implemented in mm/vmalloc.c.

Thanks again for your support.

Best regards,
D. Wythe

maintainer
#12 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 97 lines · D. Wythe
Quoted 89 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 71 lines
Quoted 45 lines
Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

To address this, we intend to use vmalloc_huge(). When it successfully allocates high-order pages, the vmalloc area is backed by a sequence of physically contiguous chunks (e.g., 2MB each). If we know this page_order, we can register these larger physical blocks instead of individual 4KB pages, reducing MTT consumption from 512 entries down to 1 for every 2MB of memory (with page_order == 9).

Quoted 8 lines

However, the result of vmalloc_huge() is currently opaque to the caller. We cannot determine whether it successfully allocated huge pages or fell back to 4KB pages based solely on the returned pointer. Therefore, we need a helper function to query the actual page order, enabling SMC-R to adapt its registration logic to the underlying physical layout.

I hope this clarifies our design motivation!

Appreciate for the explanation. Yes it clarifies an intention.

As for proposed patch above:

- A page_order is available if CONFIG_HAVE_ARCH_HUGE_VMALLOC is defined;
- It makes sense to get a node, grab a spin-lock and find VM, save page_order and release the lock.

Quoted 6 lines

You can have a look at the vmalloc_dump_obj(void *object) function. We try-spinlock there whereas you need just spin-lock. But the idea is the same.

--
Uladzislau Rezki

Quoted 8 lines

Hi Uladzislau,

Thanks very much for the detailed guidance, especially on the correct locking pattern. This is extremely helpful.I will follow it and send a v2 patch series with the new helper implemented in mm/vmalloc.c.

Thanks again for your support.

Welcome!

--
Uladzislau Rezki

#13 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Thanks

Quoted 23 lines

To address this, we intend to use vmalloc_huge(). When it successfully allocates high-order pages, the vmalloc area is backed by a sequence of physically contiguous chunks (e.g., 2MB each). If we know this page_order, we can register these larger physical blocks instead of individual 4KB pages, reducing MTT consumption from 512 entries down to 1 for every 2MB of memory (with page_order == 9).

However, the result of vmalloc_huge() is currently opaque to the caller. We cannot determine whether it successfully allocated huge pages or fell back to 4KB pages based solely on the returned pointer. Therefore, we need a helper function to query the actual page order, enabling SMC-R to adapt its registration logic to the underlying physical layout.

I hope this clarifies our design motivation!

Best regards,
D. Wythe

author maintainer
#14 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 68 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

Quoted 5 lines

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

By leveraging vmalloc_huge() and the proposed helper to increase the page_size in ib_map_mr_sg(), each MTTE covers a much larger contiguous physical block. This significantly reduces the total number of entries required to map the same amount of memory, allowing us to serve more connections under the same hardware constraints

D. Wythe

#15 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 6 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

It is not cap, but more efficient use of existing resources.

Quoted 8 lines

By leveraging vmalloc_huge() and the proposed helper to increase the page_size in ib_map_mr_sg(), each MTTE covers a much larger contiguous physical block. This significantly reduces the total number of entries required to map the same amount of memory, allowing us to serve more connections under the same hardware constraints

D. Wythe

author maintainer
#16 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 85 lines · Leon Romanovsky
Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

It is not cap, but more efficient use of existing resources.

Got it. While MRs pool might be more standard practice, but it doesn't address our specific bottleneck. In fact, smc already has its own internal MR reuse; our core issue remains reducing MTTE consumption by increasing the registration granularity to maximize the memory size mapped per MTT entry.

#17 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 92 lines · D. Wythe
Quoted 91 lines · Leon Romanovsky
Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 6 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

Quoted 6 lines
Quoted 1 line

It is not cap, but more efficient use of existing resources.

Got it. While MRs pool might be more standard practice, but it doesn't address our specific bottleneck. In fact, smc already has its own internal MR reuse; our core issue remains reducing MTTE consumption by increasing the registration granularity to maximize the memory size mapped per MTT entry.

And this is something MR pools can handle as well. We are going in circles, so let's summarize.

I see SMC‑R as one of the RDMA ULPs, and it should ideally rely on the existing ULP API used by NVMe, NFS, and others, rather than maintaining its own internal logic.

I also do not know whether vmalloc_page_order() is an appropriate solution; I only want to show that we can probably achieve the same result without introducing a new function.

Thanks

#21 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 3 lines · D. Wythe

By leveraging vmalloc_huge() and the proposed helper to increase the page_size in ib_map_mr_sg(), each MTTE covers a much larger contiguous physical block.

This doesn't seem right, if your goal is to take a vmalloc() pointer and convert it to a MR via a scatterlist and ib_map_mr_sg() then you should be asking for a helper to convert a kernel pointer into a scatterlist.

Even if you do this in a naive way and call the sg_alloc_append_table_from_pages() function it will automatically join physically contiguous ranges together for you.

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

No need to ask the MM for anything other than the list of physicals to build the scatterlist with.

Still, I wouldn't mind seeing a helper to convert a kernel pointer into a scatterlist because I have see that opencoded in a few places, and maybe there are ways to optimize that using more information from the MM - but it should be APIs used only by this helper not exposed to drivers.

Jason

author maintainer
#18 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to Leon Romanovsky
Quoted 95 lines · Leon Romanovsky
Quoted 94 lines · D. Wythe
Quoted 92 lines · Leon Romanovsky
Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 6 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

It is not cap, but more efficient use of existing resources.

Quoted 7 lines
Quoted 4 lines

Got it. While MRs pool might be more standard practice, but it doesn't address our specific bottleneck. In fact, smc already has its own internal MR reuse; our core issue remains reducing MTTE consumption by increasing the registration granularity to maximize the memory size mapped per MTT entry.

And this is something MR pools can handle as well. We are going in circles, so let's summarize.

I believe some points need to be thoroughly clarified here:

Quoted 4 lines

I see SMC‑R as one of the RDMA ULPs, and it should ideally rely on the existing ULP API used by NVMe, NFS, and others, rather than maintaining its own internal logic.

SMC is not opposed to adopting newer RDMA interfaces; in fact, I have already planned a gradual migration to the updated RDMA APIs. We are currently in the process of adapting to ib_cqe, for instance. As long as functionality remains intact, there is no reason to oppose changes that reduce maintenance overhead or provide additional gains, but such a transition takes time.

Quoted 4 lines

I also do not know whether vmalloc_page_order() is an appropriate solution; I only want to show that we can probably achieve the same result without introducing a new function.

Regarding the specific issue under discussion, I believe the newer RDMA APIs you mentioned do not solve my problem, at least for now. My understanding is that regardless of how MRs are pooled, the core requirement is to increase the page_size parameter in ib_map_mr_sg to maximize the physical size mapped per MTTE. From the code I have examined, I see no evidence of these new APIs utilizing values other than 4KB.

Of course, I believe that regardless of whether this issue currently exists, it is something the RDMA community can resolve. However, as I mentioned, adapting to new API takes time. Before a complete transition is achieved, we need to allow for some necessary updates to SMC.

Thanks

author maintainer
#22 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to Jason Gunthorpe
Quoted 21 lines · Jason Gunthorpe
Quoted 3 lines · D. Wythe

By leveraging vmalloc_huge() and the proposed helper to increase the page_size in ib_map_mr_sg(), each MTTE covers a much larger contiguous physical block.

This doesn't seem right, if your goal is to take a vmalloc() pointer and convert it to a MR via a scatterlist and ib_map_mr_sg() then you should be asking for a helper to convert a kernel pointer into a scatterlist.

Even if you do this in a naive way and call the sg_alloc_append_table_from_pages() function it will automatically join physically contiguous ranges together for you.

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

No need to ask the MM for anything other than the list of physicals to build the scatterlist with.

Quoted 7 lines

Still, I wouldn't mind seeing a helper to convert a kernel pointer into a scatterlist because I have see that opencoded in a few places, and maybe there are ways to optimize that using more information from the MM - but it should be APIs used only by this helper not exposed to drivers.

Jason

Hi Jason,

To be honest, I was previously unaware of the sg_alloc_append_table_from_pages() function, although I had indeed considered manually calculating the size of contiguous physical blocks. The reason I proposed the MM helper is that SMC is not driver, it utilizes vmalloc() for memory allocation and is thus in direct contact with the MM; from this perspective, having the MM provide the page_order would be the most straightforward approach.

Given the significant opposition and our plans to transition SMC to newer APIs in the future anyway, I agree that introducing this helper now is less justified.

I will follow your suggestion and update the next version accordingly.

Thanks.

#19 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 131 lines · D. Wythe
Quoted 95 lines · Leon Romanovsky
Quoted 94 lines · D. Wythe
Quoted 92 lines · Leon Romanovsky
Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 6 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

It is not cap, but more efficient use of existing resources.

Quoted 7 lines
Quoted 4 lines

Got it. While MRs pool might be more standard practice, but it doesn't address our specific bottleneck. In fact, smc already has its own internal MR reuse; our core issue remains reducing MTTE consumption by increasing the registration granularity to maximize the memory size mapped per MTT entry.

And this is something MR pools can handle as well. We are going in circles, so let's summarize.

I believe some points need to be thoroughly clarified here:

Quoted 4 lines

I see SMC‑R as one of the RDMA ULPs, and it should ideally rely on the existing ULP API used by NVMe, NFS, and others, rather than maintaining its own internal logic.

SMC is not opposed to adopting newer RDMA interfaces; in fact, I have already planned a gradual migration to the updated RDMA APIs. We are currently in the process of adapting to ib_cqe, for instance. As long as functionality remains intact, there is no reason to oppose changes that reduce maintenance overhead or provide additional gains, but such a transition takes time.

Quoted 4 lines

I also do not know whether vmalloc_page_order() is an appropriate solution; I only want to show that we can probably achieve the same result without introducing a new function.

Regarding the specific issue under discussion, I believe the newer RDMA APIs you mentioned do not solve my problem, at least for now. My understanding is that regardless of how MRs are pooled, the core requirement is to increase the page_size parameter in ib_map_mr_sg to maximize the physical size mapped per MTTE. From the code I have examined, I see no evidence of these new APIs utilizing values other than 4KB.

Quoted 5 lines

Of course, I believe that regardless of whether this issue currently exists, it is something the RDMA community can resolve. However, as I mentioned, adapting to new API takes time. Before a complete transition is achieved, we need to allow for some necessary updates to SMC.

I disagree with that statement.

SMC‑R has a long history of re‑implementing existing RDMA ULP APIs, and not always correctly.

https://lore.kernel.org/netdev/[email protected]/
https://lore.kernel.org/netdev/20241105112313.GE311159@unreal/#t

Thanks

Quoted 3 lines

Thanks

#23 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to D. Wythe
Quoted 2 lines · D. Wythe
Quoted 2 lines

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

I should clarify this is done after DMA mapping the scatterlist. dma mapping can improve the page size.

And maybe the core code should be helping compute the MR's target page size for a scatterlist.. We already have code to do this in umem, and it is a pretty bit tricky considering the IOVA related rules.

Jason

author maintainer
#20 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to Leon Romanovsky
Quoted 138 lines · Leon Romanovsky
Quoted 131 lines · D. Wythe
Quoted 95 lines · Leon Romanovsky
Quoted 94 lines · D. Wythe
Quoted 92 lines · Leon Romanovsky
Quoted 72 lines · D. Wythe
Quoted 71 lines · Leon Romanovsky
Quoted 58 lines · D. Wythe
Quoted 47 lines · Uladzislau Rezki

Hello, D. Wythe!

Quoted 40 lines · Uladzislau Rezki
Quoted 16 lines · D. Wythe
Quoted 15 lines

find_vm_area() provides a way to find the vm_struct associated with a virtual address. Export this symbol to modules so that modularized subsystems can perform lookups on vmalloc addresses.

---
 mm/vmalloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/vmalloc.c +0 -0
index ecbac900c35f..3eb9fe761c34 100644
@@ -3292,6 +3292,7 @@ struct vm_struct *find_vm_area(const void *addr)
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct *find_vm_area(const void *addr) · line 3296

Quoted 4 lines
mm/vmalloc.c +1 -0
 	return va->vm;
 }
+EXPORT_SYMBOL_GPL(find_vm_area);
 

This is internal. We can not just export it.

--
Uladzislau Rezki

Hi Uladzislau,

Thank you for the feedback. I agree that we should avoid exposing internal implementation details like struct vm_struct to external subsystems.

Following Christoph's suggestion, I'm planning to encapsulate the page order lookup into a minimal helper instead:

unsigned int vmalloc_page_order(const void *addr){
	struct vm_struct *vm;
 	vm = find_vm_area(addr);
	return vm ? vm->page_order : 0;
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(vmalloc_page_order);

Does this approach look reasonable to you? It would keep the vm_struct layout private while satisfying the optimization needs of SMC.

Quoted 7 lines

Could you please clarify why you need info about page_order? I have not looked at your second patch.

Thanks!

--
Uladzislau Rezki

Hi Uladzislau,

Quoted 8 lines

This stems from optimizing memory registration in SMC-R. To provide the RDMA hardware with direct access to memory buffers, we must register them with the NIC. During this process, the hardware generates one MTT entry for each physically contiguous block. Since these hardware entries are a finite and scarce resource, and SMC currently defaults to a 4KB registration granularity, a single 2MB buffer consumes 512 entries. In high-concurrency scenarios, this inefficiency quickly exhausts NIC resources and becomes a major bottleneck for system scalability.

I believe this complexity can be avoided by using the RDMA MR pool API, as other ULPs do, for example NVMe.

Quoted 7 lines
Quoted 2 lines

Thanks

Hi Leon,

Am I correct in assuming you are suggesting mr_pool to limit the number of MRs as a way to cap MTTE consumption?

I don't see this a limit, but something that is considered standard practice to reduce MTT consumption.

Quoted 6 lines

However, our goal is to maximize the total registered memory within the MTTE limits rather than to cap it. In SMC-R, each connection occupies a configurable, fixed-size registered buffer; consequently, the more memory we can register, the more concurrent connections we can support.

It is not cap, but more efficient use of existing resources.

Quoted 7 lines
Quoted 4 lines

Got it. While MRs pool might be more standard practice, but it doesn't address our specific bottleneck. In fact, smc already has its own internal MR reuse; our core issue remains reducing MTTE consumption by increasing the registration granularity to maximize the memory size mapped per MTT entry.

And this is something MR pools can handle as well. We are going in circles, so let's summarize.

I believe some points need to be thoroughly clarified here:

Quoted 4 lines

I see SMC‑R as one of the RDMA ULPs, and it should ideally rely on the existing ULP API used by NVMe, NFS, and others, rather than maintaining its own internal logic.

SMC is not opposed to adopting newer RDMA interfaces; in fact, I have already planned a gradual migration to the updated RDMA APIs. We are currently in the process of adapting to ib_cqe, for instance. As long as functionality remains intact, there is no reason to oppose changes that reduce maintenance overhead or provide additional gains, but such a transition takes time.

Quoted 4 lines

I also do not know whether vmalloc_page_order() is an appropriate solution; I only want to show that we can probably achieve the same result without introducing a new function.

Regarding the specific issue under discussion, I believe the newer RDMA APIs you mentioned do not solve my problem, at least for now. My understanding is that regardless of how MRs are pooled, the core requirement is to increase the page_size parameter in ib_map_mr_sg to maximize the physical size mapped per MTTE. From the code I have examined, I see no evidence of these new APIs utilizing values other than 4KB.

Quoted 5 lines

Of course, I believe that regardless of whether this issue currently exists, it is something the RDMA community can resolve. However, as I mentioned, adapting to new API takes time. Before a complete transition is achieved, we need to allow for some necessary updates to SMC.

Quoted 8 lines

I disagree with that statement.

SMC‑R has a long history of re‑implementing existing RDMA ULP APIs, and not always correctly.

https://lore.kernel.org/netdev/[email protected]/
https://lore.kernel.org/netdev/20241105112313.GE311159@unreal/#t

I see that this discussion has moved beyond the technical scope of the patch into historical design critiques. I do not wish to engage in a debate over SMC's history, nor am I responsible for those past decisions.

I will discontinue the conversation here.

Thanks.

author maintainer
#24 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

In reply to Jason Gunthorpe
Quoted 5 lines · Jason Gunthorpe
Quoted 2 lines · D. Wythe
Quoted 2 lines

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

Quoted 7 lines

I should clarify this is done after DMA mapping the scatterlist. dma mapping can improve the page size.

And maybe the core code should be helping compute the MR's target page size for a scatterlist.. We already have code to do this in umem, and it is a pretty bit tricky considering the IOVA related rules.

Hi Jason,

After a deep dive into ib_umem_find_best_pgsz(), I have to say it is much more subtle than it first appears. The IOVA-to-PA relative offset rules, in particular, make it quite easy to get wrong.

While SMC could duplicate this logic, it is certainly not ideal for maintenance. Are there any plans to refactor this into a generic RDMA core helper—for instance, one that can determine the best page size directly from an sg_table or scatterlist?

Best regards,
D. Wythe

#25 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 16 lines · D. Wythe
Quoted 5 lines · Jason Gunthorpe
Quoted 2 lines · D. Wythe
Quoted 2 lines

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

Quoted 7 lines

I should clarify this is done after DMA mapping the scatterlist. dma mapping can improve the page size.

And maybe the core code should be helping compute the MR's target page size for a scatterlist.. We already have code to do this in umem, and it is a pretty bit tricky considering the IOVA related rules.

Hi Jason,

Quoted 8 lines

After a deep dive into ib_umem_find_best_pgsz(), I have to say it is much more subtle than it first appears. The IOVA-to-PA relative offset rules, in particular, make it quite easy to get wrong.

While SMC could duplicate this logic, it is certainly not ideal for maintenance. Are there any plans to refactor this into a generic RDMA core helper—for instance, one that can determine the best page size directly from an sg_table or scatterlist?

I have not heard of anyone touching this.

It looks like there are only two users in the kernel that pass something other than PAGE_SIZE, so it seems nobody has cared about this till now.

With high order folios being more common it seems like something missing.

However, I wonder what the drivers do with the input page size, segmenting a scatterlist is a bit hard and we have helpers for that already too.

It is a bigger project but probably the right thing is to remove the page size input, wrap the scatterlist in a umem and fixup the drivers to use the existing umem support for building mtts, splitting scatterlists into blocks and so on.

The kernel side here has been left alone for a long time..

Jason

author maintainer
#26 lore raw

Re: [PATCH net-next 2/3] mm: vmalloc: export find_vm_area()

Quoted 39 lines · Jason Gunthorpe
Quoted 16 lines · D. Wythe
Quoted 5 lines · Jason Gunthorpe
Quoted 2 lines · D. Wythe
Quoted 2 lines

From there you can check the resulting scatterlist and compute the page_size to pass to ib_map_mr_sg().

Quoted 7 lines

I should clarify this is done after DMA mapping the scatterlist. dma mapping can improve the page size.

And maybe the core code should be helping compute the MR's target page size for a scatterlist.. We already have code to do this in umem, and it is a pretty bit tricky considering the IOVA related rules.

Hi Jason,

Quoted 8 lines

After a deep dive into ib_umem_find_best_pgsz(), I have to say it is much more subtle than it first appears. The IOVA-to-PA relative offset rules, in particular, make it quite easy to get wrong.

While SMC could duplicate this logic, it is certainly not ideal for maintenance. Are there any plans to refactor this into a generic RDMA core helper—for instance, one that can determine the best page size directly from an sg_table or scatterlist?

I have not heard of anyone touching this.

It looks like there are only two users in the kernel that pass something other than PAGE_SIZE, so it seems nobody has cared about this till now.

With high order folios being more common it seems like something missing.

However, I wonder what the drivers do with the input page size, segmenting a scatterlist is a bit hard and we have helpers for that already too.

Quoted 6 lines

It is a bigger project but probably the right thing is to remove the page size input, wrap the scatterlist in a umem and fixup the drivers to use the existing umem support for building mtts, splitting scatterlists into blocks and so on.

The kernel side here has been left alone for a long time..

I am also curious about the original design intent behind requiring the caller to explicitly pass page_size. From what I can see, its primary role is to define the memory size per MTTE, but calculating the optimal value is surprisingly complex.

I completely agree that providing an automatic way to optimize or calculate the best page size should be the responsibility of the drivers or the RDMA core themselves. Handling such low-level hardware-related details in a ULP like SMC feels misplaced.

Since it appears this isn't a high-priority issue for the community at the moment, and a proper fix requires a much larger architectural effort in the RDMA core, I will withdraw this patch series.

I'll keep an eye on the RDMA subsystem's progress and see if a more generic solution emerges in the future.

Thanks,
D. Wythe