Skip to contentSkip to conversation

udmbuf bug fix and some improvements

PATCH Superseded Huan Yang 3 participants 14 messages Aug 22, 2024 – Aug 29, 2024

Subsystems (3)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Huan Yang

author
#0 lore raw

v4 cover udmbuf bug fix and some improvements

This patchset attempts to fix some errors in udmabuf and remove the upin_list structure.

Some of this fix just gather the patches which I upload before.

Patch 1,2,4,5 has passed the udmabuf self-test suite's tests. Suggested by Kasireddy, Vivek <[email protected]> Patch5 modified the unpin function, therefore running the udmabuf self-test program in a loop did not reveal any memory leaks.

Notice: Test item 6 maybe requires running the command:
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Patch1
===
mmap just pre-fault when the first access

test this by look run create udmabuf
size 3GB, then only record the "mmap -> memset"'s time. This average show 1.2s cost

compare to direct map when mmap, this also show 1.2s cost.

environment: archlinux, linux-next, 16G, 4 core.

Patch2
===
This is the same to patch:
https://lore.kernel.org/all/[email protected]/

Patch3
===
fix current implementation of udmabuf's vmap issues, use pfn.

Patch4
===
Change codestyle and fix a potential bug.

Patch5
===
Attempt to remove unpin_list and other related data structures.

Changelog
===
v4 -> v3:
Patch1 fill vma area with pfn only when true access. Patch2 no change
Patch3 select VMAP_PFN
Patch4 use function pin each item.
Patch5 use pinned folio array instead udmabuf_folio struct.
v2 -> v3:
Patch1, avoid use page, instead, use pfn, and use vmf_insert_pfn map suggested-by Kasireddy, Vivek <[email protected]>

Patch2, update acked-by Kasireddy, Vivek <[email protected]> And keep the kvcalloc on the same line.

Patch3, avoid use page, instead, use pfn, then use vmap_pfn map

Patch4, split v2 patch4, single update codestyle to keep review easy.

Patch5, another way to remove udmabuf_folio

---
v1 -> v2:
Patch1, 3 Rectify the improper use of the sg table. suggested-by Christian König <[email protected]>

Patch2 add acked-by Christian K�nig <[email protected]> which marked in v1

Patch4
Modify the data structure to restore the use of pages and correct the misunderstanding of loop conditions such as "pgcnt". make sure pass self test.

remove v1's patch4
v3
https://lore.kernel.org/all/[email protected]/
v2
https://lore.kernel.org/all/[email protected]/
v1
https://lore.kernel.org/all/[email protected]/

Huan Yang (5):
udmabuf: direct map pfn when first page fault
udmabuf: change folios array from kmalloc to kvmalloc
udmabuf: fix vmap_udmabuf error page set
udmabuf: udmabuf_create codestyle cleanup
udmabuf: remove udmabuf_folio

 drivers/dma-buf/Kconfig   |   1 +
 drivers/dma-buf/udmabuf.c | 281 +++++++++++++++++++++++---------------
 2 files changed, 172 insertions(+), 110 deletions(-)

base-commit: 6a7917c89f219f09b1d88d09f376000914a52763

Signature · 1 line
2.45.2
author
#1 lore raw

PATCH v4 1/5 udmabuf: direct map pfn when first page fault

The current udmabuf mmap uses a page fault to populate the vma.

However, the current udmabuf has already obtained and pinned the folio upon completion of the creation.This means that the physical memory has already been acquired, rather than being accessed dynamically.

As a result, the page fault has lost its purpose as a demanding page. Due to the fact that page fault requires trapping into kernel mode and filling in when accessing the corresponding virtual address in mmap, when creating a large size udmabuf, this represents a considerable overhead.

This patch fill vma area with pfn when the first page fault trigger, so, any other access will not enter page fault.

---
 drivers/dma-buf/udmabuf.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
drivers/dma-buf/udmabuf.c +24 -2
index 047c3cd2ceff..0e33d25310ec 100644
@@ -43,7 +43,8 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
 	struct vm_area_struct *vma = vmf->vma;
 	struct udmabuf *ubuf = vma->vm_private_data;
 	pgoff_t pgoff = vmf->pgoff;
-	unsigned long pfn;
+	unsigned long addr, end, pfn;
+	vm_fault_t ret;
 
 	if (pgoff >= ubuf->pagecount)
 		return VM_FAULT_SIGBUS;
@@ -51,7 +52,28 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
 	pfn = folio_pfn(ubuf->folios[pgoff]);
 	pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
 
-	return vmf_insert_pfn(vma, vmf->address, pfn);
+	ret = vmf_insert_pfn(vma, vmf->address, pfn);
+	if (ret & VM_FAULT_ERROR)
+		return ret;
+
+	/* pre fault */
+	pgoff = vma->vm_pgoff;
+	end = vma->vm_end;
+	addr = vma->vm_start;
+
+	for (; addr < end; pgoff++, addr += PAGE_SIZE) {
+		if (addr == vmf->address)
+			continue;
+
+		pfn = folio_pfn(ubuf->folios[pgoff]);
+
+		pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
+
+		if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR)
+			break;
+	}
+
+	return ret;
 }
 
 static const struct vm_operations_struct udmabuf_vm_ops = {
Signature · 1 line
2.45.2
maintainer
#2 lore raw

RE: [PATCH v4 1/5] udmabuf: direct map pfn when first page fault

Hi Huan,

Quoted 44 lines

Subject: [PATCH v4 1/5] udmabuf: direct map pfn when first page fault

The current udmabuf mmap uses a page fault to populate the vma.

However, the current udmabuf has already obtained and pinned the folio upon completion of the creation.This means that the physical memory has already been acquired, rather than being accessed dynamically.

As a result, the page fault has lost its purpose as a demanding page. Due to the fact that page fault requires trapping into kernel mode and filling in when accessing the corresponding virtual address in mmap, when creating a large size udmabuf, this represents a considerable overhead.

This patch fill vma area with pfn when the first page fault trigger, so, any other access will not enter page fault.

---
 drivers/dma-buf/udmabuf.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
drivers/dma-buf/udmabuf.c +2 -1
index 047c3cd2ceff..0e33d25310ec 100644
@@ -43,7 +43,8 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault
 *vmf)
 	struct vm_area_struct *vma = vmf->vma;
 	struct udmabuf *ubuf = vma->vm_private_data;
 	pgoff_t pgoff = vmf->pgoff;
-	unsigned long pfn;
+	unsigned long addr, end, pfn;
+	vm_fault_t ret;
 
 	if (pgoff >= ubuf->pagecount)
drivers/dma-buf/udmabuf.c +0 -1
 		return VM_FAULT_SIGBUS;
@@ -51,7 +52,28 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault
 *vmf)
 	pfn = folio_pfn(ubuf->folios[pgoff]);
 	pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
 
-	return vmf_insert_pfn(vma, vmf->address, pfn);

drivers/dma-buf/udmabuf.c static vm_fault_t udmabuf_vm_fault(struct vm_fault · line 65

Quoted 10 lines
drivers/dma-buf/udmabuf.c +10 -0
+	ret = vmf_insert_pfn(vma, vmf->address, pfn);
+	if (ret & VM_FAULT_ERROR)
+		return ret;
+
+	/* pre fault */
+	pgoff = vma->vm_pgoff;
+	end = vma->vm_end;
+	addr = vma->vm_start;
+
+	for (; addr < end; pgoff++, addr += PAGE_SIZE) {

Although unlikely, I think we should also check for pgoff < ubuf->pagecount.

drivers/dma-buf/udmabuf.c static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) · line 72

Quoted 8 lines
drivers/dma-buf/udmabuf.c +8 -0
+		if (addr == vmf->address)
+			continue;
+
+		pfn = folio_pfn(ubuf->folios[pgoff]);
+
+		pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
+
+		if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR)

Shouldn't you store the return value of vmf_insert_pfn in ret? Otherwise, we'll return success when the above call fails.

Anyway, I am wondering if it is more optimal to just iterate over pages instead of addresses. Something like below:

+ unsigned long nr_pages = vma_pages(vma);
+ unsigned long addr = vma->vm_start;

- if (pgoff >= ubuf->pagecount)
- return VM_FAULT_SIGBUS;
+ WARN_ON(nr_pages != ubuf->pagecount);

-       pfn = folio_pfn(ubuf->folios[pgoff]);
-       pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
+       for (pg = 0; pg < nr_pages && pg < ubuf->pagecount; pg++) {

+ pfn = folio_pfn(ubuf->folios[pg]);
+ pfn += ubuf->offsets[pg] >> PAGE_SHIFT;

- return vmf_insert_pfn(vma, vmf->address, pfn);
+ ret = vmf_insert_pfn(vma, addr, pfn);
+ addr += PAGE_SIZE;
+ }

Thanks,
Vivek

Quoted 9 lines
drivers/dma-buf/udmabuf.c +4 -0
+			break;
+	}
+
+	return ret;
 }
 

static const struct vm_operations_struct udmabuf_vm_ops = {
--
2.45.2

author
#3 lore raw

Re: [PATCH v4 1/5] udmabuf: direct map pfn when first page fault

Quoted 45 lines · Vivek

Hi Huan,

Quoted 43 lines

Subject: [PATCH v4 1/5] udmabuf: direct map pfn when first page fault

The current udmabuf mmap uses a page fault to populate the vma.

However, the current udmabuf has already obtained and pinned the folio upon completion of the creation.This means that the physical memory has already been acquired, rather than being accessed dynamically.

As a result, the page fault has lost its purpose as a demanding page. Due to the fact that page fault requires trapping into kernel mode and filling in when accessing the corresponding virtual address in mmap, when creating a large size udmabuf, this represents a considerable overhead.

This patch fill vma area with pfn when the first page fault trigger, so, any other access will not enter page fault.

---
  drivers/dma-buf/udmabuf.c | 26 ++++++++++++++++++++++++--
  1 file changed, 24 insertions(+), 2 deletions(-)
drivers/dma-buf/udmabuf.c +2 -1
index 047c3cd2ceff..0e33d25310ec 100644
@@ -43,7 +43,8 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault
 *vmf)
  	struct vm_area_struct *vma = vmf->vma;
  	struct udmabuf *ubuf = vma->vm_private_data;
  	pgoff_t pgoff = vmf->pgoff;
-	unsigned long pfn;
+	unsigned long addr, end, pfn;
+	vm_fault_t ret;
 
  	if (pgoff >= ubuf->pagecount)

return VM_FAULT_SIGBUS;

drivers/dma-buf/udmabuf.c +0 -0
@@ -51,7 +52,28 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault
 *vmf)
  	pfn = folio_pfn(ubuf->folios[pgoff]);
  	pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
 
Quoted 12 lines

drivers/dma-buf/udmabuf.c static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) · line 64

Quoted 11 lines
drivers/dma-buf/udmabuf.c +10 -1
-	return vmf_insert_pfn(vma, vmf->address, pfn);
+	ret = vmf_insert_pfn(vma, vmf->address, pfn);
+	if (ret & VM_FAULT_ERROR)
+		return ret;
+
+	/* pre fault */
+	pgoff = vma->vm_pgoff;
+	end = vma->vm_end;
+	addr = vma->vm_start;
+
+	for (; addr < end; pgoff++, addr += PAGE_SIZE) {

Although unlikely, I think we should also check for pgoff < ubuf->pagecount.

Ohh, yes.

Quoted 11 lines

drivers/dma-buf/udmabuf.c static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) · line 72

Quoted 9 lines
drivers/dma-buf/udmabuf.c +8 -0
+		if (addr == vmf->address)
+			continue;
+
+		pfn = folio_pfn(ubuf->folios[pgoff]);
+
+		pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
+
+		if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR)

Shouldn't you store the return value of vmf_insert_pfn in ret? Otherwise, we'll return success when the above call fails.

If anything wrong in there, let it report when it true access and then report. Not in there.

It's pre-fault, 'this' page's fault is already dealed success. No reason to report in it's fault.

Quoted 11 lines

Anyway, I am wondering if it is more optimal to just iterate over pages instead of addresses. Something like below:

(quoted diff) +3 -2
+       unsigned long nr_pages = vma_pages(vma);
+       unsigned long addr = vma->vm_start;
 
-       if (pgoff >= ubuf->pagecount)
-               return VM_FAULT_SIGBUS;
+       WARN_ON(nr_pages != ubuf->pagecount);
 

Review

Quoted 10 lines
(quoted diff) +6 -3
-       pfn = folio_pfn(ubuf->folios[pgoff]);
-       pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
+       for (pg = 0; pg < nr_pages && pg < ubuf->pagecount; pg++) {
+               pfn = folio_pfn(ubuf->folios[pg]);
+               pfn += ubuf->offsets[pg] >> PAGE_SHIFT;
 
-       return vmf_insert_pfn(vma, vmf->address, pfn);
+               ret = vmf_insert_pfn(vma, addr, pfn);
+               addr += PAGE_SIZE;
+ }

Maybe mmap with a offset? just start from 0 not too good.

This seems clear, but is it not good to handle insert pfn error elsewhere?

Quoted 13 lines

Thanks,
Vivek

Quoted 9 lines
drivers/dma-buf/udmabuf.c +4 -0
+			break;
+	}
+
+	return ret;
  }
 

static const struct vm_operations_struct udmabuf_vm_ops = {
--
2.45.2

author
#4 lore raw

PATCH v4 2/5 udmabuf: change folios array from kmalloc to kvmalloc

In reply to Huan Yang

When PAGE_SIZE 4096, MAX_PAGE_ORDER 10, 64bit machine, page_alloc only support 4MB.
If above this, trigger this warn and return NULL.

udmabuf can change size limit, if change it to 3072(3GB), and then alloc 3GB udmabuf, will fail create.

[ 4080.876581] ------------[ cut here ]------------
[ 4080.876843] WARNING: CPU: 3 PID: 2015 at mm/page_alloc.c:4556 __alloc_pages+0x2c8/0x350
[ 4080.878839] RIP: 0010:__alloc_pages+0x2c8/0x350
[ 4080.879470] Call Trace:
[ 4080.879473]  <TASK>
[ 4080.879473]  ? __alloc_pages+0x2c8/0x350
[ 4080.879475]  ? __warn.cold+0x8e/0xe8
[ 4080.880647]  ? __alloc_pages+0x2c8/0x350
[ 4080.880909]  ? report_bug+0xff/0x140
[ 4080.881175]  ? handle_bug+0x3c/0x80
[ 4080.881556]  ? exc_invalid_op+0x17/0x70
[ 4080.881559]  ? asm_exc_invalid_op+0x1a/0x20
[ 4080.882077]  ? udmabuf_create+0x131/0x400

Because MAX_PAGE_ORDER, kmalloc can max alloc 4096 * (1 << 10), 4MB memory, each array entry is pointer(8byte), so can save 524288 pages(2GB).

Further more, costly order(order 3) may not be guaranteed that it can be applied for, due to fragmentation.

This patch change udmabuf array use kvmalloc_array, this can fallback alloc into vmalloc, which can guarantee allocation for any size and does not affect the performance of kmalloc allocations.

---
 drivers/dma-buf/udmabuf.c | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/dma-buf/udmabuf.c +13 -13
index 0e33d25310ec..1bad4576d492 100644
@@ -102,7 +102,7 @@ static int vmap_udmabuf(struct dma_buf *buf, struct iosys_map *map)
 
 	dma_resv_assert_held(buf->resv);
 
-	pages = kmalloc_array(ubuf->pagecount, sizeof(*pages), GFP_KERNEL);
+	pages = kvmalloc_array(ubuf->pagecount, sizeof(*pages), GFP_KERNEL);
 	if (!pages)
 		return -ENOMEM;
 
@@ -110,7 +110,7 @@ static int vmap_udmabuf(struct dma_buf *buf, struct iosys_map *map)
 		pages[pg] = &ubuf->folios[pg]->page;
 
 	vaddr = vm_map_ram(pages, ubuf->pagecount, -1);
-	kfree(pages);
+	kvfree(pages);
 	if (!vaddr)
 		return -EINVAL;
 
@@ -218,8 +218,8 @@ static void release_udmabuf(struct dma_buf *buf)
 		put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
 
 	unpin_all_folios(&ubuf->unpin_list);
-	kfree(ubuf->offsets);
-	kfree(ubuf->folios);
+	kvfree(ubuf->offsets);
+	kvfree(ubuf->folios);
 	kfree(ubuf);
 }
 
@@ -344,14 +344,14 @@ static long udmabuf_create(struct miscdevice *device,
 	if (!ubuf->pagecount)
 		goto err;
 
-	ubuf->folios = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->folios),
-				    GFP_KERNEL);
+	ubuf->folios = kvmalloc_array(ubuf->pagecount, sizeof(*ubuf->folios),
+				      GFP_KERNEL);
 	if (!ubuf->folios) {
 		ret = -ENOMEM;
 		goto err;
 	}
-	ubuf->offsets = kcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
-				GFP_KERNEL);
+	ubuf->offsets = kvcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
+				 GFP_KERNEL);
 	if (!ubuf->offsets) {
 		ret = -ENOMEM;
 		goto err;
@@ -365,7 +365,7 @@ static long udmabuf_create(struct miscdevice *device,
 			goto err;
 
 		pgcnt = list[i].size >> PAGE_SHIFT;
-		folios = kmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
+		folios = kvmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
 		if (!folios) {
 			ret = -ENOMEM;
 			goto err;
@@ -375,7 +375,7 @@ static long udmabuf_create(struct miscdevice *device,
 		ret = memfd_pin_folios(memfd, list[i].offset, end,
 				       folios, pgcnt, &pgoff);
 		if (ret <= 0) {
-			kfree(folios);
+			kvfree(folios);
 			if (!ret)
 				ret = -EINVAL;
 			goto err;
@@ -404,7 +404,7 @@ static long udmabuf_create(struct miscdevice *device,
 			}
 		}
 
-		kfree(folios);
+		kvfree(folios);
 		fput(memfd);
 		memfd = NULL;
 	}
@@ -420,8 +420,8 @@ static long udmabuf_create(struct miscdevice *device,
 	if (memfd)
 		fput(memfd);
 	unpin_all_folios(&ubuf->unpin_list);
-	kfree(ubuf->offsets);
-	kfree(ubuf->folios);
+	kvfree(ubuf->offsets);
+	kvfree(ubuf->folios);
 	kfree(ubuf);
 	return ret;
 }
Signature · 1 line
2.45.2
author
#5 lore raw

PATCH v4 3/5 udmabuf: fix vmap_udmabuf error page set

In reply to Huan Yang

Currently vmap_udmabuf set page's array by each folio. But, ubuf->folios is only contain's the folio's head page.

That mean we repeatedly mapped the folio head page to the vmalloc area.

Due to udmabuf can use hugetlb, if HVO enabled, tail page may not exist, so, we can't use page array to map, instead, use pfn array.

By this, we removed page usage in udmabuf totally.

---
 drivers/dma-buf/Kconfig   |  1 +
 drivers/dma-buf/udmabuf.c | 22 +++++++++++++++-------
 2 files changed, 16 insertions(+), 7 deletions(-)
drivers/dma-buf/Kconfig +1 -0
index b46eb8a552d7..fee04fdb0822 100644
@@ -36,6 +36,7 @@ config UDMABUF
 	depends on DMA_SHARED_BUFFER
 	depends on MEMFD_CREATE || COMPILE_TEST
 	depends on MMU
+	select VMAP_PFN
 	help
 	  A driver to let userspace turn memfd regions into dma-bufs.
 	  Qemu can use this to create host dmabufs for guest framebuffers.
drivers/dma-buf/udmabuf.c +15 -7
index 1bad4576d492..0bbc9df36c0a 100644
@@ -96,21 +96,29 @@ static int mmap_udmabuf(struct dma_buf *buf, struct vm_area_struct *vma)
 static int vmap_udmabuf(struct dma_buf *buf, struct iosys_map *map)
 {
 	struct udmabuf *ubuf = buf->priv;
-	struct page **pages;
+	unsigned long *pfns;
 	void *vaddr;
 	pgoff_t pg;
 
 	dma_resv_assert_held(buf->resv);
 
-	pages = kvmalloc_array(ubuf->pagecount, sizeof(*pages), GFP_KERNEL);
-	if (!pages)
+	/**
+	 * HVO may free tail pages, so just use pfn to map each folio
+	 * into vmalloc area.
+	 */
+	pfns = kvmalloc_array(ubuf->pagecount, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
 		return -ENOMEM;
 
-	for (pg = 0; pg < ubuf->pagecount; pg++)
-		pages[pg] = &ubuf->folios[pg]->page;
+	for (pg = 0; pg < ubuf->pagecount; pg++) {
+		unsigned long pfn = folio_pfn(ubuf->folios[pg]);
 
-	vaddr = vm_map_ram(pages, ubuf->pagecount, -1);
-	kvfree(pages);
+		pfn += ubuf->offsets[pg] >> PAGE_SHIFT;
+		pfns[pg] = pfn;
+	}
+
+	vaddr = vmap_pfn(pfns, ubuf->pagecount, PAGE_KERNEL);
+	kvfree(pfns);
 	if (!vaddr)
 		return -EINVAL;
 
Signature · 1 line
2.45.2
maintainer Acked-by on v4 3/5
#6 lore raw

RE: [PATCH v4 3/5] udmabuf: fix vmap_udmabuf error page set

Hi Huan,

Quoted 61 lines

Subject: [PATCH v4 3/5] udmabuf: fix vmap_udmabuf error page set

Currently vmap_udmabuf set page's array by each folio. But, ubuf->folios is only contain's the folio's head page.

That mean we repeatedly mapped the folio head page to the vmalloc area.

Due to udmabuf can use hugetlb, if HVO enabled, tail page may not exist, so, we can't use page array to map, instead, use pfn array.

By this, we removed page usage in udmabuf totally.

---
 drivers/dma-buf/Kconfig   |  1 +
 drivers/dma-buf/udmabuf.c | 22 +++++++++++++++-------
 2 files changed, 16 insertions(+), 7 deletions(-)
drivers/dma-buf/Kconfig +1 -0
index b46eb8a552d7..fee04fdb0822 100644
@@ -36,6 +36,7 @@ config UDMABUF
 	depends on DMA_SHARED_BUFFER
 	depends on MEMFD_CREATE || COMPILE_TEST
 	depends on MMU
+	select VMAP_PFN
 	help
 	  A driver to let userspace turn memfd regions into dma-bufs.
 	  Qemu can use this to create host dmabufs for guest framebuffers.
drivers/dma-buf/udmabuf.c +8 -5
index 1bad4576d492..0bbc9df36c0a 100644
@@ -96,21 +96,29 @@ static int mmap_udmabuf(struct dma_buf *buf,
 struct vm_area_struct *vma)
 static int vmap_udmabuf(struct dma_buf *buf, struct iosys_map *map)
 {
 	struct udmabuf *ubuf = buf->priv;
-	struct page **pages;
+	unsigned long *pfns;
 	void *vaddr;
 	pgoff_t pg;
 
 	dma_resv_assert_held(buf->resv);
 
-	pages = kvmalloc_array(ubuf->pagecount, sizeof(*pages),
 GFP_KERNEL);
-	if (!pages)
+	/**
+	 * HVO may free tail pages, so just use pfn to map each folio
+	 * into vmalloc area.
+	 */
+	pfns = kvmalloc_array(ubuf->pagecount, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
 		return -ENOMEM;
 
-	for (pg = 0; pg < ubuf->pagecount; pg++)
-		pages[pg] = &ubuf->folios[pg]->page;
+	for (pg = 0; pg < ubuf->pagecount; pg++) {

drivers/dma-buf/udmabuf.c static int mmap_udmabuf(struct dma_buf *buf, · line 123

Quoted 10 lines
drivers/dma-buf/udmabuf.c +7 -2
+		unsigned long pfn = folio_pfn(ubuf->folios[pg]);
 
-	vaddr = vm_map_ram(pages, ubuf->pagecount, -1);
-	kvfree(pages);
+		pfn += ubuf->offsets[pg] >> PAGE_SHIFT;
+		pfns[pg] = pfn;
+	}
+
+	vaddr = vmap_pfn(pfns, ubuf->pagecount, PAGE_KERNEL);
+	kvfree(pfns);
Quoted 5 lines
drivers/dma-buf/udmabuf.c +0 -0
 	if (!vaddr)

return -EINVAL;

--
2.45.2

author
#7 lore raw

PATCH v4 4/5 udmabuf: udmabuf_create codestyle cleanup

In reply to Huan Yang

There are some variables in udmabuf_create that are only used inside the loop. Therefore, there is no need to declare them outside the scope. This patch moved it into loop.

It is difficult to understand the loop condition of the code that adds folio to the unpin_list.

This patch move item folio pin and record into a single function, when pinned success, the outer loop of this patch iterates through folios, while the inner loop correctly sets the folio and corresponding offset into the udmabuf starting from the offset. if reach to pgcnt or nr_folios, end of loop.

If item size is huge, folios may use vmalloc to get memory, which can't cache but return into pcp(or buddy) when vfree. So, each pin may waste some time in folios array alloc.
This patch also reuse of folios when iter create head, just use max size of item.

---
 drivers/dma-buf/udmabuf.c | 165 +++++++++++++++++++++++---------------
 1 file changed, 101 insertions(+), 64 deletions(-)
drivers/dma-buf/udmabuf.c +101 -64
index 0bbc9df36c0a..eb55bb4a5fcc 100644
@@ -321,17 +321,87 @@ static int export_udmabuf(struct udmabuf *ubuf,
 	return dma_buf_fd(buf, flags);
 }
 
+static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,
+				     struct udmabuf *ubuf,
+				     struct folio **folios)
+{
+	struct file *memfd = NULL;
+	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	u32 cur_folio, cur_pgcnt;
+	struct folio **ubuf_folios;
+	pgoff_t *ubuf_offsets;
+	long nr_folios;
+	loff_t end, start;
+	int ret;
+
+	memfd = fget(item->memfd);
+	ret = check_memfd_seals(memfd);
+	if (ret < 0)
+		goto err;
+
+	start = item->offset;
+	ipgcnt = item->size >> PAGE_SHIFT;
+	end = start + (ipgcnt << PAGE_SHIFT) - 1;
+
+	nr_folios = memfd_pin_folios(memfd, start, end, folios, ipgcnt, &pgoff);
+	if (nr_folios <= 0) {
+		kvfree(folios);
+		ret = nr_folios ? nr_folios : -EINVAL;
+		goto err;
+	}
+
+	cur_pgcnt = 0;
+	ubuf_folios = ubuf->folios;
+	ubuf_offsets = ubuf->offsets;
+
+	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
+		pgoff_t subpgoff = pgoff;
+		long fsize = folio_size(folios[cur_folio]);
+
+		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
+		if (ret < 0) {
+			kfree(folios);
+			goto err;
+		}
+
+		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {
+			ubuf->folios[upgcnt] = folios[cur_folio];
+			ubuf->offsets[upgcnt] = subpgoff;
+			++upgcnt;
+
+			if (++cur_pgcnt >= ipgcnt)
+				goto end;
+		}
+
+		/**
+		 * Only first folio in item may start from offset,
+		 * so remain folio start from 0.
+		 */
+		pgoff = 0;
+	}
+end:
+	ubuf->pagecount = upgcnt;
+	fput(memfd);
+
+	return 0;
+
+err:
+	ubuf->pagecount = upgcnt;
+	if (memfd)
+		fput(memfd);
+
+	return ret;
+}
+
 static long udmabuf_create(struct miscdevice *device,
 			   struct udmabuf_create_list *head,
 			   struct udmabuf_create_item *list)
 {
-	pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0;
-	long nr_folios, ret = -EINVAL;
-	struct file *memfd = NULL;
-	struct folio **folios;
+	pgoff_t pgcnt = 0, pglimit, max_ipgcnt = 0;
+	long ret = -EINVAL;
 	struct udmabuf *ubuf;
-	u32 i, j, k, flags;
-	loff_t end;
+	struct folio **folios = NULL;
+	u32 i, flags;
 
 	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);
 	if (!ubuf)
@@ -340,82 +410,50 @@ static long udmabuf_create(struct miscdevice *device,
 	INIT_LIST_HEAD(&ubuf->unpin_list);
 	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
 	for (i = 0; i < head->count; i++) {
-		if (!IS_ALIGNED(list[i].offset, PAGE_SIZE))
+		pgoff_t itempgcnt;
+
+		if (!PAGE_ALIGNED(list[i].offset))
 			goto err;
-		if (!IS_ALIGNED(list[i].size, PAGE_SIZE))
+		if (!PAGE_ALIGNED(list[i].size))
 			goto err;
-		ubuf->pagecount += list[i].size >> PAGE_SHIFT;
-		if (ubuf->pagecount > pglimit)
+
+		itempgcnt = list[i].size >> PAGE_SHIFT;
+		pgcnt += itempgcnt;
+
+		if (pgcnt > pglimit)
 			goto err;
+
+		max_ipgcnt = max_t(unsigned long, itempgcnt, max_ipgcnt);
 	}
 
-	if (!ubuf->pagecount)
+	if (!pgcnt)
 		goto err;
 
-	ubuf->folios = kvmalloc_array(ubuf->pagecount, sizeof(*ubuf->folios),
+	ubuf->folios = kvmalloc_array(pgcnt, sizeof(*ubuf->folios),
 				      GFP_KERNEL);
 	if (!ubuf->folios) {
 		ret = -ENOMEM;
 		goto err;
 	}
-	ubuf->offsets = kvcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
-				 GFP_KERNEL);
+
+	ubuf->offsets = kvcalloc(pgcnt, sizeof(*ubuf->offsets), GFP_KERNEL);
 	if (!ubuf->offsets) {
 		ret = -ENOMEM;
 		goto err;
 	}
 
-	pgbuf = 0;
-	for (i = 0; i < head->count; i++) {
-		memfd = fget(list[i].memfd);
-		ret = check_memfd_seals(memfd);
-		if (ret < 0)
-			goto err;
-
-		pgcnt = list[i].size >> PAGE_SHIFT;
-		folios = kvmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
-		if (!folios) {
-			ret = -ENOMEM;
-			goto err;
-		}
+	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
+	if (!folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
 
-		end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1;
-		ret = memfd_pin_folios(memfd, list[i].offset, end,
-				       folios, pgcnt, &pgoff);
-		if (ret <= 0) {
-			kvfree(folios);
-			if (!ret)
-				ret = -EINVAL;
+	for (i = 0; i < head->count; i++) {
+		ret = __udmabuf_pin_list_folios(&list[i], ubuf, folios);
+		if (ret)
 			goto err;
-		}
-
-		nr_folios = ret;
-		pgoff >>= PAGE_SHIFT;
-		for (j = 0, k = 0; j < pgcnt; j++) {
-			ubuf->folios[pgbuf] = folios[k];
-			ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT;
-
-			if (j == 0 || ubuf->folios[pgbuf-1] != folios[k]) {
-				ret = add_to_unpin_list(&ubuf->unpin_list,
-							folios[k]);
-				if (ret < 0) {
-					kfree(folios);
-					goto err;
-				}
-			}
-
-			pgbuf++;
-			if (++pgoff == folio_nr_pages(folios[k])) {
-				pgoff = 0;
-				if (++k == nr_folios)
-					break;
-			}
-		}
-
-		kvfree(folios);
-		fput(memfd);
-		memfd = NULL;
 	}
+	kvfree(folios);
 
 	flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0;
 	ret = export_udmabuf(ubuf, device, flags);
@@ -425,9 +463,8 @@ static long udmabuf_create(struct miscdevice *device,
 	return ret;
 
 err:
-	if (memfd)
-		fput(memfd);
 	unpin_all_folios(&ubuf->unpin_list);
+	kvfree(folios);
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);
 	kfree(ubuf);
Signature · 1 line
2.45.2
maintainer
#8 lore raw

RE: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup

Hi Huan,

Quoted 31 lines

Subject: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup

There are some variables in udmabuf_create that are only used inside the loop. Therefore, there is no need to declare them outside the scope. This patch moved it into loop.

It is difficult to understand the loop condition of the code that adds folio to the unpin_list.

This patch move item folio pin and record into a single function, when pinned success, the outer loop of this patch iterates through folios, while the inner loop correctly sets the folio and corresponding offset into the udmabuf starting from the offset. if reach to pgcnt or nr_folios, end of loop.

If item size is huge, folios may use vmalloc to get memory, which can't cache but return into pcp(or buddy) when vfree. So, each pin may waste some time in folios array alloc.
This patch also reuse of folios when iter create head, just use max size of item.

---
 drivers/dma-buf/udmabuf.c | 165 +++++++++++++++++++++++---------------
 1 file changed, 101 insertions(+), 64 deletions(-)
drivers/dma-buf/udmabuf.c +0 -0
index 0bbc9df36c0a..eb55bb4a5fcc 100644

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 324

Quoted 4 lines
drivers/dma-buf/udmabuf.c +1 -0
@@ -321,17 +321,87 @@ static int export_udmabuf(struct udmabuf *ubuf,
 	return dma_buf_fd(buf, flags);
 }
 
+static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,

I think the name udmabuf_pin_folios() for this function would be simple and apt.

Quoted 6 lines
drivers/dma-buf/udmabuf.c +6 -0
+				     struct udmabuf *ubuf,
+				     struct folio **folios)
+{
+	struct file *memfd = NULL;
+	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	u32 cur_folio, cur_pgcnt;

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 340

Quoted 10 lines
drivers/dma-buf/udmabuf.c +10 -0
+	struct folio **ubuf_folios;
+	pgoff_t *ubuf_offsets;
+	long nr_folios;
+	loff_t end, start;
+	int ret;
+
+	memfd = fget(item->memfd);
+	ret = check_memfd_seals(memfd);
+	if (ret < 0)
+		goto err;

Please move the above hunk to udmabuf_create(). Lets just have pinning and processing of folios in this function.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 343

Quoted 3 lines
drivers/dma-buf/udmabuf.c +3 -0
+
+	start = item->offset;
+	ipgcnt = item->size >> PAGE_SHIFT;

I think it would be a bit more clear to have udmabuf_create() pass start and size values directly to this function instead of item. And rename ipgcnt to something like subpgcnt or nr_subpgs.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 348

Quoted 6 lines
drivers/dma-buf/udmabuf.c +5 -0
+	end = start + (ipgcnt << PAGE_SHIFT) - 1;
+
+	nr_folios = memfd_pin_folios(memfd, start, end, folios, ipgcnt,
 &pgoff);
+	if (nr_folios <= 0) {
+		kvfree(folios);

Please free folios in udmabuf_create() which is where it was allocated.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 355

Quoted 7 lines
drivers/dma-buf/udmabuf.c +7 -0
+		ret = nr_folios ? nr_folios : -EINVAL;
+		goto err;
+	}
+
+	cur_pgcnt = 0;
+	ubuf_folios = ubuf->folios;
+	ubuf_offsets = ubuf->offsets;

Please initialize these temp variables at declaration time above. No strong opinion but I am not sure if they are really helpful here. Something like upgcnt would be OK as it definitely improves readability.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 359

Quoted 4 lines
drivers/dma-buf/udmabuf.c +4 -0
+
+	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
+		pgoff_t subpgoff = pgoff;
+		long fsize = folio_size(folios[cur_folio]);

The return type for folio_size() is size_t. Please use that for consistency.

Quoted 8 lines
drivers/dma-buf/udmabuf.c +8 -0
+
+		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
+		if (ret < 0) {
+			kfree(folios);
+			goto err;
+		}
+
+		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 377

Quoted 10 lines
drivers/dma-buf/udmabuf.c +10 -0
+			ubuf->folios[upgcnt] = folios[cur_folio];
+			ubuf->offsets[upgcnt] = subpgoff;
+			++upgcnt;
+
+			if (++cur_pgcnt >= ipgcnt)
+				goto end;
+		}
+
+		/**
+		 * Only first folio in item may start from offset,

I prefer to use the term range instead of item, in this context.

Quoted 49 lines
drivers/dma-buf/udmabuf.c +22 -6
+		 * so remain folio start from 0.
+		 */
+		pgoff = 0;
+	}
+end:
+	ubuf->pagecount = upgcnt;
+	fput(memfd);
+
+	return 0;
+
+err:
+	ubuf->pagecount = upgcnt;
+	if (memfd)
+		fput(memfd);
+
+	return ret;
+}
+
 static long udmabuf_create(struct miscdevice *device,
 			   struct udmabuf_create_list *head,
 			   struct udmabuf_create_item *list)
 {
-	pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0;
-	long nr_folios, ret = -EINVAL;
-	struct file *memfd = NULL;
-	struct folio **folios;
+	pgoff_t pgcnt = 0, pglimit, max_ipgcnt = 0;
+	long ret = -EINVAL;
 	struct udmabuf *ubuf;
-	u32 i, j, k, flags;
-	loff_t end;
+	struct folio **folios = NULL;
+	u32 i, flags;
 
 	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);
drivers/dma-buf/udmabuf.c +4 -2
 	if (!ubuf)
@@ -340,82 +410,50 @@ static long udmabuf_create(struct miscdevice
 *device,
 	INIT_LIST_HEAD(&ubuf->unpin_list);
 	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
 	for (i = 0; i < head->count; i++) {
-		if (!IS_ALIGNED(list[i].offset, PAGE_SIZE))
+		pgoff_t itempgcnt;
+
+		if (!PAGE_ALIGNED(list[i].offset))
 			goto err;
-		if (!IS_ALIGNED(list[i].size, PAGE_SIZE))
+		if (!PAGE_ALIGNED(list[i].size))
 			goto err;

drivers/dma-buf/udmabuf.c static long udmabuf_create(struct miscdevice · line 427

Quoted 10 lines
drivers/dma-buf/udmabuf.c +7 -2
-		ubuf->pagecount += list[i].size >> PAGE_SHIFT;
-		if (ubuf->pagecount > pglimit)
+
+		itempgcnt = list[i].size >> PAGE_SHIFT;
+		pgcnt += itempgcnt;
+
+		if (pgcnt > pglimit)
 			goto err;
+
+		max_ipgcnt = max_t(unsigned long, itempgcnt, max_ipgcnt);

Is this optimization really necessary given that, in practice, the userspace provides only a few ranges? It can stay but please pull these changes into a separate patch.

Thanks,
Vivek

Quoted 100 lines
drivers/dma-buf/udmabuf.c +13 -52
 	}
 
-	if (!ubuf->pagecount)
+	if (!pgcnt)
 		goto err;
 
-	ubuf->folios = kvmalloc_array(ubuf->pagecount, sizeof(*ubuf-
 >folios),
+	ubuf->folios = kvmalloc_array(pgcnt, sizeof(*ubuf->folios),
 				      GFP_KERNEL);
 	if (!ubuf->folios) {
 		ret = -ENOMEM;
 		goto err;
 	}
-	ubuf->offsets = kvcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
-				 GFP_KERNEL);
+
+	ubuf->offsets = kvcalloc(pgcnt, sizeof(*ubuf->offsets), GFP_KERNEL);
 	if (!ubuf->offsets) {
 		ret = -ENOMEM;
 		goto err;
 	}
 
-	pgbuf = 0;
-	for (i = 0; i < head->count; i++) {
-		memfd = fget(list[i].memfd);
-		ret = check_memfd_seals(memfd);
-		if (ret < 0)
-			goto err;
-
-		pgcnt = list[i].size >> PAGE_SHIFT;
-		folios = kvmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
-		if (!folios) {
-			ret = -ENOMEM;
-			goto err;
-		}
+	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
+	if (!folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
 
-		end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1;
-		ret = memfd_pin_folios(memfd, list[i].offset, end,
-				       folios, pgcnt, &pgoff);
-		if (ret <= 0) {
-			kvfree(folios);
-			if (!ret)
-				ret = -EINVAL;
+	for (i = 0; i < head->count; i++) {
+		ret = __udmabuf_pin_list_folios(&list[i], ubuf, folios);
+		if (ret)
 			goto err;
-		}
-
-		nr_folios = ret;
-		pgoff >>= PAGE_SHIFT;
-		for (j = 0, k = 0; j < pgcnt; j++) {
-			ubuf->folios[pgbuf] = folios[k];
-			ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT;
-
-			if (j == 0 || ubuf->folios[pgbuf-1] != folios[k]) {
-				ret = add_to_unpin_list(&ubuf->unpin_list,
-							folios[k]);
-				if (ret < 0) {
-					kfree(folios);
-					goto err;
-				}
-			}
-
-			pgbuf++;
-			if (++pgoff == folio_nr_pages(folios[k])) {
-				pgoff = 0;
-				if (++k == nr_folios)
-					break;
-			}
-		}
-
-		kvfree(folios);
-		fput(memfd);
-		memfd = NULL;
 	}
+	kvfree(folios);
 
drivers/dma-buf/udmabuf.c +1 -2
 	flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0;
 	ret = export_udmabuf(ubuf, device, flags);
@@ -425,9 +463,8 @@ static long udmabuf_create(struct miscdevice
 *device,
 	return ret;
 
 err:
-	if (memfd)
-		fput(memfd);
 	unpin_all_folios(&ubuf->unpin_list);
+	kvfree(folios);
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);

kfree(ubuf);
--
2.45.2

author
#9 lore raw

Re: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup

Quoted 101 lines · Vivek

Hi Huan,

Quoted 31 lines

Subject: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup

There are some variables in udmabuf_create that are only used inside the loop. Therefore, there is no need to declare them outside the scope. This patch moved it into loop.

It is difficult to understand the loop condition of the code that adds folio to the unpin_list.

This patch move item folio pin and record into a single function, when pinned success, the outer loop of this patch iterates through folios, while the inner loop correctly sets the folio and corresponding offset into the udmabuf starting from the offset. if reach to pgcnt or nr_folios, end of loop.

If item size is huge, folios may use vmalloc to get memory, which can't cache but return into pcp(or buddy) when vfree. So, each pin may waste some time in folios array alloc.
This patch also reuse of folios when iter create head, just use max size of item.

---
  drivers/dma-buf/udmabuf.c | 165 +++++++++++++++++++++++---------------
  1 file changed, 101 insertions(+), 64 deletions(-)
drivers/dma-buf/udmabuf.c +0 -0
index 0bbc9df36c0a..eb55bb4a5fcc 100644

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 324

Quoted 4 lines
drivers/dma-buf/udmabuf.c +1 -0
@@ -321,17 +321,87 @@ static int export_udmabuf(struct udmabuf *ubuf,
  	return dma_buf_fd(buf, flags);
  }
 
+static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,

I think the name udmabuf_pin_folios() for this function would be simple and apt.

Quoted 6 lines
drivers/dma-buf/udmabuf.c +6 -0
+				     struct udmabuf *ubuf,
+				     struct folio **folios)
+{
+	struct file *memfd = NULL;
+	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	u32 cur_folio, cur_pgcnt;

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 340

Quoted 10 lines
drivers/dma-buf/udmabuf.c +10 -0
+	struct folio **ubuf_folios;
+	pgoff_t *ubuf_offsets;
+	long nr_folios;
+	loff_t end, start;
+	int ret;
+
+	memfd = fget(item->memfd);
+	ret = check_memfd_seals(memfd);
+	if (ret < 0)
+		goto err;

Please move the above hunk to udmabuf_create(). Lets just have pinning and processing of folios in this function.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 343

Quoted 3 lines
drivers/dma-buf/udmabuf.c +3 -0
+
+	start = item->offset;
+	ipgcnt = item->size >> PAGE_SHIFT;

I think it would be a bit more clear to have udmabuf_create() pass start and size values directly to this function instead of item. And rename ipgcnt to something like subpgcnt or nr_subpgs.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 348

Quoted 6 lines
drivers/dma-buf/udmabuf.c +5 -0
+	end = start + (ipgcnt << PAGE_SHIFT) - 1;
+
+	nr_folios = memfd_pin_folios(memfd, start, end, folios, ipgcnt,
 &pgoff);
+	if (nr_folios <= 0) {
+		kvfree(folios);

Please free folios in udmabuf_create() which is where it was allocated.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 355

Quoted 7 lines
drivers/dma-buf/udmabuf.c +7 -0
+		ret = nr_folios ? nr_folios : -EINVAL;
+		goto err;
+	}
+
+	cur_pgcnt = 0;
+	ubuf_folios = ubuf->folios;
+	ubuf_offsets = ubuf->offsets;

Please initialize these temp variables at declaration time above. No strong opinion but I am not sure if they are really helpful here. Something like upgcnt would be OK as it definitely improves readability.

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 359

Quoted 4 lines
drivers/dma-buf/udmabuf.c +4 -0
+
+	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
+		pgoff_t subpgoff = pgoff;
+		long fsize = folio_size(folios[cur_folio]);

The return type for folio_size() is size_t. Please use that for consistency.

Quoted 11 lines
drivers/dma-buf/udmabuf.c +11 -0
+
+		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
+		if (ret < 0) {
+			kfree(folios);
+			goto err;
+		}
+
+		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {
+			ubuf->folios[upgcnt] = folios[cur_folio];
+			ubuf->offsets[upgcnt] = subpgoff;
+			++upgcnt;
Quoted 8 lines

drivers/dma-buf/udmabuf.c static int export_udmabuf(struct udmabuf *ubuf, · line 377

Quoted 7 lines
drivers/dma-buf/udmabuf.c +7 -0
+
+			if (++cur_pgcnt >= ipgcnt)
+				goto end;
+		}
+
+		/**
+		 * Only first folio in item may start from offset,

I prefer to use the term range instead of item, in this context.

All above I'll rework.

Quoted 54 lines
Quoted 54 lines
drivers/dma-buf/udmabuf.c +22 -6
+		 * so remain folio start from 0.
+		 */
+		pgoff = 0;
+	}
+end:
+	ubuf->pagecount = upgcnt;
+	fput(memfd);
+
+	return 0;
+
+err:
+	ubuf->pagecount = upgcnt;
+	if (memfd)
+		fput(memfd);
+
+	return ret;
+}
+
  static long udmabuf_create(struct miscdevice *device,
  			   struct udmabuf_create_list *head,
  			   struct udmabuf_create_item *list)
  {
-	pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0;
-	long nr_folios, ret = -EINVAL;
-	struct file *memfd = NULL;
-	struct folio **folios;
+	pgoff_t pgcnt = 0, pglimit, max_ipgcnt = 0;
+	long ret = -EINVAL;
  	struct udmabuf *ubuf;
-	u32 i, j, k, flags;
-	loff_t end;
+	struct folio **folios = NULL;
+	u32 i, flags;
 
  	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);

if (!ubuf)

drivers/dma-buf/udmabuf.c +6 -4
@@ -340,82 +410,50 @@ static long udmabuf_create(struct miscdevice
 *device,
  	INIT_LIST_HEAD(&ubuf->unpin_list);
  	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
  	for (i = 0; i < head->count; i++) {
-		if (!IS_ALIGNED(list[i].offset, PAGE_SIZE))
+		pgoff_t itempgcnt;
+
+		if (!PAGE_ALIGNED(list[i].offset))
  			goto err;
-		if (!IS_ALIGNED(list[i].size, PAGE_SIZE))
+		if (!PAGE_ALIGNED(list[i].size))
  			goto err;
-		ubuf->pagecount += list[i].size >> PAGE_SHIFT;
-		if (ubuf->pagecount > pglimit)
+
+		itempgcnt = list[i].size >> PAGE_SHIFT;
Quoted 8 lines

drivers/dma-buf/udmabuf.c static long udmabuf_create(struct miscdevice *device, · line 426

Quoted 6 lines
drivers/dma-buf/udmabuf.c +5 -0
+		pgcnt += itempgcnt;
+
+		if (pgcnt > pglimit)
  			goto err;
+
+		max_ipgcnt = max_t(unsigned long, itempgcnt, max_ipgcnt);

Is this optimization really necessary given that, in practice, the userspace provides only a few ranges? It can stay but please pull these changes into a separate patch.

OK, I'll separate it.

If few ranges, folios can alloc from kmalloc, if low, from slub is fast. If low than PCP order, also fast.

But if trigger vmalloc, I think it's slow more. Consider 3GB udmabuf create(even if currently not used)

page array will cost 6MB, from vmalloc will iter alloc 4K page upon to 1536 time.

So, a little help if only reuse the max size folio.

Thanks for your suggestions.

Quoted 104 lines

Thanks,
Vivek

Quoted 100 lines
drivers/dma-buf/udmabuf.c +13 -52
  	}
 
-	if (!ubuf->pagecount)
+	if (!pgcnt)
  		goto err;
 
-	ubuf->folios = kvmalloc_array(ubuf->pagecount, sizeof(*ubuf-
 > folios),
+	ubuf->folios = kvmalloc_array(pgcnt, sizeof(*ubuf->folios),
  				      GFP_KERNEL);
  	if (!ubuf->folios) {
  		ret = -ENOMEM;
  		goto err;
  	}
-	ubuf->offsets = kvcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
-				 GFP_KERNEL);
+
+	ubuf->offsets = kvcalloc(pgcnt, sizeof(*ubuf->offsets), GFP_KERNEL);
  	if (!ubuf->offsets) {
  		ret = -ENOMEM;
  		goto err;
  	}
 
-	pgbuf = 0;
-	for (i = 0; i < head->count; i++) {
-		memfd = fget(list[i].memfd);
-		ret = check_memfd_seals(memfd);
-		if (ret < 0)
-			goto err;
-
-		pgcnt = list[i].size >> PAGE_SHIFT;
-		folios = kvmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
-		if (!folios) {
-			ret = -ENOMEM;
-			goto err;
-		}
+	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
+	if (!folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
 
-		end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1;
-		ret = memfd_pin_folios(memfd, list[i].offset, end,
-				       folios, pgcnt, &pgoff);
-		if (ret <= 0) {
-			kvfree(folios);
-			if (!ret)
-				ret = -EINVAL;
+	for (i = 0; i < head->count; i++) {
+		ret = __udmabuf_pin_list_folios(&list[i], ubuf, folios);
+		if (ret)
  			goto err;
-		}
-
-		nr_folios = ret;
-		pgoff >>= PAGE_SHIFT;
-		for (j = 0, k = 0; j < pgcnt; j++) {
-			ubuf->folios[pgbuf] = folios[k];
-			ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT;
-
-			if (j == 0 || ubuf->folios[pgbuf-1] != folios[k]) {
-				ret = add_to_unpin_list(&ubuf->unpin_list,
-							folios[k]);
-				if (ret < 0) {
-					kfree(folios);
-					goto err;
-				}
-			}
-
-			pgbuf++;
-			if (++pgoff == folio_nr_pages(folios[k])) {
-				pgoff = 0;
-				if (++k == nr_folios)
-					break;
-			}
-		}
-
-		kvfree(folios);
-		fput(memfd);
-		memfd = NULL;
  	}
+	kvfree(folios);
 
  	flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0;
  	ret = export_udmabuf(ubuf, device, flags);
drivers/dma-buf/udmabuf.c +1 -2
@@ -425,9 +463,8 @@ static long udmabuf_create(struct miscdevice
 *device,
  	return ret;
 
  err:
-	if (memfd)
-		fput(memfd);
  	unpin_all_folios(&ubuf->unpin_list);
+	kvfree(folios);
  	kvfree(ubuf->offsets);
  	kvfree(ubuf->folios);

kfree(ubuf);
--
2.45.2

#10 lore raw

Re: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup

In reply to Huan Yang

Hi Huan,

kernel test robot noticed the following build warnings:

url:    https://github.com/intel-lab-lkp/linux/commits/Huan-Yang/udmabuf-direct-map-pfn-when-first-page-fault/20240826-105359
base:   6a7917c89f219f09b1d88d09f376000914a52763
patch link:    https://lore.kernel.org/r/20240822084342.1574914-5-link%40vivo.com
patch subject: [PATCH v4 4/5] udmabuf: udmabuf_create codestyle cleanup
config: x86_64-randconfig-161-20240829 (https://download.01.org/0day-ci/archive/20240829/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

smatch warnings:
drivers/dma-buf/udmabuf.c:467 udmabuf_create() error: double free of 'folios'

vim +/folios +467 drivers/dma-buf/udmabuf.c

c1bbed66899726 Gurchetan Singh 2019-12-02  396  static long udmabuf_create(struct miscdevice *device,
c1bbed66899726 Gurchetan Singh 2019-12-02  397  			   struct udmabuf_create_list *head,
c1bbed66899726 Gurchetan Singh 2019-12-02  398  			   struct udmabuf_create_item *list)
fbb0de79507819 Gerd Hoffmann   2018-08-27  399  {
fb2c508270085b Huan Yang       2024-08-22  400  	pgoff_t pgcnt = 0, pglimit, max_ipgcnt = 0;
fb2c508270085b Huan Yang       2024-08-22  401  	long ret = -EINVAL;
fbb0de79507819 Gerd Hoffmann   2018-08-27  402  	struct udmabuf *ubuf;
fb2c508270085b Huan Yang       2024-08-22  403  	struct folio **folios = NULL;
fb2c508270085b Huan Yang       2024-08-22  404  	u32 i, flags;
fbb0de79507819 Gerd Hoffmann   2018-08-27  405  
33f35429fc49c0 Gerd Hoffmann   2018-09-11  406  	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);
fbb0de79507819 Gerd Hoffmann   2018-08-27  407  	if (!ubuf)
fbb0de79507819 Gerd Hoffmann   2018-08-27  408  		return -ENOMEM;
fbb0de79507819 Gerd Hoffmann   2018-08-27  409  
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  410  	INIT_LIST_HEAD(&ubuf->unpin_list);
dc4716d75154b3 Gerd Hoffmann   2018-09-11  411  	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
fbb0de79507819 Gerd Hoffmann   2018-08-27  412  	for (i = 0; i < head->count; i++) {
fb2c508270085b Huan Yang       2024-08-22  413  		pgoff_t itempgcnt;
fb2c508270085b Huan Yang       2024-08-22  414  
fb2c508270085b Huan Yang       2024-08-22  415  		if (!PAGE_ALIGNED(list[i].offset))
0d17455ca85ecb Gerd Hoffmann   2018-09-11  416  			goto err;
fb2c508270085b Huan Yang       2024-08-22  417  		if (!PAGE_ALIGNED(list[i].size))
0d17455ca85ecb Gerd Hoffmann   2018-09-11  418  			goto err;
fb2c508270085b Huan Yang       2024-08-22  419  
fb2c508270085b Huan Yang       2024-08-22  420  		itempgcnt = list[i].size >> PAGE_SHIFT;
fb2c508270085b Huan Yang       2024-08-22  421  		pgcnt += itempgcnt;
fb2c508270085b Huan Yang       2024-08-22  422  
fb2c508270085b Huan Yang       2024-08-22  423  		if (pgcnt > pglimit)
0d17455ca85ecb Gerd Hoffmann   2018-09-11  424  			goto err;
fb2c508270085b Huan Yang       2024-08-22  425  
fb2c508270085b Huan Yang       2024-08-22  426  		max_ipgcnt = max_t(unsigned long, itempgcnt, max_ipgcnt);
fbb0de79507819 Gerd Hoffmann   2018-08-27  427  	}
2b6dd600dd7257 Pavel Skripkin  2021-12-30  428  
fb2c508270085b Huan Yang       2024-08-22  429  	if (!pgcnt)
2b6dd600dd7257 Pavel Skripkin  2021-12-30  430  		goto err;
2b6dd600dd7257 Pavel Skripkin  2021-12-30  431  
fb2c508270085b Huan Yang       2024-08-22  432  	ubuf->folios = kvmalloc_array(pgcnt, sizeof(*ubuf->folios),
fbb0de79507819 Gerd Hoffmann   2018-08-27  433  				      GFP_KERNEL);
5e72b2b41a21e5 Vivek Kasireddy 2024-06-23  434  	if (!ubuf->folios) {
fbb0de79507819 Gerd Hoffmann   2018-08-27  435  		ret = -ENOMEM;
0d17455ca85ecb Gerd Hoffmann   2018-09-11  436  		goto err;
fbb0de79507819 Gerd Hoffmann   2018-08-27  437  	}
fb2c508270085b Huan Yang       2024-08-22  438  
fb2c508270085b Huan Yang       2024-08-22  439  	ubuf->offsets = kvcalloc(pgcnt, sizeof(*ubuf->offsets), GFP_KERNEL);
0c8b91ef5100ea Vivek Kasireddy 2024-06-23  440  	if (!ubuf->offsets) {
0c8b91ef5100ea Vivek Kasireddy 2024-06-23  441  		ret = -ENOMEM;
0c8b91ef5100ea Vivek Kasireddy 2024-06-23  442  		goto err;
0c8b91ef5100ea Vivek Kasireddy 2024-06-23  443  	}
fbb0de79507819 Gerd Hoffmann   2018-08-27  444  
fb2c508270085b Huan Yang       2024-08-22  445  	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  446  	if (!folios) {
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  447  		ret = -ENOMEM;
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  448  		goto err;
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  449  	}
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  450  
fb2c508270085b Huan Yang       2024-08-22  451  	for (i = 0; i < head->count; i++) {
fb2c508270085b Huan Yang       2024-08-22  452  		ret = __udmabuf_pin_list_folios(&list[i], ubuf, folios);

There is a kfree(folios) hidden inside this function. It doesn't belong there.

fb2c508270085b Huan Yang       2024-08-22  453  		if (ret)
0d17455ca85ecb Gerd Hoffmann   2018-09-11  454  			goto err;
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  455  	}
452dc1b0221804 Huan Yang       2024-08-22  456  	kvfree(folios);
                                                        ^^^^^^^^^^^^^^

A second free

fbb0de79507819 Gerd Hoffmann   2018-08-27  457  
5e72b2b41a21e5 Vivek Kasireddy 2024-06-23  458  	flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0;
5e72b2b41a21e5 Vivek Kasireddy 2024-06-23  459  	ret = export_udmabuf(ubuf, device, flags);
5e72b2b41a21e5 Vivek Kasireddy 2024-06-23  460  	if (ret < 0)
0d17455ca85ecb Gerd Hoffmann   2018-09-11  461  		goto err;
                                                                ^^^^^^^^
fbb0de79507819 Gerd Hoffmann   2018-08-27  462  
5e72b2b41a21e5 Vivek Kasireddy 2024-06-23  463  	return ret;
fbb0de79507819 Gerd Hoffmann   2018-08-27  464  
0d17455ca85ecb Gerd Hoffmann   2018-09-11  465  err:
c6a3194c05e7e6 Vivek Kasireddy 2024-06-23  466  	unpin_all_folios(&ubuf->unpin_list);
fb2c508270085b Huan Yang       2024-08-22 @467  	kvfree(folios);
                                                        ^^^^^^^^^^^^^

Double free

452dc1b0221804 Huan Yang       2024-08-22  468  	kvfree(ubuf->offsets);
452dc1b0221804 Huan Yang       2024-08-22  469  	kvfree(ubuf->folios);
fbb0de79507819 Gerd Hoffmann   2018-08-27  470  	kfree(ubuf);
fbb0de79507819 Gerd Hoffmann   2018-08-27  471  	return ret;
fbb0de79507819 Gerd Hoffmann   2018-08-27  472  }
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#11 lore raw

PATCH v4 5/5 udmabuf: remove udmabuf_folio

In reply to Huan Yang

Currently, udmabuf handles folio by creating an unpin list to record each folio obtained from the list and unpinning them when released. To maintain this approach, many data structures have been established.

However, maintaining this type of data structure requires a significant amount of memory and traversing the list is a substantial overhead, which is not friendly to the CPU cache.

We actually don't need to use unpin_list to unpin during release. Instead, just use a folios array record each folio is ok. Compare udmabuf_folio 24 byte, folio array is 8 byte. Even if array need to be pgcnt*8, may waste some memory when use large folio. The access of array is faster than list, also, if 4K, array can also save memory than list.

---
 drivers/dma-buf/udmabuf.c | 80 ++++++++++++++++++---------------------
 1 file changed, 37 insertions(+), 43 deletions(-)
drivers/dma-buf/udmabuf.c +37 -43
index eb55bb4a5fcc..a45cec1f82b3 100644
@@ -27,15 +27,21 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a dmabuf, in megabytes. Default is
 struct udmabuf {
 	pgoff_t pagecount;
 	struct folio **folios;
+
+	/**
+	 * Unlike folios, pinned_folios is only used for unpin.
+	 * So, nr_pinned is not the same to pagecount, the pinned_folios
+	 * only set each folio which already pinned when udmabuf_create.
+	 * Note that, since a folio may be pinned multiple times, each folio
+	 * can be added to pinned_folios multiple times, depending on how many
+	 * times the folio has been pinned when create.
+	 */
+	pgoff_t nr_pinned;
+	struct folio **pinned_folios;
+
 	struct sg_table *sg;
 	struct miscdevice *device;
 	pgoff_t *offsets;
-	struct list_head unpin_list;
-};
-
-struct udmabuf_folio {
-	struct folio *folio;
-	struct list_head list;
 };
 
 static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
@@ -189,32 +195,12 @@ static void unmap_udmabuf(struct dma_buf_attachment *at,
 	return put_sg_table(at->dev, sg, direction);
 }
 
-static void unpin_all_folios(struct list_head *unpin_list)
-{
-	struct udmabuf_folio *ubuf_folio;
-
-	while (!list_empty(unpin_list)) {
-		ubuf_folio = list_first_entry(unpin_list,
-					      struct udmabuf_folio, list);
-		unpin_folio(ubuf_folio->folio);
-
-		list_del(&ubuf_folio->list);
-		kfree(ubuf_folio);
-	}
-}
-
-static int add_to_unpin_list(struct list_head *unpin_list,
-			     struct folio *folio)
+static __always_inline void unpin_all_folios(struct udmabuf *ubuf)
 {
-	struct udmabuf_folio *ubuf_folio;
-
-	ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL);
-	if (!ubuf_folio)
-		return -ENOMEM;
+	pgoff_t i;
 
-	ubuf_folio->folio = folio;
-	list_add_tail(&ubuf_folio->list, unpin_list);
-	return 0;
+	for (i = 0; i < ubuf->nr_pinned; ++i)
+		unpin_folio(ubuf->pinned_folios[i]);
 }
 
 static void release_udmabuf(struct dma_buf *buf)
@@ -225,7 +211,8 @@ static void release_udmabuf(struct dma_buf *buf)
 	if (ubuf->sg)
 		put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
 
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
+	kvfree(ubuf->pinned_folios);
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);
 	kfree(ubuf);
@@ -326,9 +313,9 @@ static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,
 				     struct folio **folios)
 {
 	struct file *memfd = NULL;
-	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	pgoff_t pgoff, ipgcnt, upgcnt, nr_pinned;
 	u32 cur_folio, cur_pgcnt;
-	struct folio **ubuf_folios;
+	struct folio **ubuf_folios, **pinned_folios;
 	pgoff_t *ubuf_offsets;
 	long nr_folios;
 	loff_t end, start;
@@ -351,22 +338,21 @@ static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,
 	}
 
 	cur_pgcnt = 0;
+	nr_pinned = ubuf->nr_pinned;
+	upgcnt = ubuf->pagecount;
 	ubuf_folios = ubuf->folios;
 	ubuf_offsets = ubuf->offsets;
+	pinned_folios = ubuf->pinned_folios;
 
 	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
 		pgoff_t subpgoff = pgoff;
 		long fsize = folio_size(folios[cur_folio]);
 
-		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
-		if (ret < 0) {
-			kfree(folios);
-			goto err;
-		}
+		pinned_folios[nr_pinned++] = folios[cur_folio];
 
 		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {
-			ubuf->folios[upgcnt] = folios[cur_folio];
-			ubuf->offsets[upgcnt] = subpgoff;
+			ubuf_folios[upgcnt] = folios[cur_folio];
+			ubuf_offsets[upgcnt] = subpgoff;
 			++upgcnt;
 
 			if (++cur_pgcnt >= ipgcnt)
@@ -381,12 +367,12 @@ static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item,
 	}
 end:
 	ubuf->pagecount = upgcnt;
+	ubuf->nr_pinned = nr_pinned;
 	fput(memfd);
 
 	return 0;
 
 err:
-	ubuf->pagecount = upgcnt;
 	if (memfd)
 		fput(memfd);
 
@@ -407,7 +393,6 @@ static long udmabuf_create(struct miscdevice *device,
 	if (!ubuf)
 		return -ENOMEM;
 
-	INIT_LIST_HEAD(&ubuf->unpin_list);
 	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
 	for (i = 0; i < head->count; i++) {
 		pgoff_t itempgcnt;
@@ -442,6 +427,14 @@ static long udmabuf_create(struct miscdevice *device,
 		goto err;
 	}
 
+	ubuf->pinned_folios = kvmalloc_array(pgcnt,
+					     sizeof(*ubuf->pinned_folios),
+					     GFP_KERNEL);
+	if (!ubuf->pinned_folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
 	if (!folios) {
 		ret = -ENOMEM;
@@ -463,8 +456,9 @@ static long udmabuf_create(struct miscdevice *device,
 	return ret;
 
 err:
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
 	kvfree(folios);
+	kvfree(ubuf->pinned_folios);
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);
 	kfree(ubuf);
Signature · 1 line
2.45.2
maintainer
#12 lore raw

RE: [PATCH v4 5/5] udmabuf: remove udmabuf_folio

Hi Huan,

Quoted 95 lines

Subject: [PATCH v4 5/5] udmabuf: remove udmabuf_folio

Currently, udmabuf handles folio by creating an unpin list to record each folio obtained from the list and unpinning them when released. To maintain this approach, many data structures have been established.

However, maintaining this type of data structure requires a significant amount of memory and traversing the list is a substantial overhead, which is not friendly to the CPU cache.

We actually don't need to use unpin_list to unpin during release. Instead, just use a folios array record each folio is ok. Compare udmabuf_folio 24 byte, folio array is 8 byte. Even if array need to be pgcnt*8, may waste some memory when use large folio. The access of array is faster than list, also, if 4K, array can also save memory than list.

---
 drivers/dma-buf/udmabuf.c | 80 ++++++++++++++++++---------------------
 1 file changed, 37 insertions(+), 43 deletions(-)
drivers/dma-buf/udmabuf.c +0 -0
index eb55bb4a5fcc..a45cec1f82b3 100644
@@ -27,15 +27,21 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a

dmabuf, in megabytes. Default is
struct udmabuf {

drivers/dma-buf/udmabuf.c +7 -0
 	pgoff_t pagecount;
 	struct folio **folios;
+
+	/**
+	 * Unlike folios, pinned_folios is only used for unpin.
+	 * So, nr_pinned is not the same to pagecount, the pinned_folios
+	 * only set each folio which already pinned when udmabuf_create.
+	 * Note that, since a folio may be pinned multiple times, each folio
+	 * can be added to pinned_folios multiple times, depending on how

many

drivers/dma-buf/udmabuf.c +9 -30
+	 * times the folio has been pinned when create.
+	 */
+	pgoff_t nr_pinned;
+	struct folio **pinned_folios;
+
 	struct sg_table *sg;
 	struct miscdevice *device;
 	pgoff_t *offsets;
-	struct list_head unpin_list;
-};
-
-struct udmabuf_folio {
-	struct folio *folio;
-	struct list_head list;
 };
 
 static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
@@ -189,32 +195,12 @@ static void unmap_udmabuf(struct
 dma_buf_attachment *at,
 	return put_sg_table(at->dev, sg, direction);
 }
 
-static void unpin_all_folios(struct list_head *unpin_list)
-{
-	struct udmabuf_folio *ubuf_folio;
-
-	while (!list_empty(unpin_list)) {
-		ubuf_folio = list_first_entry(unpin_list,
-					      struct udmabuf_folio, list);
-		unpin_folio(ubuf_folio->folio);
-
-		list_del(&ubuf_folio->list);
-		kfree(ubuf_folio);
-	}
-}
-
-static int add_to_unpin_list(struct list_head *unpin_list,
-			     struct folio *folio)
+static __always_inline void unpin_all_folios(struct udmabuf *ubuf)
 {
-	struct udmabuf_folio *ubuf_folio;
-
-	ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL);
-	if (!ubuf_folio)
-		return -ENOMEM;
+	pgoff_t i;
 
-	ubuf_folio->folio = folio;
-	list_add_tail(&ubuf_folio->list, unpin_list);
-	return 0;
+	for (i = 0; i < ubuf->nr_pinned; ++i)
+		unpin_folio(ubuf->pinned_folios[i]);
 }
 

static void release_udmabuf(struct dma_buf *buf)

drivers/dma-buf/udmabuf.c static void release_udmabuf(struct dma_buf *buf) · line 215

Quoted 6 lines
drivers/dma-buf/udmabuf.c +2 -1
@@ -225,7 +211,8 @@ static void release_udmabuf(struct dma_buf *buf)
 	if (ubuf->sg)
 		put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
 
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
+	kvfree(ubuf->pinned_folios);

Move the kvfree() of pinned_folios into unpin_all_folios().

Quoted 33 lines
drivers/dma-buf/udmabuf.c +2 -2
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);
 	kfree(ubuf);
@@ -326,9 +313,9 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
 				     struct folio **folios)
 {
 	struct file *memfd = NULL;
-	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	pgoff_t pgoff, ipgcnt, upgcnt, nr_pinned;
 	u32 cur_folio, cur_pgcnt;
-	struct folio **ubuf_folios;
+	struct folio **ubuf_folios, **pinned_folios;
 	pgoff_t *ubuf_offsets;
 	long nr_folios;
drivers/dma-buf/udmabuf.c +3 -2
 	loff_t end, start;
@@ -351,22 +338,21 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
 	}
 
 	cur_pgcnt = 0;
+	nr_pinned = ubuf->nr_pinned;
+	upgcnt = ubuf->pagecount;
 	ubuf_folios = ubuf->folios;
 	ubuf_offsets = ubuf->offsets;
+	pinned_folios = ubuf->pinned_folios;
 
 	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
 		pgoff_t subpgoff = pgoff;
 		long fsize = folio_size(folios[cur_folio]);
 
-		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
-		if (ret < 0) {

drivers/dma-buf/udmabuf.c static int __udmabuf_pin_list_folios(struct · line 356

Quoted 10 lines
drivers/dma-buf/udmabuf.c +3 -5
-			kfree(folios);
-			goto err;
-		}
+		pinned_folios[nr_pinned++] = folios[cur_folio];
 
 		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {
-			ubuf->folios[upgcnt] = folios[cur_folio];
-			ubuf->offsets[upgcnt] = subpgoff;
+			ubuf_folios[upgcnt] = folios[cur_folio];
+			ubuf_offsets[upgcnt] = subpgoff;

These are unrelated changes that should belong to the previous patch. I suggest moving this patch before the codestyle cleanup patch.

Thanks,
Vivek

Quoted 57 lines
drivers/dma-buf/udmabuf.c +0 -0
 			++upgcnt;
 
drivers/dma-buf/udmabuf.c +1 -2
 			if (++cur_pgcnt >= ipgcnt)
@@ -381,12 +367,12 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
 	}
 end:
 	ubuf->pagecount = upgcnt;
+	ubuf->nr_pinned = nr_pinned;
 	fput(memfd);
 
 	return 0;
 
 err:
-	ubuf->pagecount = upgcnt;
 	if (memfd)
 		fput(memfd);
@@ -407,7 +393,6 @@ static long udmabuf_create(struct miscdevice
 *device,
 	if (!ubuf)
 		return -ENOMEM;
 
-	INIT_LIST_HEAD(&ubuf->unpin_list);
 	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
 	for (i = 0; i < head->count; i++) {
drivers/dma-buf/udmabuf.c +8 -0
 		pgoff_t itempgcnt;
@@ -442,6 +427,14 @@ static long udmabuf_create(struct miscdevice
 *device,
 		goto err;
 	}
 
+	ubuf->pinned_folios = kvmalloc_array(pgcnt,
+					     sizeof(*ubuf->pinned_folios),
+					     GFP_KERNEL);
+	if (!ubuf->pinned_folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
 	if (!folios) {
drivers/dma-buf/udmabuf.c +2 -1
 		ret = -ENOMEM;
@@ -463,8 +456,9 @@ static long udmabuf_create(struct miscdevice
 *device,
 	return ret;
 
 err:
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
 	kvfree(folios);
+	kvfree(ubuf->pinned_folios);
 	kvfree(ubuf->offsets);
 	kvfree(ubuf->folios);

kfree(ubuf);
--
2.45.2

author
#13 lore raw

Re: [PATCH v4 5/5] udmabuf: remove udmabuf_folio

Quoted 143 lines · Vivek

Hi Huan,

Quoted 95 lines

Subject: [PATCH v4 5/5] udmabuf: remove udmabuf_folio

Currently, udmabuf handles folio by creating an unpin list to record each folio obtained from the list and unpinning them when released. To maintain this approach, many data structures have been established.

However, maintaining this type of data structure requires a significant amount of memory and traversing the list is a substantial overhead, which is not friendly to the CPU cache.

We actually don't need to use unpin_list to unpin during release. Instead, just use a folios array record each folio is ok. Compare udmabuf_folio 24 byte, folio array is 8 byte. Even if array need to be pgcnt*8, may waste some memory when use large folio. The access of array is faster than list, also, if 4K, array can also save memory than list.

---
  drivers/dma-buf/udmabuf.c | 80 ++++++++++++++++++---------------------
  1 file changed, 37 insertions(+), 43 deletions(-)
drivers/dma-buf/udmabuf.c +0 -0
index eb55bb4a5fcc..a45cec1f82b3 100644
@@ -27,15 +27,21 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a

dmabuf, in megabytes. Default is
struct udmabuf {
pgoff_t pagecount;
struct folio **folios;

drivers/dma-buf/udmabuf.c +7 -0
+
+	/**
+	 * Unlike folios, pinned_folios is only used for unpin.
+	 * So, nr_pinned is not the same to pagecount, the pinned_folios
+	 * only set each folio which already pinned when udmabuf_create.
+	 * Note that, since a folio may be pinned multiple times, each folio
+	 * can be added to pinned_folios multiple times, depending on how

many

drivers/dma-buf/udmabuf.c +9 -30
+	 * times the folio has been pinned when create.
+	 */
+	pgoff_t nr_pinned;
+	struct folio **pinned_folios;
+
  	struct sg_table *sg;
  	struct miscdevice *device;
  	pgoff_t *offsets;
-	struct list_head unpin_list;
-};
-
-struct udmabuf_folio {
-	struct folio *folio;
-	struct list_head list;
  };
 
  static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
@@ -189,32 +195,12 @@ static void unmap_udmabuf(struct
 dma_buf_attachment *at,
  	return put_sg_table(at->dev, sg, direction);
  }
 
-static void unpin_all_folios(struct list_head *unpin_list)
-{
-	struct udmabuf_folio *ubuf_folio;
-
-	while (!list_empty(unpin_list)) {
-		ubuf_folio = list_first_entry(unpin_list,
-					      struct udmabuf_folio, list);
-		unpin_folio(ubuf_folio->folio);
-
-		list_del(&ubuf_folio->list);
-		kfree(ubuf_folio);
-	}
-}
-
-static int add_to_unpin_list(struct list_head *unpin_list,
-			     struct folio *folio)
+static __always_inline void unpin_all_folios(struct udmabuf *ubuf)
  {
-	struct udmabuf_folio *ubuf_folio;
-
-	ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL);
-	if (!ubuf_folio)
-		return -ENOMEM;
+	pgoff_t i;
 
-	ubuf_folio->folio = folio;
-	list_add_tail(&ubuf_folio->list, unpin_list);
-	return 0;
+	for (i = 0; i < ubuf->nr_pinned; ++i)
+		unpin_folio(ubuf->pinned_folios[i]);
  }
 

static void release_udmabuf(struct dma_buf *buf)

drivers/dma-buf/udmabuf.c static void release_udmabuf(struct dma_buf *buf) · line 215

Quoted 6 lines
drivers/dma-buf/udmabuf.c +2 -1
@@ -225,7 +211,8 @@ static void release_udmabuf(struct dma_buf *buf)
  	if (ubuf->sg)
  		put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
 
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
+	kvfree(ubuf->pinned_folios);

Move the kvfree() of pinned_folios into unpin_all_folios().

Quoted 38 lines
drivers/dma-buf/udmabuf.c +2 -2
  	kvfree(ubuf->offsets);
  	kvfree(ubuf->folios);
  	kfree(ubuf);
@@ -326,9 +313,9 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
  				     struct folio **folios)
  {
  	struct file *memfd = NULL;
-	pgoff_t pgoff, ipgcnt, upgcnt = ubuf->pagecount;
+	pgoff_t pgoff, ipgcnt, upgcnt, nr_pinned;
  	u32 cur_folio, cur_pgcnt;
-	struct folio **ubuf_folios;
+	struct folio **ubuf_folios, **pinned_folios;
  	pgoff_t *ubuf_offsets;
  	long nr_folios;

loff_t end, start;

drivers/dma-buf/udmabuf.c +4 -5
@@ -351,22 +338,21 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
  	}
 
  	cur_pgcnt = 0;
+	nr_pinned = ubuf->nr_pinned;
+	upgcnt = ubuf->pagecount;
  	ubuf_folios = ubuf->folios;
  	ubuf_offsets = ubuf->offsets;
+	pinned_folios = ubuf->pinned_folios;
 
  	for (cur_folio = 0; cur_folio < nr_folios; ++cur_folio) {
  		pgoff_t subpgoff = pgoff;
  		long fsize = folio_size(folios[cur_folio]);
 
-		ret = add_to_unpin_list(&ubuf->unpin_list, folios[cur_folio]);
-		if (ret < 0) {
-			kfree(folios);
-			goto err;
-		}
+		pinned_folios[nr_pinned++] = folios[cur_folio];
 
Quoted 7 lines

drivers/dma-buf/udmabuf.c static int __udmabuf_pin_list_folios(struct udmabuf_create_item *item, · line 355

Quoted 5 lines
drivers/dma-buf/udmabuf.c +2 -2
  		for (; subpgoff < fsize; subpgoff += PAGE_SIZE) {
-			ubuf->folios[upgcnt] = folios[cur_folio];
-			ubuf->offsets[upgcnt] = subpgoff;
+			ubuf_folios[upgcnt] = folios[cur_folio];
+			ubuf_offsets[upgcnt] = subpgoff;

These are unrelated changes that should belong to the previous patch. I suggest moving this patch before the codestyle cleanup patch.

Yes, sorry for this.

Quoted 61 lines

Thanks,
Vivek

Quoted 57 lines
drivers/dma-buf/udmabuf.c +0 -0
  			++upgcnt;
 

if (++cur_pgcnt >= ipgcnt)

drivers/dma-buf/udmabuf.c +1 -2
@@ -381,12 +367,12 @@ static int __udmabuf_pin_list_folios(struct
 udmabuf_create_item *item,
  	}
  end:
  	ubuf->pagecount = upgcnt;
+	ubuf->nr_pinned = nr_pinned;
  	fput(memfd);
 
  	return 0;
 
  err:
-	ubuf->pagecount = upgcnt;
  	if (memfd)
  		fput(memfd);
@@ -407,7 +393,6 @@ static long udmabuf_create(struct miscdevice
 *device,
  	if (!ubuf)
  		return -ENOMEM;
 
-	INIT_LIST_HEAD(&ubuf->unpin_list);
  	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
  	for (i = 0; i < head->count; i++) {

pgoff_t itempgcnt;

drivers/dma-buf/udmabuf.c +8 -0
@@ -442,6 +427,14 @@ static long udmabuf_create(struct miscdevice
 *device,
  		goto err;
  	}
 
+	ubuf->pinned_folios = kvmalloc_array(pgcnt,
+					     sizeof(*ubuf->pinned_folios),
+					     GFP_KERNEL);
+	if (!ubuf->pinned_folios) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
  	folios = kvmalloc_array(max_ipgcnt, sizeof(*folios), GFP_KERNEL);
  	if (!folios) {
  		ret = -ENOMEM;
drivers/dma-buf/udmabuf.c +2 -1
@@ -463,8 +456,9 @@ static long udmabuf_create(struct miscdevice
 *device,
  	return ret;
 
  err:
-	unpin_all_folios(&ubuf->unpin_list);
+	unpin_all_folios(ubuf);
  	kvfree(folios);
+	kvfree(ubuf->pinned_folios);
  	kvfree(ubuf->offsets);
  	kvfree(ubuf->folios);

kfree(ubuf);
--
2.45.2