mm/vmap: map contiguous pages in batches whenever possible
Subsystems (3)
RFC PATCH 1/1 mm/vmap: map contiguous pages in batches whenever possible
From: Barry Song <[email protected]>
In many cases, the pages passed to vmap() may include high-order pages—for example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually—even the pages inside a high-order block are handled one by one. This patch detects high-order pages and maps them as a single contiguous block whenever possible.
Another possibility is to implement a new API, vmap_sg(). However, that change seems to be quite large in scope.
When vmapping a 128MB dma-buf using the systemheap, this RFC appears to make system_heap_do_vmap() 16× faster:
W/ patch:
[ 51.363682] system_heap_do_vmap took 2474000 ns [ 53.307044] system_heap_do_vmap took 2469008 ns [ 55.061985] system_heap_do_vmap took 2519008 ns [ 56.653810] system_heap_do_vmap took 2674000 ns
W/o patch:
[ 8.260880] system_heap_do_vmap took 39490000 ns [ 32.513292] system_heap_do_vmap took 38784000 ns [ 82.673374] system_heap_do_vmap took 40711008 ns [ 84.579062] system_heap_do_vmap took 40236000 ns
--- mm/vmalloc.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-)
mm/vmalloc.c +43 -6
@@ -642,6 +642,34 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,return err; } +static inline int get_vmap_batch_order (struct page **pages, +unsigned int stride, +int max_steps, +unsigned int idx) +{ +/* +* Currently, batching is only supported in vmap_pages_range +* when page_shift == PAGE_SHIFT. +*/ +if (stride !=1 ) +return 0 ; + +struct page *base = pages[idx]; +if (!PageHead (base)) +return 0 ; + +int order =compound_order (base); +int nr_pages =1 << order; + +if (max_steps < nr_pages) +return 0 ; + +for (int i =0 ; i < nr_pages; i++) +if (pages[idx + i] != base + i) +return 0 ; +return order; +} +/* * vmap_pages_range_noflush is similar to vmap_pages_range, but does not * flush caches.
@@ -655,23 +683,32 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned int stride;WARN_ON (page_shift <PAGE_SHIFT ); +/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))return vmap_small_pages_range_noflush (addr, end, prot, pages); -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; + stride =1U << (page_shift -PAGE_SHIFT ); +for (i =0 ; i < nr; ) { +int err, order; - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + order =get_vmap_batch_order (pages, stride, nr - i, i); + err =vmap_range_noflush (addr, addr + (1UL << (page_shift + order)),page_to_phys (pages[i]), prot, - page_shift); + page_shift + order);if (err)return err; - addr +=1UL << page_shift; + addr +=1UL << (page_shift + order); + i +=1U << (order + page_shift -PAGE_SHIFT ); }return 0 ;
Signature · 1 line
2.39.3 (Apple Git-146)
Uladzislau Rezki
maintainerQuoted 83 lines · Barry Song
From: Barry Song <[email protected]>
In many cases, the pages passed to
vmap()may include high-order pages—for example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently,vmap()iterates over every page individually—even the pages inside a high-order block are handled one by one. This patch detects high-order pages and maps them as a single contiguous block whenever possible.Another possibility is to implement a new API,
vmap_sg(). However, that change seems to be quite large in scope.When vmapping a 128MB dma-buf using the systemheap, this RFC appears to make
system_heap_do_vmap()16× faster:W/ patch:
[ 51.363682] system_heap_do_vmap took 2474000 ns [ 53.307044] system_heap_do_vmap took 2469008 ns [ 55.061985] system_heap_do_vmap took 2519008 ns [ 56.653810] system_heap_do_vmap took 2674000 nsW/o patch:
[ 8.260880] system_heap_do_vmap took 39490000 ns [ 32.513292] system_heap_do_vmap took 38784000 ns [ 82.673374] system_heap_do_vmap took 40711008 ns [ 84.579062] system_heap_do_vmap took 40236000 nsCc Uladzislau Rezki <[email protected]>Cc Sumit Semwal <[email protected]>Cc John Stultz <[email protected]>Cc Maxime Ripard <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/vmalloc.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-)mm/vmalloc.c +29
-0@@ -642,6 +642,34 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,return err; } +static inline int get_vmap_batch_order (struct page **pages, +unsigned int stride, +int max_steps, +unsigned int idx) +{ +/* +* Currently, batching is only supported in vmap_pages_range +* when page_shift == PAGE_SHIFT. +*/ +if (stride !=1 ) +return 0 ; + +struct page *base = pages[idx]; +if (!PageHead (base)) +return 0 ; + +int order =compound_order (base); +int nr_pages =1 << order; + +if (max_steps < nr_pages) +return 0 ; + +for (int i =0 ; i < nr_pages; i++) +if (pages[idx + i] != base + i) +return 0 ; +return order; +} +/* * vmap_pages_range_noflush is similar to vmap_pages_range, but does not * flush caches. @@ -655,23 +683,32 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned int stride;WARN_ON (page_shift <PAGE_SHIFT );
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 697
Quoted 10 lines
mm/vmalloc.c +6
-1+/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))
Do we support __GFP_COMP as vmalloc/vmap flag? As i see from latest:
/*
* See __vmalloc_node_range() for a clear list of supported vmalloc flags.
* This gfp lists all flags currently passed through vmalloc. Currently,
* __GFP_ZERO is used by BPF and __GFP_NORETRY is used by percpu. Both drm
* and BPF also use GFP_USER. Additionally, various users pass
* GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP.
*/
#define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
__GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\
GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
GFP_USER | __GFP_NOLOCKDEP)Could you please clarify when PageCompound(pages[0]) returns true?
Quoted 26 lines
mm/vmalloc.c +8
-5return vmap_small_pages_range_noflush (addr, end, prot, pages); -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; + stride =1U << (page_shift -PAGE_SHIFT ); +for (i =0 ; i < nr; ) { +int err, order; - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + order =get_vmap_batch_order (pages, stride, nr - i, i); + err =vmap_range_noflush (addr, addr + (1UL << (page_shift + order)),page_to_phys (pages[i]), prot, - page_shift); + page_shift + order);if (err)return err; - addr +=1UL << page_shift; + addr +=1UL << (page_shift + order); + i +=1U << (order + page_shift -PAGE_SHIFT ); }return 0;
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 11 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 10 lines
mm/vmalloc.c +6
-1+/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))Do we support
__GFP_COMPas vmalloc/vmap flag? As i see from latest:
This is not the case for vmalloc, but applies to dma-bufs that are allocated using alloc_pages() with GFP_COMP.
#define LOW_ORDER_GFP (GFP_HIGHUSER | __GFP_ZERO)
#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
| __GFP_NORETRY) & ~__GFP_RECLAIM) \
| __GFP_COMP)Quoted 8 lines
/*
* See__vmalloc_node_range()for a clear list of supported vmalloc flags.
* This gfp lists all flags currently passed through vmalloc. Currently,
*__GFP_ZEROis used by BPF and__GFP_NORETRYis used by percpu. Both drm
* and BPF also useGFP_USER. Additionally, various users pass
*GFP_KERNEL_ACCOUNT. Xfs uses__GFP_NOLOCKDEP.
*/
Quoted 7 lines
#define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)Could you please clarify when
PageCompound(pages[0])returns true?
In this case, dma-buf attempts to allocate as many compound high-order pages as possible, falling back to 0-order allocations if necessary.
Then, dma_buf_vmap() is called by the GPU drivers:
1 404 drivers/accel/amdxdna/amdxdna_gem.c <<amdxdna_gem_obj_vmap>>
dma_buf_vmap(abo->dma_buf, map);
2 1568 drivers/dma-buf/dma-buf.c <<dma_buf_vmap_unlocked>>
ret = dma_buf_vmap(dmabuf, map);
3 354 drivers/gpu/drm/drm_gem_shmem_helper.c
<<drm_gem_shmem_vmap_locked>>
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
4 85 drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c
<<etnaviv_gem_prime_vmap_impl>>
ret = dma_buf_vmap(etnaviv_obj->base.import_attach->dmabuf, &map);
5 433 drivers/gpu/drm/vmwgfx/vmwgfx_blit.c <<map_external>>
ret = dma_buf_vmap(bo->tbo.base.dma_buf, map);
6 88 drivers/gpu/drm/vmwgfx/vmwgfx_gem.c <<vmw_gem_vmap>>
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);Thanks
Barry
Uladzislau Rezki
maintainerQuoted 33 lines · Barry Song
Quoted 11 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 10 lines
mm/vmalloc.c +6
-1+/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))Do we support
__GFP_COMPas vmalloc/vmap flag? As i see from latest:This is not the case for vmalloc, but applies to dma-bufs that are allocated using
alloc_pages()withGFP_COMP.#define
LOW_ORDER_GFP(GFP_HIGHUSER|__GFP_ZERO)#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ | __GFP_NORETRY) & ~__GFP_RECLAIM) \ | __GFP_COMP)Quoted 13 lines
/*
* See__vmalloc_node_range()for a clear list of supported vmalloc flags.
* This gfp lists all flags currently passed through vmalloc. Currently,
*__GFP_ZEROis used by BPF and__GFP_NORETRYis used by percpu. Both drm
* and BPF also useGFP_USER. Additionally, various users pass
*GFP_KERNEL_ACCOUNT. Xfs uses__GFP_NOLOCKDEP.
*/#define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
Quoted 6 lines
Quoted 2 lines
Could you please clarify when
PageCompound(pages[0])returns true?In this case, dma-buf attempts to allocate as many compound high-order pages as possible, falling back to 0-order allocations if necessary.
OK, it is folio who uses it.
Quoted 17 lines
Then,
dma_buf_vmap()is called by the GPU drivers:1 404
drivers/accel/amdxdna/amdxdna_gem.c<<amdxdna_gem_obj_vmap>>dma_buf_vmap(abo->dma_buf, map); 2 1568 drivers/dma-buf/dma-buf.c <<dma_buf_vmap_unlocked>> ret = dma_buf_vmap(dmabuf, map); 3 354 drivers/gpu/drm/drm_gem_shmem_helper.c <<drm_gem_shmem_vmap_locked>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map); 4 85 drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c <<etnaviv_gem_prime_vmap_impl>> ret = dma_buf_vmap(etnaviv_obj->base.import_attach->dmabuf, &map); 5 433 drivers/gpu/drm/vmwgfx/vmwgfx_blit.c <<map_external>> ret = dma_buf_vmap(bo->tbo.base.dma_buf, map); 6 88 drivers/gpu/drm/vmwgfx/vmwgfx_gem.c <<vmw_gem_vmap>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
Thank you for clarification. That would be good to reflect it in the commit message. Also, please note that:
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 4 lines
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||mm/vmalloc.c +1
-1- page_shift == PAGE_SHIFT) + (page_shift == PAGE_SHIFT && !PageCompound(pages[0])))
we rely on page_shift == PAGE_SHIFT condition for the non-sleep vmalloc() allocations(GFP_ATOMIC, GFP_NOWAIT), so we go via vmap_small_pages_range_noflush() path. Your patch adds !PageCompound(pages[0]) also. It is not a problem since it is vmap() path but we need to comment that.
--
Uladzislau Rezki
Quoted 60 lines · Uladzislau Rezki
Quoted 33 lines · Barry Song
Quoted 11 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 10 lines
mm/vmalloc.c +6
-1+/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))Do we support
__GFP_COMPas vmalloc/vmap flag? As i see from latest:This is not the case for vmalloc, but applies to dma-bufs that are allocated using
alloc_pages()withGFP_COMP.#define
LOW_ORDER_GFP(GFP_HIGHUSER|__GFP_ZERO)#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ | __GFP_NORETRY) & ~__GFP_RECLAIM) \ | __GFP_COMP)Quoted 13 lines
/*
* See__vmalloc_node_range()for a clear list of supported vmalloc flags.
* This gfp lists all flags currently passed through vmalloc. Currently,
*__GFP_ZEROis used by BPF and__GFP_NORETRYis used by percpu. Both drm
* and BPF also useGFP_USER. Additionally, various users pass
*GFP_KERNEL_ACCOUNT. Xfs uses__GFP_NOLOCKDEP.
*/#define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)Quoted 6 lines
Quoted 2 lines
Could you please clarify when
PageCompound(pages[0])returns true?In this case, dma-buf attempts to allocate as many compound high-order pages as possible, falling back to 0-order allocations if necessary.
OK, it is folio who uses it.
Quoted 17 lines
Then,
dma_buf_vmap()is called by the GPU drivers:1 404
drivers/accel/amdxdna/amdxdna_gem.c<<amdxdna_gem_obj_vmap>>dma_buf_vmap(abo->dma_buf, map); 2 1568 drivers/dma-buf/dma-buf.c <<dma_buf_vmap_unlocked>> ret = dma_buf_vmap(dmabuf, map); 3 354 drivers/gpu/drm/drm_gem_shmem_helper.c <<drm_gem_shmem_vmap_locked>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map); 4 85 drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c <<etnaviv_gem_prime_vmap_impl>> ret = dma_buf_vmap(etnaviv_obj->base.import_attach->dmabuf, &map); 5 433 drivers/gpu/drm/vmwgfx/vmwgfx_blit.c <<map_external>> ret = dma_buf_vmap(bo->tbo.base.dma_buf, map); 6 88 drivers/gpu/drm/vmwgfx/vmwgfx_gem.c <<vmw_gem_vmap>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
Quoted 2 lines
Thank you for clarification. That would be good to reflect it in the commit message. Also, please note that:
Sure.
Quoted 9 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 5 lines
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||mm/vmalloc.c +1
-1- page_shift == PAGE_SHIFT) + (page_shift == PAGE_SHIFT && !PageCompound(pages[0])))we rely on
page_shift==PAGE_SHIFTcondition for the non-sleepvmalloc()allocations(GFP_ATOMIC, GFP_NOWAIT), so we go viavmap_small_pages_range_noflush()path. Your patch adds !PageCompound(pages[0]) also. It is not a problem since it isvmap()path but we need to comment that.
Sure. Would the following work?
/*
* For vmap(), users may allocate pages from high orders down
to order 0,
* while always using PAGE_SHIFT as the page_shift.
* We first check whether the initial page is a compound page. If so,
* there may be an opportunity to batch multiple pages together.
*/
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
(page_shift == PAGE_SHIFT && !PageCompound(pages[0])))
return vmap_small_pages_range_noflush(addr, end, prot, pages);Thanks
Barry
Uladzislau Rezki
maintainerQuoted 81 lines · Barry Song
Quoted 60 lines · Uladzislau Rezki
Quoted 33 lines · Barry Song
Quoted 11 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 10 lines
mm/vmalloc.c +6
-1+/* +* Some users may allocate pages from high-order down to order 0. +* We roughly check if the first page is a compound page. If so, +* there is a chance to batch multiple pages together. +*/ if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) + (page_shift ==PAGE_SHIFT && !PageCompound (pages[0 ])))Do we support
__GFP_COMPas vmalloc/vmap flag? As i see from latest:This is not the case for vmalloc, but applies to dma-bufs that are allocated using
alloc_pages()withGFP_COMP.#define
LOW_ORDER_GFP(GFP_HIGHUSER|__GFP_ZERO)#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ | __GFP_NORETRY) & ~__GFP_RECLAIM) \ | __GFP_COMP)Quoted 13 lines
/*
* See__vmalloc_node_range()for a clear list of supported vmalloc flags.
* This gfp lists all flags currently passed through vmalloc. Currently,
*__GFP_ZEROis used by BPF and__GFP_NORETRYis used by percpu. Both drm
* and BPF also useGFP_USER. Additionally, various users pass
*GFP_KERNEL_ACCOUNT. Xfs uses__GFP_NOLOCKDEP.
*/#define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)Quoted 6 lines
Quoted 2 lines
Could you please clarify when
PageCompound(pages[0])returns true?In this case, dma-buf attempts to allocate as many compound high-order pages as possible, falling back to 0-order allocations if necessary.
OK, it is folio who uses it.
Quoted 17 lines
Then,
dma_buf_vmap()is called by the GPU drivers:1 404
drivers/accel/amdxdna/amdxdna_gem.c<<amdxdna_gem_obj_vmap>>dma_buf_vmap(abo->dma_buf, map); 2 1568 drivers/dma-buf/dma-buf.c <<dma_buf_vmap_unlocked>> ret = dma_buf_vmap(dmabuf, map); 3 354 drivers/gpu/drm/drm_gem_shmem_helper.c <<drm_gem_shmem_vmap_locked>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map); 4 85 drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c <<etnaviv_gem_prime_vmap_impl>> ret = dma_buf_vmap(etnaviv_obj->base.import_attach->dmabuf, &map); 5 433 drivers/gpu/drm/vmwgfx/vmwgfx_blit.c <<map_external>> ret = dma_buf_vmap(bo->tbo.base.dma_buf, map); 6 88 drivers/gpu/drm/vmwgfx/vmwgfx_gem.c <<vmw_gem_vmap>> ret = dma_buf_vmap(obj->import_attach->dmabuf, map);Quoted 2 lines
Thank you for clarification. That would be good to reflect it in the commit message. Also, please note that:
Sure.
Quoted 9 lines
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 696
Quoted 5 lines
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||mm/vmalloc.c +1
-1- page_shift == PAGE_SHIFT) + (page_shift == PAGE_SHIFT && !PageCompound(pages[0])))we rely on
page_shift==PAGE_SHIFTcondition for the non-sleepvmalloc()allocations(GFP_ATOMIC, GFP_NOWAIT), so we go viavmap_small_pages_range_noflush()path. Your patch adds !PageCompound(pages[0]) also. It is not a problem since it isvmap()path but we need to comment that.Sure. Would the following work?
/*
* Forvmap(), users may allocate pages from high orders down
to order 0,
Quoted 8 lines
* while always using PAGE_SHIFT as the page_shift.
* We first check whether the initial page is a compound page. If so,
* there may be an opportunity to batch multiple pages together.
*/
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
(page_shift == PAGE_SHIFT && !PageCompound(pages[0])))
return vmap_small_pages_range_noflush(addr, end, prot, pages);Sounds good!
Thank you.
--
Uladzislau Rezki
Quoted 42 lines · Barry Song
From: Barry Song <[email protected]>
In many cases, the pages passed to
vmap()may include high-order pages—for example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently,vmap()iterates over every page individually—even the pages inside a high-order block are handled one by one. This patch detects high-order pages and maps them as a single contiguous block whenever possible.Another possibility is to implement a new API,
vmap_sg(). However, that change seems to be quite large in scope.When vmapping a 128MB dma-buf using the systemheap, this RFC appears to make
system_heap_do_vmap()16× faster:W/ patch:
[ 51.363682] system_heap_do_vmap took 2474000 ns [ 53.307044] system_heap_do_vmap took 2469008 ns [ 55.061985] system_heap_do_vmap took 2519008 ns [ 56.653810] system_heap_do_vmap took 2674000 nsW/o patch:
[ 8.260880] system_heap_do_vmap took 39490000 ns [ 32.513292] system_heap_do_vmap took 38784000 ns [ 82.673374] system_heap_do_vmap took 40711008 ns [ 84.579062] system_heap_do_vmap took 40236000 nsCc Uladzislau Rezki <[email protected]>Cc Sumit Semwal <[email protected]>Cc John Stultz <[email protected]>Cc Maxime Ripard <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/vmalloc.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-)mm/vmalloc.c +0
-0
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 648
Quoted 7 lines
mm/vmalloc.c +4
-0@@ -642,6 +642,34 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,return err; } +static inline int get_vmap_batch_order (struct page **pages, +unsigned int stride, +int max_steps, +unsigned int idx)
These fit into less lines.
ideally
\t\tunsigned int stride, int max_steps, unsigned int idx)
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 649
Quoted 1 line
mm/vmalloc.c +1
-0+{
int order, nr_pages, i;
struct page *base;
But I think you can just drop "base". And order.
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 662
Quoted 13 lines
mm/vmalloc.c +13
-0+/* +* Currently, batching is only supported in vmap_pages_range +* when page_shift == PAGE_SHIFT. +*/ +if (stride !=1 ) +return 0 ; + +struct page *base = pages[idx]; +if (!PageHead (base)) +return 0 ; + +int order =compound_order (base); +int nr_pages =1 << order;
You can drop the head check etc and simply do
nr_pages = compound_nr(pages[idx]);
if (nr_pages == 1)
return 0;
Which raises the question: are these things folios? I assume not.
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 669
Quoted 7 lines
mm/vmalloc.c +7
-0+ +if (max_steps < nr_pages) +return 0 ; + +for (int i =0 ; i < nr_pages; i++) +if (pages[idx + i] != base + i) +return 0 ;
if (num_pages_contiguous(&pages[idx], nr_pages) == nr_pages)
return compound_order(pages[idx]);
return 0;
Signature · 3 lines
Cheers David
On Mon, Dec 1, 2025 at 6:36 PM David Hildenbrand (Red Hat) <[email protected]> wrote:
[...]
Quoted 16 lines
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 648
Quoted 13 lines
mm/vmalloc.c +4
-0@@ -642,6 +642,34 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,return err; } +static inline int get_vmap_batch_order (struct page **pages, +unsigned int stride, +int max_steps, +unsigned int idx)These fit into less lines.
Quoted 3 lines
ideally
\t\tunsigned int stride, int
max_steps, unsigned int idx)
Right, thanks!
Quoted 7 lines
mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 649
Quoted 2 lines
mm/vmalloc.c +1
-0+{int order,
nr_pages, i;
struct page *base;But I think you can just drop "base". And order.
Right, thanks!
Quoted 18 lines
Quoted 4 lines
mm/vmalloc.c +3
-0+/* +* Currently, batching is only supported in vmap_pages_range +* when page_shift == PAGE_SHIFT. mm/vmalloc.c static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, · line 662
Quoted 10 lines
mm/vmalloc.c +10
-0+*/ +if (stride !=1 ) +return 0 ; + +struct page *base = pages[idx]; +if (!PageHead (base)) +return 0 ; + +int order =compound_order (base); +int nr_pages =1 << order;You can drop the head check etc and simply do
Quoted 4 lines
nr_pages=compound_nr(pages[idx]);
if (nr_pages== 1)return 0;
Nice. Since compound_nr() returns 1 for tail pages.
Quoted 1 line
Which raises the question: are these things folios? I assume not.
In my case, it’s simply alloc_pages with GFP_COMP. I assume that folios allocated via folio_alloc() would also automatically benefit from this patch?
Currently, vmap() takes a pages array as an argument. So even for a folio, we need to expand it into individual pages. Simply passing a folios array to vmalloc likely won’t work, since vmap() could start and end at subpages in the middle of a folio.
Thanks
Barry