io_uring/memmap: fix non-compound alloc fallback with memcg accounting
PATCH 1/1 io_uring/memmap: fix non-compound alloc fallback with memcg accounting
io_region_allocate_pages() first tries io_mem_alloc_compound(). If that fails, it uses alloc_pages_bulk_node() to allocate separate pages.
The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is unconditionally supplied, so when memcg_kmem_online() is true, it only allocates one page. For a multi-page request, io_uring treats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.
When this happens, fill the rest of the request with order 0 allocations via alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.
--- io_uring/memmap.c | 8 ++++++++ 1 file changed, 8 insertions(+)
io_uring/memmap.c +8 -0
@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,size_t size =io_region_size (mr);unsigned long nr_allocated;struct page **pages; +struct page *page; pages =kvmalloc_objs (*pages, mr->nr_pages, gfp);if (!pages)
@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr, nr_allocated =alloc_pages_bulk_node (gfp,NUMA_NO_NODE , mr->nr_pages, pages); +while (nr_allocated < mr->nr_pages) { + page =alloc_page (gfp); +if (!page) +break ; + + pages[nr_allocated++] = page; + }if (nr_allocated != mr->nr_pages) {if (nr_allocated)release_pages (pages, nr_allocated);
Signature · 1 line
2.53.0-Meta
Quoted 32 lines · David Wei
io_region_allocate_pages()first triesio_mem_alloc_compound(). If that fails, it usesalloc_pages_bulk_node()to allocate separate pages.The bulk allocator does not support memcg accounting.
__GFP_ACCOUNTis unconditionally supplied, so whenmemcg_kmem_online()is true, it only allocates one page. For a multi-page request,io_uringtreats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.When this happens, fill the rest of the request with order 0 allocations via
alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.Signed-off-by David Wei <[email protected]>--- io_uring/memmap.c | 8 ++++++++ 1 file changed, 8 insertions(+)io_uring/memmap.c +1
-0@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,size_t size =io_region_size (mr);unsigned long nr_allocated;struct page **pages; +struct page *page; pages =kvmalloc_objs (*pages, mr->nr_pages, gfp);if (!pages)
io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 205
Quoted 10 lines
io_uring/memmap.c +7
-0@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr, nr_allocated =alloc_pages_bulk_node (gfp,NUMA_NO_NODE , mr->nr_pages, pages); +while (nr_allocated < mr->nr_pages) { + page =alloc_page (gfp); +if (!page) +break ; + + pages[nr_allocated++] = page; + }
FWIW, I looked at this recently and really think this should be the alloc_pages_bulk_/kmem_alloc_bulk* actual semantics. But that is a bigger sell and I only found io_uring would actually benefit from it.
Feel free to add:
Perhaps also
Quoted 6 lines
io_uring/memmap.c +0
-0if (nr_allocated != mr->nr_pages) {if (nr_allocated)release_pages (pages, nr_allocated);Signature · 1 line
2.53.0-Meta
Signature · 1 line
Gabriel Krisman Bertazi
Quoted 49 lines · Gabriel Krisman Bertazi
Quoted 32 lines · David Wei
io_region_allocate_pages()first triesio_mem_alloc_compound(). If that fails, it usesalloc_pages_bulk_node()to allocate separate pages.The bulk allocator does not support memcg accounting.
__GFP_ACCOUNTis unconditionally supplied, so whenmemcg_kmem_online()is true, it only allocates one page. For a multi-page request,io_uringtreats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.When this happens, fill the rest of the request with order 0 allocations via
alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.Signed-off-by David Wei <[email protected]>--- io_uring/memmap.c | 8 ++++++++ 1 file changed, 8 insertions(+)io_uring/memmap.c +1
-0@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,size_t size =io_region_size (mr);unsigned long nr_allocated;struct page **pages; +struct page *page; pages =kvmalloc_objs (*pages, mr->nr_pages, gfp);if (!pages)io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 205
Quoted 10 lines
io_uring/memmap.c +7
-0@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr, nr_allocated =alloc_pages_bulk_node (gfp,NUMA_NO_NODE , mr->nr_pages, pages); +while (nr_allocated < mr->nr_pages) { + page =alloc_page (gfp); +if (!page) +break ; + + pages[nr_allocated++] = page; + }FWIW, I looked at this recently and really think this should be the alloc_pages_bulk_/kmem_alloc_bulk* actual semantics. But that is a bigger sell and I only found
io_uringwould actually benefit from it.
Quoted 7 lines
Feel free to add:
Reviewed-by Gabriel Krisman Bertazi <[email protected]>Perhaps also
Fixes 1e21df691ffa ("io_uring/memmap: implement kernel allocated regions")
Hi Gabriel, thanks for your review. Yes, I had also considered changing alloc_pages_bulk_*(), but as you said that is a much bigger uplift. I thought I'd go for a smaller surgical fix first, unless there was an appetite for the larger change from maintainers.
Sorry for missing the Fixes tag. I'll add it if this requires another spin, otherwise Jens please add it when you apply.
Quoted 10 lines · David Wei
io_region_allocate_pages()first triesio_mem_alloc_compound(). If that fails, it usesalloc_pages_bulk_node()to allocate separate pages.The bulk allocator does not support memcg accounting.
__GFP_ACCOUNTis unconditionally supplied, so whenmemcg_kmem_online()is true, it only allocates one page. For a multi-page request,io_uringtreats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.[...]
Applied, thanks!
[1/1] io_uring/memmap: fix non-compound alloc fallback with memcg accounting
commit: 68477e4c562e3ec1333f64e010d75b3a257e7997
Best regards,
Signature · 1 line
Jens Axboe