Skip to contentSkip to conversation

io_uring/memmap: fix non-compound alloc fallback with memcg accounting

PATCH Applied David Wei 3 participants 4 messages Oct 1 – Oct 2 io_uring

lore
Share & tools
lore

David Wei

author
#0 lore raw

PATCH 1/1 io_uring/memmap: fix non-compound alloc fallback with memcg accounting

io_region_allocate_pages() first tries io_mem_alloc_compound(). If that fails, it uses alloc_pages_bulk_node() to allocate separate pages.

The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is unconditionally supplied, so when memcg_kmem_online() is true, it only allocates one page. For a multi-page request, io_uring treats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.

When this happens, fill the rest of the request with order 0 allocations via alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.

---
 io_uring/memmap.c | 8 ++++++++
 1 file changed, 8 insertions(+)
io_uring/memmap.c +8 -0
index cb3db32e8255..cc8ac129a0c7 100644
@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
 	size_t size = io_region_size(mr);
 	unsigned long nr_allocated;
 	struct page **pages;
+	struct page *page;
 
 	pages = kvmalloc_objs(*pages, mr->nr_pages, gfp);
 	if (!pages)
@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
 
 	nr_allocated = alloc_pages_bulk_node(gfp, NUMA_NO_NODE,
 					     mr->nr_pages, pages);
+	while (nr_allocated < mr->nr_pages) {
+		page = alloc_page(gfp);
+		if (!page)
+			break;
+
+		pages[nr_allocated++] = page;
+	}
 	if (nr_allocated != mr->nr_pages) {
 		if (nr_allocated)
 			release_pages(pages, nr_allocated);
Signature · 1 line
2.53.0-Meta
Reviewed-by on v1 1/1
#1 lore raw
Quoted 32 lines · David Wei

io_region_allocate_pages() first tries io_mem_alloc_compound(). If that fails, it uses alloc_pages_bulk_node() to allocate separate pages.

The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is unconditionally supplied, so when memcg_kmem_online() is true, it only allocates one page. For a multi-page request, io_uring treats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.

When this happens, fill the rest of the request with order 0 allocations via alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.

---
 io_uring/memmap.c | 8 ++++++++
 1 file changed, 8 insertions(+)
io_uring/memmap.c +1 -0
index cb3db32e8255..cc8ac129a0c7 100644
@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
 	size_t size = io_region_size(mr);
 	unsigned long nr_allocated;
 	struct page **pages;
+	struct page *page;
 
 	pages = kvmalloc_objs(*pages, mr->nr_pages, gfp);
 	if (!pages)

io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 205

Quoted 10 lines
io_uring/memmap.c +7 -0
@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
 
 	nr_allocated = alloc_pages_bulk_node(gfp, NUMA_NO_NODE,
 					     mr->nr_pages, pages);
+	while (nr_allocated < mr->nr_pages) {
+		page = alloc_page(gfp);
+		if (!page)
+			break;
+
+		pages[nr_allocated++] = page;
+	}

FWIW, I looked at this recently and really think this should be the alloc_pages_bulk_/kmem_alloc_bulk* actual semantics. But that is a bigger sell and I only found io_uring would actually benefit from it.

Feel free to add:

Perhaps also

Fixes 1e21df691ffa ("io_uring/memmap: implement kernel allocated regions")
Quoted 6 lines
io_uring/memmap.c +0 -0
 	if (nr_allocated != mr->nr_pages) {
 		if (nr_allocated)
 			release_pages(pages, nr_allocated);
Signature · 1 line
2.53.0-Meta
Signature · 1 line
Gabriel Krisman Bertazi
author
#2 lore raw
Quoted 49 lines · Gabriel Krisman Bertazi
Quoted 32 lines · David Wei

io_region_allocate_pages() first tries io_mem_alloc_compound(). If that fails, it uses alloc_pages_bulk_node() to allocate separate pages.

The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is unconditionally supplied, so when memcg_kmem_online() is true, it only allocates one page. For a multi-page request, io_uring treats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.

When this happens, fill the rest of the request with order 0 allocations via alloc_page(), which support memcg accounting. This maintains the existing fast path, while fixing the incorrect ENOMEM return when memcg is used.

---
  io_uring/memmap.c | 8 ++++++++
  1 file changed, 8 insertions(+)
io_uring/memmap.c +1 -0
index cb3db32e8255..cc8ac129a0c7 100644
@@ -183,6 +183,7 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
  	size_t size = io_region_size(mr);
  	unsigned long nr_allocated;
  	struct page **pages;
+	struct page *page;
 
  	pages = kvmalloc_objs(*pages, mr->nr_pages, gfp);
  	if (!pages)

io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 205

Quoted 10 lines
io_uring/memmap.c +7 -0
@@ -195,6 +196,13 @@ static int io_region_allocate_pages(struct io_mapped_region *mr,
 
  	nr_allocated = alloc_pages_bulk_node(gfp, NUMA_NO_NODE,
  					     mr->nr_pages, pages);
+	while (nr_allocated < mr->nr_pages) {
+		page = alloc_page(gfp);
+		if (!page)
+			break;
+
+		pages[nr_allocated++] = page;
+	}

FWIW, I looked at this recently and really think this should be the alloc_pages_bulk_/kmem_alloc_bulk* actual semantics. But that is a bigger sell and I only found io_uring would actually benefit from it.

Quoted 7 lines

Feel free to add:

Perhaps also

Fixes 1e21df691ffa ("io_uring/memmap: implement kernel allocated regions")

Hi Gabriel, thanks for your review. Yes, I had also considered changing alloc_pages_bulk_*(), but as you said that is a much bigger uplift. I thought I'd go for a smaller surgical fix first, unless there was an appetite for the larger change from maintainers.

Sorry for missing the Fixes tag. I'll add it if this requires another spin, otherwise Jens please add it when you apply.

maintainer Applied
#3 lore raw
In reply to David Wei
Quoted 10 lines · David Wei

io_region_allocate_pages() first tries io_mem_alloc_compound(). If that fails, it uses alloc_pages_bulk_node() to allocate separate pages.

The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is unconditionally supplied, so when memcg_kmem_online() is true, it only allocates one page. For a multi-page request, io_uring treats this short result as failure and returns ENOMEM, even when the memcg has enough space for all pages.

[...]

Applied, thanks!

[1/1] io_uring/memmap: fix non-compound alloc fallback with memcg accounting

      commit: 68477e4c562e3ec1333f64e010d75b3a257e7997

Best regards,

Signature · 1 line
Jens Axboe