Skip to contentSkip to conversation

io_uring mm related abuses

PATCH Applied Pavel Begunkov 3 participants 11 messages Jun 24, 2025 – Jun 26, 2025 io_uring

lore
Share & tools
lore
#0 lore raw

v2 cover io_uring mm related abuses

Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.

v2: change patch 1 tags
use folio_page_idx()

Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinning io_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment in io_vec_fill_bvec

 io_uring/rsrc.c | 27 ++++++++++++++++++++-------
 io_uring/rsrc.h |  1 +
 2 files changed, 21 insertions(+), 7 deletions(-)
Signature · 1 line
2.49.0
#1 lore raw

PATCH v2 1/3 io_uring/rsrc: fix folio unpinning

[  108.070381][   T14] kernel BUG at mm/gup.c:71!
[  108.070502][   T14] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  108.123672][   T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025
[  108.127458][   T14] Workqueue: iou_exit io_ring_exit_work
[  108.174205][   T14] Call trace:
[  108.175649][   T14]  sanity_check_pinned_pages+0x7cc/0x7d0 (P)
[  108.178138][   T14]  unpin_user_page+0x80/0x10c
[  108.180189][   T14]  io_release_ubuf+0x84/0xf8
[  108.182196][   T14]  io_free_rsrc_node+0x250/0x57c
[  108.184345][   T14]  io_rsrc_data_free+0x148/0x298
[  108.186493][   T14]  io_sqe_buffers_unregister+0x84/0xa0
[  108.188991][   T14]  io_ring_ctx_free+0x48/0x480
[  108.191057][   T14]  io_ring_exit_work+0x764/0x7d8
[  108.193207][   T14]  process_one_work+0x7e8/0x155c
[  108.195431][   T14]  worker_thread+0x958/0xed8
[  108.197561][   T14]  kthread+0x5fc/0x75c
[  108.199362][   T14]  ret_from_fork+0x10/0x20

We can pin a tail page of a folio, but then io_uring will try to unpin the the head page of the folio. While it should be fine in terms of keeping the page actually alive, but mm folks say it's wrong and triggers a debug warning. Use unpin_user_folio() instead of unpin_user_page*.

Reported-by [email protected]
Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
---
 io_uring/rsrc.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
io_uring/rsrc.c +7 -3
index c592ceace97d..e83a294c718b 100644
@@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)
 	struct io_mapped_ubuf *imu = priv;
 	unsigned int i;
 
-	for (i = 0; i < imu->nr_bvecs; i++)
-		unpin_user_page(imu->bvec[i].bv_page);
+	for (i = 0; i < imu->nr_bvecs; i++) {
+		struct folio *folio = page_folio(imu->bvec[i].bv_page);
+
+		unpin_user_folio(folio, 1);
+	}
 }
 
 static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 	imu->nr_bvecs = nr_pages;
 	ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
 	if (ret) {
-		unpin_user_pages(pages, nr_pages);
+		for (i = 0; i < nr_pages; i++)
+			unpin_user_folio(page_folio(pages[i]), 1);
 		goto done;
 	}
 
Signature · 1 line
2.49.0
Acked-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning

Quoted 53 lines · Pavel Begunkov
[  108.070381][   T14] kernel BUG at mm/gup.c:71!
[  108.070502][   T14] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  108.123672][   T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025
[  108.127458][   T14] Workqueue: iou_exit io_ring_exit_work
[  108.174205][   T14] Call trace:
[  108.175649][   T14]  sanity_check_pinned_pages+0x7cc/0x7d0 (P)
[  108.178138][   T14]  unpin_user_page+0x80/0x10c
[  108.180189][   T14]  io_release_ubuf+0x84/0xf8
[  108.182196][   T14]  io_free_rsrc_node+0x250/0x57c
[  108.184345][   T14]  io_rsrc_data_free+0x148/0x298
[  108.186493][   T14]  io_sqe_buffers_unregister+0x84/0xa0
[  108.188991][   T14]  io_ring_ctx_free+0x48/0x480
[  108.191057][   T14]  io_ring_exit_work+0x764/0x7d8
[  108.193207][   T14]  process_one_work+0x7e8/0x155c
[  108.195431][   T14]  worker_thread+0x958/0xed8
[  108.197561][   T14]  kthread+0x5fc/0x75c
[  108.199362][   T14]  ret_from_fork+0x10/0x20

We can pin a tail page of a folio, but then io_uring will try to unpin the the head page of the folio. While it should be fine in terms of keeping the page actually alive, but mm folks say it's wrong and triggers a debug warning. Use unpin_user_folio() instead of unpin_user_page*.

Reported-by [email protected]
Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
---
  io_uring/rsrc.c | 10 +++++++---
  1 file changed, 7 insertions(+), 3 deletions(-)
io_uring/rsrc.c +5 -2
index c592ceace97d..e83a294c718b 100644
@@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)
  	struct io_mapped_ubuf *imu = priv;
  	unsigned int i;
 
-	for (i = 0; i < imu->nr_bvecs; i++)
-		unpin_user_page(imu->bvec[i].bv_page);
+	for (i = 0; i < imu->nr_bvecs; i++) {
+		struct folio *folio = page_folio(imu->bvec[i].bv_page);
+
+		unpin_user_folio(folio, 1);
+	}
  }
 
  static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,

io_uring/rsrc.c static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, · line 820

Quoted 9 lines
io_uring/rsrc.c +2 -1
@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
  	imu->nr_bvecs = nr_pages;
  	ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
  	if (ret) {
-		unpin_user_pages(pages, nr_pages);
+		for (i = 0; i < nr_pages; i++)
+			unpin_user_folio(page_folio(pages[i]), 1);
  		goto done;
  	}
 

It should fix the issue, but it's a bit suboptimal in the case where we didn't coalesc, but there are folio ranges to coalesc:

unpin_user_pages() does a per-folio coalescing.

So in an ideal world, we would cleanly split both paths, and work with folios after we coalesced to use folios, and work with pages, when we didn't coalesc to use folios.

Then, we can just use unpin_folios() after we coalesced.

In any case, for a fix this is good enough, but probably we can do better later.

Signature · 3 lines
Cheers,

David / dhildenb
#3 lore raw

Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning

On 6/25/25 08:53, David Hildenbrand wrote:
...>>   static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,

Quoted 12 lines

io_uring/rsrc.c static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, · line 819

Quoted 9 lines
io_uring/rsrc.c +2 -1
@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
       imu->nr_bvecs = nr_pages;
       ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
       if (ret) {
-        unpin_user_pages(pages, nr_pages);
+        for (i = 0; i < nr_pages; i++)
+            unpin_user_folio(page_folio(pages[i]), 1);
           goto done;
       }

It should fix the issue, but it's a bit suboptimal in the case where we didn't coalesc, but there are folio ranges to coalesc:

Quoted 3 lines

unpin_user_pages() does a per-folio coalescing.

So in an ideal world, we would cleanly split both paths, and work with folios after we coalesced to use folios, and work with pages, when we didn't coalesc to use folios.

Agreed, but I'm not too much concerned, it's a slow path, users should be mindful while registering memory. Localised hammering on the refcount is not great but ultimately shouldn't matter much.

Quoted 5 lines

Then, we can just use unpin_folios() after we coalesced.

In any case, for a fix this is good enough, but probably we can do better later.

Thanks for the report!

Signature · 1 line
Pavel Begunkov
#4 lore raw

PATCH v2 2/3 io_uring/rsrc: don't rely on user vaddr alignment

In reply to Pavel Begunkov

There is no guaranteed alignment for user pointers, however the calculation of an offset of the first page into a folio after coalescing uses some weird bit mask logic, get rid of it.

Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
---
 io_uring/rsrc.c | 7 ++++++-
 io_uring/rsrc.h | 1 +
 2 files changed, 7 insertions(+), 1 deletion(-)
io_uring/rsrc.c +6 -1
index e83a294c718b..8b06c732d136 100644
@@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
 
 	data->nr_pages_mid = folio_nr_pages(folio);
 	data->folio_shift = folio_shift(folio);
+	data->first_folio_page_idx = folio_page_idx(folio, page_array[0]);
 
 	/*
 	 * Check if pages are contiguous inside a folio, and all folios have
@@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 	if (coalesced)
 		imu->folio_shift = data.folio_shift;
 	refcount_set(&imu->refs, 1);
-	off = (unsigned long) iov->iov_base & ((1UL << imu->folio_shift) - 1);
+
+	off = (unsigned long)iov->iov_base & ~PAGE_MASK;
+	if (coalesced)
+		off += data.first_folio_page_idx << PAGE_SHIFT;
+
 	node->buf = imu;
 	ret = 0;
 
io_uring/rsrc.h +1 -0
index 0d2138f16322..25e7e998dcfd 100644
@@ -49,6 +49,7 @@ struct io_imu_folio_data {
 	unsigned int	nr_pages_mid;
 	unsigned int	folio_shift;
 	unsigned int	nr_folios;
+	unsigned long	first_folio_page_idx;
 };
 
 bool io_rsrc_cache_init(struct io_ring_ctx *ctx);
Signature · 1 line
2.49.0
Acked-by on v2 2/3
#5 lore raw

Re: [PATCH v2 2/3] io_uring/rsrc: don't rely on user vaddr alignment

Quoted 43 lines · Pavel Begunkov

There is no guaranteed alignment for user pointers, however the calculation of an offset of the first page into a folio after coalescing uses some weird bit mask logic, get rid of it.

Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
---
  io_uring/rsrc.c | 7 ++++++-
  io_uring/rsrc.h | 1 +
  2 files changed, 7 insertions(+), 1 deletion(-)
io_uring/rsrc.c +6 -1
index e83a294c718b..8b06c732d136 100644
@@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
 
  	data->nr_pages_mid = folio_nr_pages(folio);
  	data->folio_shift = folio_shift(folio);
+	data->first_folio_page_idx = folio_page_idx(folio, page_array[0]);
 
  	/*
  	 * Check if pages are contiguous inside a folio, and all folios have
@@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
  	if (coalesced)
  		imu->folio_shift = data.folio_shift;
  	refcount_set(&imu->refs, 1);
-	off = (unsigned long) iov->iov_base & ((1UL << imu->folio_shift) - 1);
+
+	off = (unsigned long)iov->iov_base & ~PAGE_MASK;
+	if (coalesced)
+		off += data.first_folio_page_idx << PAGE_SHIFT;
+
  	node->buf = imu;
  	ret = 0;
 
io_uring/rsrc.h +0 -0
index 0d2138f16322..25e7e998dcfd 100644

io_uring/rsrc.h struct io_imu_folio_data { · line 55

Quoted 7 lines
io_uring/rsrc.h +1 -0
@@ -49,6 +49,7 @@ struct io_imu_folio_data {
  	unsigned int	nr_pages_mid;
  	unsigned int	folio_shift;
  	unsigned int	nr_folios;
+	unsigned long	first_folio_page_idx;
  };
 
  bool io_rsrc_cache_init(struct io_ring_ctx *ctx);
Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw

PATCH v2 3/3 io_uring: don't assume uaddr alignment in io_vec_fill_bvec

In reply to Pavel Begunkov

There is no guaranteed alignment for user pointers. Don't use mask trickery and adjust the offset by bv_offset.

Fixes 9ef4cbbcb4ac3 ("io_uring: add infra for importing vectored reg buffers")
---
 io_uring/rsrc.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
io_uring/rsrc.c +7 -3
index 8b06c732d136..c58fe736f297 100644
@@ -1336,7 +1336,6 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter,
 {
 	unsigned long folio_size = 1 << imu->folio_shift;
 	unsigned long folio_mask = folio_size - 1;
-	u64 folio_addr = imu->ubuf & ~folio_mask;
 	struct bio_vec *res_bvec = vec->bvec;
 	size_t total_len = 0;
 	unsigned bvec_idx = 0;
@@ -1358,8 +1357,13 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter,
 		if (unlikely(check_add_overflow(total_len, iov_len, &total_len)))
 			return -EOVERFLOW;
 
-		/* by using folio address it also accounts for bvec offset */
-		offset = buf_addr - folio_addr;
+		offset = buf_addr - imu->ubuf;
+		/*
+		 * Only the first bvec can have non zero bv_offset, account it
+		 * here and work with full folios below.
+		 */
+		offset += imu->bvec[0].bv_offset;
+
 		src_bvec = imu->bvec + (offset >> imu->folio_shift);
 		offset &= folio_mask;
 
Signature · 1 line
2.49.0
maintainer
#7 lore raw
In reply to Pavel Begunkov
Quoted 7 lines · Pavel Begunkov

Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.

v2: change patch 1 tags
use folio_page_idx()

Quoted 8 lines

Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinning io_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment in io_vec_fill_bvec

 io_uring/rsrc.c | 27 ++++++++++++++++++++-------
 io_uring/rsrc.h |  1 +
 2 files changed, 21 insertions(+), 7 deletions(-)

Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!

Signature · 1 line
Jens Axboe
#8 lore raw
Quoted 13 lines · Jens Axboe
Quoted 12 lines · Pavel Begunkov

Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.

v2: change patch 1 tags
use folio_page_idx()

Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinning io_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment in io_vec_fill_bvec

Quoted 6 lines
Quoted 3 lines
  io_uring/rsrc.c | 27 ++++++++++++++++++++-------
  io_uring/rsrc.h |  1 +
  2 files changed, 21 insertions(+), 7 deletions(-)

Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!

Turned to be for-next from a couple of days ago. Patch 1 looks the same, should be fine.

Signature · 1 line
Pavel Begunkov
maintainer
#9 lore raw
Quoted 18 lines · Pavel Begunkov
Quoted 17 lines · Jens Axboe
Quoted 15 lines · Pavel Begunkov

Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.

v2: change patch 1 tags
use folio_page_idx()

Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinning io_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment in io_vec_fill_bvec

  io_uring/rsrc.c | 27 ++++++++++++++++++++-------
  io_uring/rsrc.h |  1 +
  2 files changed, 21 insertions(+), 7 deletions(-)
Quoted 5 lines
Quoted 2 lines

Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!

Turned to be for-next from a couple of days ago. Patch 1 looks the same, should be fine.

I don't always put current fixes in for-next, though I've tried to do it consistently more recently. But it's conflicting with the error path cleanup from about a week ago:

commit e1c75831f682eef0f68b35723437146ed86070b1 (tag: io_uring-6.16-20250619)

Author: Penglei Jiang <[email protected]>
Date:   Tue Jun 17 09:56:44 2025 -0700

io_uring: fix potential page leak in io_sqe_buffer_register()

and my hand-edit just put your hunk 2 of patch 1 into that cleanup path too. Thanks for checking!

Signature · 1 line
Jens Axboe
maintainer Applied
#10 lore raw
In reply to Pavel Begunkov
Quoted 8 lines · Pavel Begunkov

Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.

v2: change patch 1 tags
use folio_page_idx()

[...]

Applied, thanks!

[1/3] io_uring/rsrc: fix folio unpinning

      commit: 5afb4bf9fc62d828647647ec31745083637132e4
[2/3] io_uring/rsrc: don't rely on user vaddr alignment
      commit: 3a3c6d61577dbb23c09df3e21f6f9eda1ecd634b
[3/3] io_uring: don't assume uaddr alignment in io_vec_fill_bvec
      commit: e1d7727b73a1f78035316ac35ee184d477059f0b

Best regards,

Signature · 1 line
Jens Axboe