io_uring mm related abuses
v2 cover io_uring mm related abuses
Patch 1 uses unpin_user_folio instead of the page variant. Patches 2-3 make sure io_uring doesn't make any assumptions about user pointer alignments.
v2: change patch 1 tags
use folio_page_idx()
Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinning io_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment in io_vec_fill_bvec
io_uring/rsrc.c | 27 ++++++++++++++++++++------- io_uring/rsrc.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)
Signature · 1 line
2.49.0
PATCH v2 1/3 io_uring/rsrc: fix folio unpinning
[ 108.070381][ T14] kernel BUG at mm/gup.c:71! [ 108.070502][ T14] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 108.123672][ T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025 [ 108.127458][ T14] Workqueue: iou_exit io_ring_exit_work [ 108.174205][ T14] Call trace: [ 108.175649][ T14] sanity_check_pinned_pages+0x7cc/0x7d0 (P) [ 108.178138][ T14] unpin_user_page+0x80/0x10c [ 108.180189][ T14] io_release_ubuf+0x84/0xf8 [ 108.182196][ T14] io_free_rsrc_node+0x250/0x57c [ 108.184345][ T14] io_rsrc_data_free+0x148/0x298 [ 108.186493][ T14] io_sqe_buffers_unregister+0x84/0xa0 [ 108.188991][ T14] io_ring_ctx_free+0x48/0x480 [ 108.191057][ T14] io_ring_exit_work+0x764/0x7d8 [ 108.193207][ T14] process_one_work+0x7e8/0x155c [ 108.195431][ T14] worker_thread+0x958/0xed8 [ 108.197561][ T14] kthread+0x5fc/0x75c [ 108.199362][ T14] ret_from_fork+0x10/0x20
We can pin a tail page of a folio, but then io_uring will try to unpin the the head page of the folio. While it should be fine in terms of keeping the page actually alive, but mm folks say it's wrong and triggers a debug warning. Use unpin_user_folio() instead of unpin_user_page*.
--- io_uring/rsrc.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
io_uring/rsrc.c +7 -3
@@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)struct io_mapped_ubuf *imu = priv;unsigned int i; -for (i =0 ; i < imu->nr_bvecs; i++) -unpin_user_page (imu->bvec[i].bv_page); +for (i =0 ; i < imu->nr_bvecs; i++) { +struct folio *folio =page_folio (imu->bvec[i].bv_page); + +unpin_user_folio (folio,1 ); + } } static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->nr_bvecs = nr_pages; ret =io_buffer_account_pin (ctx, pages, nr_pages, imu, last_hpage);if (ret) { -unpin_user_pages (pages, nr_pages); +for (i =0 ; i < nr_pages; i++) +unpin_user_folio (page_folio (pages[i]),1 ); goto done; }
Signature · 1 line
2.49.0
Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning
Quoted 53 lines · Pavel Begunkov
[ 108.070381][ T14] kernel BUG at mm/gup.c:71! [ 108.070502][ T14] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 108.123672][ T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025 [ 108.127458][ T14] Workqueue: iou_exit io_ring_exit_work [ 108.174205][ T14] Call trace: [ 108.175649][ T14] sanity_check_pinned_pages+0x7cc/0x7d0 (P) [ 108.178138][ T14] unpin_user_page+0x80/0x10c [ 108.180189][ T14] io_release_ubuf+0x84/0xf8 [ 108.182196][ T14] io_free_rsrc_node+0x250/0x57c [ 108.184345][ T14] io_rsrc_data_free+0x148/0x298 [ 108.186493][ T14] io_sqe_buffers_unregister+0x84/0xa0 [ 108.188991][ T14] io_ring_ctx_free+0x48/0x480 [ 108.191057][ T14] io_ring_exit_work+0x764/0x7d8 [ 108.193207][ T14] process_one_work+0x7e8/0x155c [ 108.195431][ T14] worker_thread+0x958/0xed8 [ 108.197561][ T14] kthread+0x5fc/0x75c [ 108.199362][ T14] ret_from_fork+0x10/0x20We can pin a tail page of a folio, but then
io_uringwill try to unpin the the head page of the folio. While it should be fine in terms of keeping the page actually alive, but mm folks say it's wrong and triggers a debug warning. Useunpin_user_folio()instead of unpin_user_page*.Debugged-by David Hildenbrand <[email protected]>Reported-by [email protected]Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")Signed-off-by Pavel Begunkov <[email protected]>--- io_uring/rsrc.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)io_uring/rsrc.c +5
-2@@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)struct io_mapped_ubuf *imu = priv;unsigned int i; -for (i =0 ; i < imu->nr_bvecs; i++) -unpin_user_page (imu->bvec[i].bv_page); +for (i =0 ; i < imu->nr_bvecs; i++) { +struct folio *folio =page_folio (imu->bvec[i].bv_page); + +unpin_user_folio (folio,1 ); + } } static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
io_uring/rsrc.c static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, · line 820
Quoted 9 lines
io_uring/rsrc.c +2
-1@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->nr_bvecs = nr_pages; ret =io_buffer_account_pin (ctx, pages, nr_pages, imu, last_hpage);if (ret) { -unpin_user_pages (pages, nr_pages); +for (i =0 ; i < nr_pages; i++) +unpin_user_folio (page_folio (pages[i]),1 ); goto done; }
It should fix the issue, but it's a bit suboptimal in the case where we didn't coalesc, but there are folio ranges to coalesc:
unpin_user_pages() does a per-folio coalescing.
So in an ideal world, we would cleanly split both paths, and work with folios after we coalesced to use folios, and work with pages, when we didn't coalesc to use folios.
Then, we can just use unpin_folios() after we coalesced.
In any case, for a fix this is good enough, but probably we can do better later.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning
On 6/25/25 08:53, David Hildenbrand wrote:
...>> static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
Quoted 12 lines
io_uring/rsrc.c static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, · line 819
Quoted 9 lines
io_uring/rsrc.c +2
-1@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->nr_bvecs = nr_pages; ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage); if (ret) { - unpin_user_pages(pages, nr_pages); + for (i = 0; i < nr_pages; i++) + unpin_user_folio(page_folio(pages[i]), 1); goto done; }It should fix the issue, but it's a bit suboptimal in the case where we didn't coalesc, but there are folio ranges to coalesc:
Quoted 3 lines
unpin_user_pages()does a per-folio coalescing.So in an ideal world, we would cleanly split both paths, and work with folios after we coalesced to use folios, and work with pages, when we didn't coalesc to use folios.
Agreed, but I'm not too much concerned, it's a slow path, users should be mindful while registering memory. Localised hammering on the refcount is not great but ultimately shouldn't matter much.
Quoted 5 lines
Then, we can just use
unpin_folios()after we coalesced.In any case, for a fix this is good enough, but probably we can do better later.
Thanks for the report!
Signature · 1 line
Pavel Begunkov
PATCH v2 2/3 io_uring/rsrc: don't rely on user vaddr alignment
In reply to Pavel BegunkovThere is no guaranteed alignment for user pointers, however the calculation of an offset of the first page into a folio after coalescing uses some weird bit mask logic, get rid of it.
--- io_uring/rsrc.c | 7 ++++++- io_uring/rsrc.h | 1 + 2 files changed, 7 insertions(+), 1 deletion(-)
io_uring/rsrc.c +6 -1
@@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, data->nr_pages_mid =folio_nr_pages (folio); data->folio_shift =folio_shift (folio); + data->first_folio_page_idx =folio_page_idx (folio, page_array[0 ]);/* * Check if pages are contiguous inside a folio, and all folios have
@@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,if (coalesced) imu->folio_shift = data.folio_shift;refcount_set (&imu->refs,1 ); - off = (unsigned long ) iov->iov_base & ((1UL << imu->folio_shift) -1 ); + + off = (unsigned long )iov->iov_base & ~PAGE_MASK ; +if (coalesced) + off += data.first_folio_page_idx <<PAGE_SHIFT ; + node->buf = imu; ret =0 ;
io_uring/rsrc.h +1 -0
@@ -49,6 +49,7 @@ struct io_imu_folio_data {unsigned int nr_pages_mid;unsigned int folio_shift;unsigned int nr_folios; +unsigned long first_folio_page_idx; };bool io_rsrc_cache_init (struct io_ring_ctx *ctx);
Signature · 1 line
2.49.0
Re: [PATCH v2 2/3] io_uring/rsrc: don't rely on user vaddr alignment
Quoted 43 lines · Pavel Begunkov
There is no guaranteed alignment for user pointers, however the calculation of an offset of the first page into a folio after coalescing uses some weird bit mask logic, get rid of it.
Reported-by David Hildenbrand <[email protected]>Fixes a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")Signed-off-by Pavel Begunkov <[email protected]>--- io_uring/rsrc.c | 7 ++++++- io_uring/rsrc.h | 1 + 2 files changed, 7 insertions(+), 1 deletion(-)io_uring/rsrc.c +6
-1@@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, data->nr_pages_mid =folio_nr_pages (folio); data->folio_shift =folio_shift (folio); + data->first_folio_page_idx =folio_page_idx (folio, page_array[0 ]);/* * Check if pages are contiguous inside a folio, and all folios have @@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,if (coalesced) imu->folio_shift = data.folio_shift;refcount_set (&imu->refs,1 ); - off = (unsigned long ) iov->iov_base & ((1UL << imu->folio_shift) -1 ); + + off = (unsigned long )iov->iov_base & ~PAGE_MASK ; +if (coalesced) + off += data.first_folio_page_idx <<PAGE_SHIFT ; + node->buf = imu; ret =0 ;io_uring/rsrc.h +0
-0
io_uring/rsrc.h struct io_imu_folio_data { · line 55
Quoted 7 lines
io_uring/rsrc.h +1
-0@@ -49,6 +49,7 @@ struct io_imu_folio_data {unsigned int nr_pages_mid;unsigned int folio_shift;unsigned int nr_folios; +unsigned long first_folio_page_idx; };bool io_rsrc_cache_init (struct io_ring_ctx *ctx);
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 3/3 io_uring: don't assume uaddr alignment in io_vec_fill_bvec
In reply to Pavel BegunkovThere is no guaranteed alignment for user pointers. Don't use mask trickery and adjust the offset by bv_offset.
--- io_uring/rsrc.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
io_uring/rsrc.c +7 -3
@@ -1336,7 +1336,6 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, {unsigned long folio_size =1 << imu->folio_shift;unsigned long folio_mask = folio_size -1 ; -u64 folio_addr = imu->ubuf & ~folio_mask;struct bio_vec *res_bvec = vec->bvec;size_t total_len =0 ;unsigned bvec_idx =0 ;
@@ -1358,8 +1357,13 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter,if (unlikely (check_add_overflow (total_len, iov_len, &total_len)))return -EOVERFLOW ; -/* by using folio address it also accounts for bvec offset */ - offset = buf_addr - folio_addr; + offset = buf_addr - imu->ubuf; +/* +* Only the first bvec can have non zero bv_offset, account it +* here and work with full folios below. +*/ + offset += imu->bvec[0 ].bv_offset; + src_bvec = imu->bvec + (offset >> imu->folio_shift); offset &= folio_mask;
Signature · 1 line
2.49.0
Jens Axboe
maintainerQuoted 7 lines · Pavel Begunkov
Patch 1 uses
unpin_user_folioinstead of the page variant. Patches 2-3 make sureio_uringdoesn't make any assumptions about user pointer alignments.v2: change patch 1 tags
usefolio_page_idx()
Quoted 8 lines
Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinningio_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment inio_vec_fill_bvecio_uring/rsrc.c | 27 ++++++++++++++++++++------- io_uring/rsrc.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)
Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!
Signature · 1 line
Jens Axboe
Quoted 13 lines · Jens Axboe
Quoted 12 lines · Pavel Begunkov
Patch 1 uses
unpin_user_folioinstead of the page variant. Patches 2-3 make sureio_uringdoesn't make any assumptions about user pointer alignments.v2: change patch 1 tags
usefolio_page_idx()Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinningio_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment inio_vec_fill_bvec
Quoted 6 lines
Quoted 3 lines
io_uring/rsrc.c | 27 ++++++++++++++++++++------- io_uring/rsrc.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!
Turned to be for-next from a couple of days ago. Patch 1 looks the same, should be fine.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 18 lines · Pavel Begunkov
Quoted 17 lines · Jens Axboe
Quoted 15 lines · Pavel Begunkov
Patch 1 uses
unpin_user_folioinstead of the page variant. Patches 2-3 make sureio_uringdoesn't make any assumptions about user pointer alignments.v2: change patch 1 tags
usefolio_page_idx()Pavel Begunkov (3):
io_uring/rsrc: fix folio unpinningio_uring/rsrc: don't rely on user vaddr alignment
io_uring: don't assume uaddr alignment inio_vec_fill_bvecio_uring/rsrc.c | 27 ++++++++++++++++++++------- io_uring/rsrc.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)
Quoted 5 lines
Quoted 2 lines
Hand applied, as this is against an older tree. Please check patch 1 in the current tree. Thanks!
Turned to be for-next from a couple of days ago. Patch 1 looks the same, should be fine.
I don't always put current fixes in for-next, though I've tried to do it consistently more recently. But it's conflicting with the error path cleanup from about a week ago:
commit e1c75831f682eef0f68b35723437146ed86070b1 (tag: io_uring-6.16-20250619)
Author: Penglei Jiang <[email protected]> Date: Tue Jun 17 09:56:44 2025 -0700
io_uring: fix potential page leak in io_sqe_buffer_register()
and my hand-edit just put your hunk 2 of patch 1 into that cleanup path too. Thanks for checking!
Signature · 1 line
Jens Axboe
Quoted 8 lines · Pavel Begunkov
Patch 1 uses
unpin_user_folioinstead of the page variant. Patches 2-3 make sureio_uringdoesn't make any assumptions about user pointer alignments.v2: change patch 1 tags
usefolio_page_idx()[...]
Applied, thanks!
[1/3] io_uring/rsrc: fix folio unpinning
commit: 5afb4bf9fc62d828647647ec31745083637132e4
[2/3] io_uring/rsrc: don't rely on user vaddr alignment
commit: 3a3c6d61577dbb23c09df3e21f6f9eda1ecd634b
[3/3] io_uring: don't assume uaddr alignment in io_vec_fill_bvec
commit: e1d7727b73a1f78035316ac35ee184d477059f0bBest regards,
Signature · 1 line
Jens Axboe